Evaluate AI Agents With Traces, Not Vibes

A demo is not an evaluation. Watching an agent restart nginx once on a Friday is how teams ship a loop that also restarts it on a cascade of 502s from the app. Evaluation for agents is replay: frozen observations, expected tool names, forbidden tools, and a trace you can diff. If you cannot say why last week's prompt change is better, you are tuning vibes.

This is not academic benchmarking. It is a pytest file and a directory of JSONL traces on the same Linux VPS that runs the bot. The model API is a dependency you stub for unit tests and call live for a small nightly set.

What a Trace Must Contain

Each turn is one JSON object: step, role, tool name, argument fingerprint, observation hash, tokens in, tokens out, latency, finish reason. Without argument fingerprints you cannot tell a restart of nginx from a restart of app. Without finish reason you cannot tell a real answer from a step-budget abort.

Write traces even when you stub tools. The loop should not know whether read_unit_status hit systemd or a fixture. That seam is what makes offline eval possible.

import json
import hashlib

def fp(value):
    blob = json.dumps(value, sort_keys=True, default=str).encode()
    return hashlib.sha256(blob).hexdigest()[:12]

def trace_turn(path, step, tool, args, observation, usage, reason):
    row = {
        "step": step,
        "tool": tool,
        "args_fp": fp(args),
        "obs_fp": fp(observation[:500]),
        "prompt_tokens": usage.get("prompt_tokens", 0),
        "completion_tokens": usage.get("completion_tokens", 0),
        "reason": reason,
    }
    with open(path, "a", encoding="utf-8") as handle:
        handle.write(json.dumps(row) + "\n")

Store the raw observation in a sidecar file if you need it for debugging. The scored artifact is the compact JSONL. Compact means you can keep months of evals on High-Speed SSD without drama.

Golden Tasks, Not Open-Ended Chats

Write tasks with a pass condition. Examples that pay rent:

• nginx inactive, app health 200: expected tool restart_unit with unit nginx.service, no app restart.
• both health endpoints 200: expected zero write tools, final message cites the 200s.
• unknown unit name from the model: expected error string, no subprocess.
• step budget 4, tool that always returns the same error: expected abort, not a 40-call bill.

Each task has a fixtures dict that the dispatch uses instead of the real world. Replay is deterministic if the model is stubbed. When you include a live model, run three times and record pass rate, not a single screenshot.

GOLDEN = [
    {
        "id": "restart-nginx-only",
        "fixtures": {
            "read_unit_status": "ActiveState=inactive",
            "http_get_health": "status=200 body=ok",
        },
        "expect_tools": ["read_unit_status", "http_get_health", "restart_unit"],
        "forbid_tools": ["restart_unit:app.service"],
        "must_finish": "stop",
    },
    {
        "id": "healthy-no-writes",
        "fixtures": {
            "read_unit_status": "ActiveState=active",
            "http_get_health": "status=200 body=ok",
        },
        "expect_tools": ["read_unit_status", "http_get_health"],
        "forbid_tools": ["restart_unit", "reload_unit"],
        "must_finish": "stop",
    },
]

def score(trace_rows, task):
    names = [row["tool"] for row in trace_rows if row.get("tool")]
    for needed in task["expect_tools"]:
        if needed not in names:
            return False, "missing " + needed
    for banned in task["forbid_tools"]:
        if banned in names:
            return False, "forbid " + banned
    if trace_rows[-1].get("reason") != task["must_finish"]:
        return False, "bad finish"
    return True, "ok"

Start with ten tasks. If you cannot write ten, you do not know what the agent is for. Resist a huge suite of vague "be helpful" items. Those are vibes with YAML.

Nightly Live, Always Offline

Offline tests run in CI with a recorded model response or a fake that emits a fixed tool_calls payload. They catch dispatch bugs, schema breaks, and guardrail regressions without spending tokens.

Live tests run from a systemd timer at night against the real API with fixtures still in place for the tools. They catch prompt drift and provider format changes. Cap them. A nightly bill should be smaller than a single confused production run.

When a live test fails, keep the trace. Diff it with the last pass. The diff is the review. Do not accept "it felt better after we added a sentence to the system prompt" as a merge reason.

Intel Xeon Platinum on a Lucknow VPS will run pytest in seconds. The bottleneck is the model API, which is why fixtures exist. You do not need a GPU SKU and you do not need a cluster to evaluate a bot that has six tools.

Version the golden tasks with the prompt. A prompt change that is not a code change still needs a trace diff. Store the system prompt hash in the trace header so you can group failures by prompt, not by vibes. If pass rate on the live nightly set drops, revert the prompt the same way you revert a bad nginx config: last known good file, then investigate. High-Speed SSD holds the JSONL; git holds the tasks; neither is optional.

Takeaway

If you cannot score a trace against a golden task, you are not evaluating the agent. You are watching it. Run pytest on a Netbay Linux VPS in Lucknow and keep the JSONL next to the bot at netbayhosts.in.