Audit-Log Every Agent Tool Call in Production

When a worker refunds the wrong invoice or comments the wrong secret on a GitHub issue, the question is not "did the model mean well." The question is which tool ran, with which arguments, as which Unix user, after which model response. If that sentence is not reconstructible from a file on disk, you do not have an agent in production. You have a process that talks.

An audit log is not journald of stdout. It is a JSONL stream with a schema, written before the tool runs and closed after, with hashes of arguments and results so you can prove a line was not edited. High-Speed SSD in Lucknow is cheap enough to keep 30 days. Intel Xeon Platinum will not notice the extra write.

Wrap tools, do not trust them to log

Every tool is a function you own. The model does not call HTTP; your dispatcher does. That dispatcher is the only place the audit line should be born. If a tool logs internally, you will lose the lines from the one tool someone added on Friday.

# /opt/agents/audit.py
import hashlib, json, os, time, uuid
from pathlib import Path

LOG = Path("/var/log/agents/audit.jsonl")
ROLE = os.environ.get("AGENT_ROLE", "unknown")

def _hash(obj):
    blob = json.dumps(obj, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(blob).hexdigest()

def wrap(name, fn):
    def inner(args):
        rec = {
            "id": str(uuid.uuid4()),
            "ts": time.time(),
            "role": ROLE,
            "pid": os.getpid(),
            "tool": name,
            "args_hash": _hash(args),
            "args": redacted(args),
            "status": "started",
        }
        append(rec)
        try:
            out = fn(args)
        except Exception as exc:
            rec["status"] = "error"
            rec["error"] = type(exc).__name__
            rec["dur_ms"] = int((time.time() - rec["ts"]) * 1000)
            append(rec)
            raise
        rec["status"] = "ok"
        rec["result_hash"] = _hash(out)
        rec["dur_ms"] = int((time.time() - rec["ts"]) * 1000)
        append(rec)
        return out
    return inner

def append(rec):
    line = json.dumps(rec, separators=(",", ":")) + "
"
    with LOG.open("a", encoding="utf-8") as fh:
        fh.write(line)
        fh.flush()
        os.fsync(fh.fileno())

fsync is the difference between a log and a wish. A crash after a refund must still have the started line. Two lines per call (started, ok/error) let you find tools that never returned.

Redact by default

Args go into the log in a redacted shape, never raw. A token, a cookie, a PEM, a password, and anything named secret, key, token, authorization get replaced with their hash. Results are hashed, not stored, unless the tool is labeled safe_to_log (link_status, lookup_label). Issue bodies and customer mail are not safe_to_log.

# /opt/agents/redact.py
SENSITIVE = ("secret", "key", "token", "password", "authorization", "cookie", "pem")

def redacted(obj):
    if isinstance(obj, dict):
        out = {}
        for k, v in obj.items():
            lk = k.lower()
            if any(s in lk for s in SENSITIVE):
                out[k] = "h:" + _hash(v)[:12]
            else:
                out[k] = redacted(v)
        return out
    if isinstance(obj, list):
        return [redacted(x) for x in obj]
    if isinstance(obj, str) and len(obj) > 400:
        return obj[:400] + "...trunc"
    return obj

If a GitHub comment tool must keep the body for replay, write it under /var/lib/agents/blob/id with mode 0640 and store the path in the audit line. Do not put the body in JSONL. JSONL is what you will one day ship to a SIEM or paste into a ticket.

Who reads it, who cannot change it

The agent user appends. A second user, agent-audit, owns a nightly copy to /var/log/agents/audit/ with chattr +a on the live file if you want append-only at the filesystem. logrotate with copytruncate is the wrong mode here; rotate by rename and reopen. A small systemd path unit can sha256sum yesterday's file into a manifest.

Access: operators read, agents write, nobody truncate. That is the same policy you want for payment logs. If an incident asks "did we post the API key," you grep args_hash and redacted fields, you do not hope.

Correlation, clocks, and what you refuse to store

Stamp every audit line with a job id from the envelope and a request id from the model HTTP wrapper. Without those two fields you cannot join "the model said X" to "the tool did Y." Use time.time() as a float plus the unit's CLOCK_REALTIME; if you NTP-step the VPS, a 30-second jump is less painful than missing fsync.

Do not store full prompts in the audit JSONL. Prompts contain customer mail, runbook chunks, and sometimes keys that a junior pasted into a ticket. Store prompt_hash and a byte length. If a regulator or a customer asks what the model saw, you reconstruct from the blob store and the retrieval hit ids, not from a world-readable log. The same rule applies to screenshots and to browser accessibility trees if you ignored the earlier post and still run one.

What you grep during an incident

Start with tool name and a time window. Then role. Then args_hash if you have a known payload. A useful one-liner on the VPS:

python3 -c "import json,sys
for line in open('/var/log/agents/audit.jsonl'):
    r=json.loads(line)
    if r.get('tool')=='github_comment' and r.get('status')=='ok':
        print(r['ts'], r['role'], r['id'], r.get('args',{}).get('issue'))"

That is enough to list every comment the bot actually posted, which is the list GitHub's UI will not give you in one click. Pair it with the GitHub delivery id from the webhook post if you need to prove a duplicate.

Takeaway: wrap the dispatcher, fsync a started line, redact secrets, hash results, keep blobs off the JSONL. That is an audit trail you can hand a human. Run it on Ubuntu 24.04 in Lucknow DC01 from Netbay — netbayhosts.in.