Log-Triage Agent Over journalctl on a VPS

A VPS that has been up for six months will happily emit 40,000 journal lines in a bad fifteen minutes. Pasting that blob into a chat window is not triage. A log-triage agent over journald is a small pipeline: collect a bounded window, strip secrets, group by unit and fingerprint, then ask the model to rank likely causes and the next command a human should run. The agent does not SSH as root and it does not restart services. It reads. On a Lucknow VPS with Intel Xeon Platinum cores and High-Speed SSD, journald is already on disk; you are teaching a model to read it the way an on-call engineer does at 3 a.m.

Bound the window before you call the model

journalctl without a since filter is how you blow the context window and your token bill. Always pass --since, a unit list, and a line cap. JSON output (-o json or -o json-pretty) is easier to group than the default short format because every record already has _SYSTEMD_UNIT, PRIORITY, MESSAGE, and SYSLOG_IDENTIFIER. Collect from a systemd timer every five minutes, or from a pager webhook that only fires when a unit flaps. Write the raw slice under /var/lib/log-triage/ with mode 0640. Do not ship the raw slice off-box if it may contain session cookies or Authorization headers.

Redact before the prompt. A short Python pass that blanks Bearer tokens, AWS-style keys, and anything matching password= is mandatory. Cap at 8,000 lines or 400 KB, whichever is smaller. If you hit the cap, tell the model the window was truncated and include the dropped-line count. Truncation is a fact, not a failure; hiding it makes the ranking look more confident than it is.

Fingerprint first, summarize second

The model should not see 8,000 unique lines if 7,400 of them are the same stack trace. Group by unit plus a normalized message: strip PIDs, strip ISO timestamps, strip hex pointers, then hash. Send the top N fingerprints with counts, first-seen, last-seen, and one example line. That is the same move a tired human makes by running uniq -c. The agent then assigns severity (info, warn, error, page), a one-line cause, and a next_cmd drawn from an allowlist. If the model invents a command that is not in the list, drop it. This path is read-only. Restarts belong to a different agent with a different key.

# /usr/local/bin/journal-slice  (run by systemd timer)
set -euo pipefail
OUT=/var/lib/log-triage/slice.jsonl
install -d -m 0750 /var/lib/log-triage
journalctl --since "15 min ago" -o json --no-pager   -u api.service -u worker.service -u nginx.service -u caddy.service   | head -n 8000 > "$OUT"
wc -l "$OUT"

Keep the unit list in a file, not in the model prompt as a free-form request. If a new unit starts crashing, add it to the list in git. journald on High-Speed SSD is cheap to query for fifteen minutes; it is not cheap to dump since boot into a prompt.

What a useful ranking looks like

Ask for JSON, validate it, and render a short Slack or email body from the JSON, never from free prose. Fields that have paid for themselves in real incidents: unit, fingerprint, count, first_ts, last_ts, severity, likely_cause, next_cmd, confidence. Confidence below 0.5 should still show the fingerprint counts but label the cause as unknown. The next_cmd values you should allow are boring on purpose: systemctl status UNIT, journalctl -u UNIT -n 50, df -h, ss -tulpn, curl -fsS http://127.0.0.1:8080/health. That is enough to tell OOM (priority 3, killed process) from a full disk (journald complaining, df 100 percent) from a crash loop (N starts in fifteen minutes).

import json, collections, re, hashlib

def norm(msg):
    msg = re.sub(r"pid=d+", "pid=N", msg)
    msg = re.sub(r"0x[0-9a-f]+", "0xX", msg, flags=re.I)
    msg = re.sub(r"d{4}-d{2}-d{2}T[d:.]+Z", "TS", msg)
    return msg

groups = collections.Counter()
example = {}
for line in open("/var/lib/log-triage/slice.jsonl"):
    rec = json.loads(line)
    unit = rec.get("_SYSTEMD_UNIT") or rec.get("SYSLOG_IDENTIFIER") or "unknown"
    key = unit + "|" + hashlib.sha1(norm(rec.get("MESSAGE", "")).encode()).hexdigest()[:12]
    groups[key] += 1
    example.setdefault(key, rec.get("MESSAGE", "")[:240])
top = groups.most_common(20)
print(json.dumps([{"fp": k, "n": n, "ex": example[k]} for k, n in top], indent=2))

Feed that top-20 JSON to the model, not the raw journal. Mention host facts that change interpretation: this is a VPS, not a laptop, so CPU steal and disk wait matter; L3/L4 DDoS filtering may already have dropped junk before it hit nginx. Do not claim a packet flood if ss is quiet and the journal is full of application stack traces.

Wire it as a timer, not a chat habit

A oneshot systemd service plus a five-minute timer is enough. The service runs the slice, the fingerprint script, then the agent, then posts to a webhook if any fingerprint is severity page or if a unit has more than 200 lines in the window. Rate-limit posts: one page per unit per hour unless the fingerprint changes. Store the last ranking at /var/lib/log-triage/last.json so the next run can say this is still the same crash. Vacuum the journal on a schedule separately; triage is not cleanup.

When the agent is wrong, add the slice and the expected ranking to a fixture directory. That is how you stop the model from calling every 502 a database outage. Fixture-based evals belong in a later post; the operational habit starts here: never promote a prompt change without replaying last week's slices.

Takeaway

Triage is bounded journald, fingerprints, and an allowlisted next command. The model ranks; the box still belongs to you. Spin up an Ubuntu 24.04 VPS on Netbay in Lucknow in under 60 seconds and point journalctl at a noisy unit — netbayhosts.in.