Draft Status Page Copy with an Incident Agent

Status copy written in a panic is how you get "we are aware of an issue" with no timestamp, no product name, and a promise you cannot keep. An incident-draft agent does not publish. It collects the same facts an incident commander would scrape in the first ten minutes and emits two documents: a public status draft and an internal timeline. A human edits, then a human hits publish. That split is the whole product. On a small team running apps on Lucknow VPS hosts, you already have the inputs: alert JSON, the last deploy SHA, nginx upstream health, and journal fingerprints. The agent is a writer with a fact bag, not a spokesperson.

Build a fact bag, not a novel

The prompt should receive a JSON object you assembled, not a screenshot of Slack. Collect at incident open: alert name and threshold, first-fire time in UTC, affected hostname, last successful deploy SHA and who merged it, current curl of /health from localhost, df -h of the data disk, and the top three journal fingerprints from the last fifteen minutes. Drop secrets. Drop customer names unless they are already in the public ticket. If a field is missing, send null, not a guess. The model is a compressor. Garbage in the fact bag becomes confident fiction on the status page.

Store the fact bag at /var/lib/incident/current.json with mode 0640. The agent writes /var/lib/incident/draft-public.md and draft-internal.md next to it. Both files are gitignored on the box and copied into the incident ticket. Lucknow DC01 clocks should be NTP-synced so timestamps in the draft match systemd and your uptime checker. A VPS on High-Speed SSD will still lie about timing if the clock is wrong.

Public copy has a smaller vocabulary than internal copy

Public status is four states: investigating, identified, monitoring, resolved. Each public sentence names the product, the user-visible symptom, and what you are doing next. It does not name Intel Xeon Platinum, systemd, or the database host. It does not guess a root cause in the first update. Internal copy can say worker.service crash-looped after SHA 9f3c, disk was 94 percent, and the likely next step is vacuum plus a revert. Train the prompt with three worked examples of each state. Ban phrases like "we take this very seriously" and "unprecedented." Ban ETAs unless the fact bag contains a real rollback already in progress.

import json, datetime

STATES = ("investigating", "identified", "monitoring", "resolved")

def validate_draft(doc):
    if doc.get("state") not in STATES:
        raise ValueError("bad state")
    pub = doc.get("public") or {}
    if not pub.get("headline") or not pub.get("body"):
        raise ValueError("public copy missing")
    if "ETA" in pub.get("body", "") and not doc.get("facts", {}).get("rollback_started"):
        raise ValueError("ETA without rollback")
    if len(pub.get("body", "").split()) > 120:
        raise ValueError("public body too long")
    return True

facts = json.load(open("/var/lib/incident/current.json"))
print(json.dumps({
    "generated_at": datetime.datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ"),
    "state": "investigating",
    "facts_keys": sorted(facts.keys()),
}))

The validator is the real product. If the model returns a fifth state, or an ETA, or a public body longer than 120 words, discard the draft and keep the previous one. On-call should never wrestle a broken paragraph under pressure.

Timeline entries are one fact each

Internal timelines rot when someone pastes a novel. Force the agent to emit a list of timestamped bullets, each one a single fact: alert fired, deploy SHA, health failed, disk percent, action taken, owner. No adjectives. When a human runs a command, they append a bullet; the agent may only add bullets sourced from the fact bag. That rule stops the model from inventing "restarted nginx" because it sounded plausible.

# /usr/local/bin/incident-open  (called by the pager webhook)
set -euo pipefail
DIR=/var/lib/incident
install -d -m 0750 "$DIR"
HOST=$(hostname)
SHA=$(cat /opt/app/.deploy-sha 2>/dev/null || echo unknown)
HEALTH=$(curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8080/health || echo 000)
DISK=$(df -P /var | awk "END {print $5}")
printf '{"host":"%s","sha":"%s","health_http":"%s","disk":"%s","opened_at":"%s"}
'   "$HOST" "$SHA" "$HEALTH" "$DISK" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" > "$DIR/current.json"
python3 /opt/incident/draft_agent.py "$DIR/current.json"

Note the df line uses awk with a quoted script so you never need brace expansion for the field. Wire this from the same webhook that pages you. The agent should finish in under fifteen seconds on a quiet Xeon Platinum vCPU. If it cannot, ship a template with blanks rather than hanging the pager.

Publishing is a separate button

Do not give the agent the status-page API token. A second user, or a checked box in your chat ops bot, posts the public paragraph. That human is also the person who can say "do not publish yet, this is a single-tenant glitch." L3/L4 DDoS filtering on the VPS edge can make a flood look like an app outage; the draft should mention only what users see until you know. When the incident closes, the agent can propose a resolved update from the same fact bag plus the close reason a human typed. Keep the drafts. They are training data for the next prompt change.

Takeaway

An incident agent is a fact compressor with two output files and no publish rights. Feed it alerts, SHA, health, and disk; demand short public copy and a timeline of single facts. You can run the fact-bag script on a Netbay Lucknow VPS in under an hour, and netbayhosts.in will have the Ubuntu 24.04 host up in under 60 seconds.