Disk and CPU Watchdog Agent with Allowlists

A watchdog agent that can run arbitrary shell is a rootkit with a chat transcript. The version that belongs on a VPS reads df, load, and memory, then picks from a named allowlist: vacuum the journal, apt-get clean, reload nginx, restart a known unit, or page a human. Nothing else. No rm. No chmod 777. No iptables flush. Every action is logged with the metric that triggered it. On Intel Xeon Platinum in Lucknow DC01, High-Speed SSD fills because of logs, dumps, and container layers, not because the disks are slow. The agent should know those three suspects before it starts inventing hardware failures.

Sample first, with numbers a human would trust

Every minute, collect: df -P of / and /var, inode percent, load average versus vCPU count, MemAvailable, systemd failed units, and disk await if you have iostat. Write /run/watchdog/sample.json. Thresholds live in /etc/watchdog-agent.yaml: disk 90 percent warn, 95 percent act, load 2x vCPU for 10 minutes, MemAvailable under 150 MB. A 2 vCPU Lucknow plan that sits at load 0.3 all day and spikes to 6 during a backup is not a CPU emergency; it is a backup window. Encode that in cooldown rules, not in vibes.

Do not scrape nvidia or other-vendor CPU names. The host is Intel Xeon Platinum. Steal time from vmstat still matters on a VPS; if steal is high, the allowlist action is page, not restart nginx.

The allowlist is a map of names to argv

The model returns action, reason, and maybe a unit from a unit allowlist. The runner looks up argv. If the name is missing, it pages. That is the entire security model. Put the map in git. Review changes like you review sudoers.

import json, subprocess, time

ACTIONS = {
    "journal_vacuum": ["journalctl", "--vacuum-size=200M"],
    "apt_clean": ["apt-get", "-y", "clean"],
    "nginx_reload": ["systemctl", "reload", "nginx.service"],
    "caddy_reload": ["systemctl", "reload", "caddy.service"],
    "restart_api": ["systemctl", "restart", "api.service"],
    "page": ["true"],
}
UNITS_OK = {"nginx.service", "caddy.service", "api.service", "worker.service"}
COOLDOWN_S = {"journal_vacuum": 3600, "restart_api": 900}

def run_action(name, last):
    if name not in ACTIONS:
        raise ValueError("not allowed")
    now = time.time()
    if name in COOLDOWN_S and now - last.get(name, 0) < COOLDOWN_S[name]:
        return {"skipped": "cooldown"}
    subprocess.check_call(ACTIONS[name])
    last[name] = now
    return {"ran": name}

Notice restart_api is a fixed argv, not systemctl restart plus a string the model invented. If you want worker restarts, add restart_worker as its own key. Cooldowns stop crash loops from becoming the agent's personality.

# /usr/local/bin/watchdog-sample
set -euo pipefail
install -d /run/watchdog
DF=$(df -P / | awk "END {print $5}" | tr -d %)
LOAD=$(cut -d " " -f 1 /proc/loadavg)
MEM=$(awk "/MemAvailable:/ {print $2}" /proc/meminfo)
printf '{"disk_pct":%s,"load1":%s,"mem_avail_kb":%s}
' "$DF" "$LOAD" "$MEM"   > /run/watchdog/sample.json
python3 /opt/watchdog/agent_pick.py /run/watchdog/sample.json

awk again uses $2 inside the TypeScript template so the blog shows $2. The sample runs as a dedicated user with polkit rules for the exact systemctl verbs, not NOPASSWD: ALL. apt_clean needs extra privileges; if you do not want the agent to have them, delete the key.

Act small, then re-sample

After journal_vacuum, df should move. If it does not, the fill is not the journal; next action is page, not vacuum again. After nginx_reload, curl /health. After restart_api, wait for the health loop you already use in deploys. Log JSON lines to /var/log/watchdog-agent.jsonl: sample, pick, argv, exit code, df after. Rotate. When a human asks why nginx reloaded at 02:14, that file is the answer.

Failed units from systemctl --failed belong in the sample. The agent may pick restart_api only if api.service is in that list and the cooldown allows it. It may not pick restart_api because load is high; high load plus a healthy API is a different story (backup, SQL, or a neighbor on the node). L3/L4 DDoS filtering may already be dropping junk; a load spike with no local sockets is a page to look at the edge, not a reload.

What never goes on the list

rm -rf, find -delete, mkfs, swapoff, iptables -F, ufw disable, userdel, dd, and anything that takes a free-form path. Disk-full caused by /var/backups is a retention timer, not an agent with delete powers. If you must free space by deleting, write a script that deletes only /var/backups/*.dump older than 14 days and add that exact script as a single argv. Review it like production DDL.

Takeaway

A watchdog agent is metrics, thresholds, and a map of names to argv with cooldowns. The model picks a name; the runner refuses everything else. Install the sampler on a Netbay Lucknow VPS — Ubuntu 24.04 in under 60 seconds at netbayhosts.in — and keep rm off the list.