Backup Verification Agent for Nightly Dumps

A backup job that exits zero after writing a 20-byte file is a successful lie. A backup-verification agent treats last night's dump as a test artifact: it must be recent, larger than a floor, have a real archive header, and restore into a throwaway directory or database. The agent does not invent extra copies in other cities. Netbay is Lucknow only. It also does not push to a mystery object store you do not run. It reads /var/backups, runs checks you already wrote, and writes a JSON report a human or a pager can understand. The model chooses which checks to emphasize when something looks off. The scripts still own gzip, pg_restore, and tar.

Static checks catch the cheap lies

Before you spend CPU on a restore, ask four questions. Does the expected filename exist? Is mtime within 26 hours? Is the size at least yesterday's size times 0.5, and at least a configured floor (for example 1 MB for a tiny app, much more for Postgres)? Do the first bytes match gzip (1f 8b) or your custom header? A zero-byte file, a 47-byte HTML error saved as .dump, and a file dated last month all fail here. Put the floor in a config file, not in the model. On High-Speed SSD the stat is instant; the lie is still a lie.

Keep dumps on the VPS and, if you copy off-box, verify both copies with the same checks. Do not claim a second datacenter. If you scp to a second Lucknow VPS, say that. L3/L4 DDoS filtering on the public NIC does not protect a backup that never ran.

# /usr/local/bin/backup-stat
set -euo pipefail
FILE=$1
FLOOR=$2
python3 - "$FILE" "$FLOOR" << 'PY'
import os, sys, time, json
path, floor = sys.argv[1], int(sys.argv[2])
st = os.stat(path)
age = time.time() - st.st_mtime
with open(path, "rb") as f:
    magic = f.read(2)
print(json.dumps({
    "path": path,
    "bytes": st.st_size,
    "age_s": int(age),
    "gzip": magic == b"‹",
    "ok_age": age < 26 * 3600,
    "ok_size": st.st_size >= floor,
}))
PY

The heredoc delimiter is quoted so the shell does not expand anything. Feed that JSON to the agent alongside yesterday's bytes so it can say the dump shrank 80 percent instead of "looks small."

Restore is a list, then a real open

For Postgres custom format, pg_restore --list is the cheap restore: it proves the archive TOC parses. For tar.gz of /var/www, tar -tzf is the cheap restore. Once a day, not on every timer tick, do the expensive path: restore into /tmp/restore-test or into a throwaway database restore_agent, run a row-count query you already trust, then drop it. Bound the work with timeout(1). Intel Xeon Platinum will chew a small dump quickly; a huge dump should restore on a timer at 05:00, not when the agent feels curious.

# /usr/local/bin/backup-restore-check
set -euo pipefail
DUMP=$1
export PGDATABASE=restore_agent
dropdb --if-exists "$PGDATABASE"
createdb "$PGDATABASE"
timeout 120 pg_restore --no-owner --role=restore_agent -d "$PGDATABASE" "$DUMP"
psql -c "SELECT relname, n_live_tup FROM pg_stat_user_tables ORDER BY n_live_tup DESC LIMIT 10;"
dropdb "$PGDATABASE"

Connect as a role that can only create the throwaway database, not touch production. Never point this at the app DSN. Never let the agent choose the dump path with a relative ../../../. The timer always passes an absolute path from a glob you control: /var/backups/pg/$(date +%F).dump.

The agent writes the narrative, not the commands

After the scripts run, the model receives the JSON and must return: status pass/fail, which check failed, likely cause, and next_cmd from an allowlist (ls -l /var/backups, journalctl -u pg-dump.service -n 50, df -h /var). It does not get to run pg_dump itself. It does not get to delete old dumps; retention is a separate timer with a documented find -mtime. If verification fails, page. If verification cannot run because the disk is 100 percent, that is a watchdog problem, and the report should say so instead of blaming Postgres.

Store reports under /var/lib/backup-verify/ with the date in the filename. Graph pass/fail. A week of silent passes is how you trust the 3 a.m. restore you have never done with a customer on the line.

Takeaway

Backup verification is age, size, magic bytes, and a restore list, plus an agent that explains failures without touching production. Keep dumps on High-Speed SSD in Lucknow and test them. You can put this timer on a Netbay Ubuntu 24.04 VPS in under an hour — netbayhosts.in, under 60 seconds to boot.