Test Your Restores: Unverified Backups Are Not Backups

A backup that has never been restored is, at best, metadata: a timestamp and a size in a listing. File presence proves the scheduler ran; it says almost nothing about whether the archive can become a running application. The cheapest club you can join in operations is the one that regularly hands a backup to a clean machine and watches what actually happens. This post turns that instinct into a repeatable restore-test drill.

What a restore test actually verifies

The obvious check is archive integrity: can the tool read the file back? Behind that sit four quieter checks that fail everything anyway: the restore toolchain exists on the target, credentials and keys can be recovered, dependencies and permissions land in the right places, and the application boots and serves a request. A dump that restores into a bare psql but not into a running web app has failed its real mission.

The cheapest meaningful test: integrity only

Start daily, in one line, on the same box that wrote the archives. Verify that the newest archive parses and reports its contents:

pg_restore --list /var/backups/postgres/appdb.dump | head -20
gunzip -t /var/backups/mysql/appdb.sql.gz
restic check -r sftp:offsite:/backups/app

These catch corruption, truncated uploads, and repository damage. They cannot see missing tables, stale credentials, or a schema that changed after the dump. That is what the weekly drill is for.

The real test: restore to a scratch machine

Provision a disposable instance, restore the archive into it, boot the application, and hit a health endpoint. This is where a runbook earns its keep, because the same people doing the drill will do the incident.

#!/usr/bin/env bash
# /usr/local/bin/restore_drill.sh
set -euo pipefail
TARGET=/var/restore-drill
restic -r sftp:offsite:/backups/app restore latest --target $TARGET
pg_restore --create --dbname=postgres $TARGET/appdb.dump
systemctl start app
curl -fsS --max-time 10 http://127.0.0.1/healthz
pgrep -f app-server >/dev/null

Run the drill weekly, after the Monday dump lands, and keep a log the whole team can read:

# /etc/cron.d/restore-drill
30 5 * * 0 root /usr/local/bin/restore_drill.sh >> /var/log/restore-drill.log 2>&1

Metric: how long did it take

While the drill runs, start a stopwatch. The wall-clock time from the failure declaration to a green health check is your real RTO; whatever number your planning document claims, the drill result is the only honest one. Log it weekly, chart it monthly, and watch it fall as the runbook improves.

Failures you will actually hit

Dependencies that exist only on the source host. Keys embedded in a .env that was never backed up. Cron jobs that write to paths the restore target has not created. An encrypted archive whose private key lives on the dead host. Every one of these gets found by drills and skipped by hope. Keep the checklist in the runbook and add a line whenever a drill surprises you.

Takeaway

A backup becomes a backup at the moment you prove it restores. Integrity checks daily, a full restore-plus-boot drill weekly, and a recorded restore time that keeps you honest about RTO. Run the drill on a disposable Netbay VPS — spin one up in under a minute at netbayhosts.in whenever you need a clean target.