Disaster Recovery Runbook: From Backup to Running Server

A disaster recovery runbook is the difference between a bad night and a ruined month. It converts one ambiguous statement, "something is broken, restore the backups", into a sequence of specific commands that any on-call engineer can execute at 3 a.m. with a functioning but alarmed brain. The runbook does not solve the disaster; it shortens it to the one skill that actually remains: restoring data correctly. This post lays out a runbook skeleton you can fill in for your own stack in an afternoon.

The ten-step skeleton

1. Declare the incident and the on-call pair; freeze new migrations and deploys.
2. Confirm scope: database, application files, or both. Keep the failing host powered down so nothing writes to it.
3. Identify the latest known-good backup and its timestamp from the restore log, not from a guess.
4. Provision a replacement server matching the old one's spec and OS.
5. Restore configuration and secrets first; encrypted secrets need their keys, which live in the runbook appendix.
6. Restore the data archive and apply any point-in-time follow-ups.
7. Validate: service status, a health endpoint, and a spot-check query against real numbers.
8. Cut traffic to the new host and keep the old host read-only for a grace window.
9. Watch logs and metrics for an hour, and confirm backups are already running on the new host.
10. Write the post-mortem and update the runbook; the incident is not over until the runbook reflects it.

Provisioning the replacement host

The earlier the replacement comes up, the more calmly everything else goes. An API call beats clicking through dashboards while people wait, and provisioning happens in well under a minute:

curl -s -X POST https://api.netbayhosts.in/api/v1/services/purchase   -H "X-API-Key: $API_KEY" -H "X-API-Secret: $API_SECRET"   -d '{"plan":"vps-4gb","os":"ubuntu-24-04"}'

Keep the identical spec in the runbook so step four never involves a pricing debate.

Restore in one boring order

Run the commands in the same order every time, so the runbook stays mechanical: data first, then configuration, then services, then verification.

restic -r sftp:offsite:/backups/app   restore latest --target /var/restore
gunzip -c /var/restore/dumps/appdb.20260822.sql.gz   | psql -U app_user -d appdb
systemctl start app
curl -fsS --max-time 10 http://127.0.0.1/healthz

If the health check fails, do not improvise a fix; the runbook's troubleshooting table is the next page, and skipping it is how one incident becomes two.

The validation script lives in the repo

Keep a committed script that the runbook invokes, so validation never depends on who is on call:

#!/usr/bin/env bash
# /usr/local/bin/recovery_validate.sh
set -euo pipefail
systemctl is-active app
curl -fsS --max-time 10 http://127.0.0.1/_health
psql -U app_user -d appdb -Atc   "SELECT count(*) FROM orders;"

The expected outputs are written next to each line in the runbook, so a wrong number is visible even at 3 a.m.

The metrics that matter

Track three numbers: RTO as measured in drills, restore time for data versus the full stack, and how often the drill needed deviations. If drills routinely take detours, the gaps you are finding are the value of the runbook. Record each drill's wall-clock time next to the runbook revision so improvement is visible.

Practice on schedule, never skip the first hour

Run the first full drill within a month of writing the runbook, before you feel ready — that is exactly when the runbook is wrong. After every drill, update the document with the same commit discipline you give to code, and let the drill log prove the runbook is not fiction.

Takeaway

A runbook turns a disaster into a procedure: declare, scope, provision, restore, validate, cut over, post-mortem. Write the ten steps now, test the restore path on a real API-provisioned instance within 30 days, and keep the metrics honest. Netbay provisions replacement VPS instances in under a minute, so step four of your runbook is never the bottleneck — start the checklist at netbayhosts.in.