Rollback Strategies That Are Tested, Not Improvised

Every team plans a rollout; almost nobody rehearses the rollback. Then a bad release hits, and the on-call person improvises under pressure: reverse the migration, flip an environment variable, hope. Real rollback readiness is a tested, scripted, rehearsed capability — and CI is the place to keep it honest.

Decide your rollback shape in advance

Before you can test a rollback, decide what it means for your stack. Three common shapes:

• Revert the deployment to the previous artifact (immutable, instant in blue-green).
• Roll forward — fix forward and deploy a new version (best for schema changes that cannot be unmade).
• Data reconciliation — handle partially applied migrations or writes.

Different failures call for different shapes, so the decision is a matrix, not a single button. A pure code typo wants a revert; a destructive migration wants roll-forward plus repair. Write the decision down as a runbook mapping symptom to action.

The simplest, most rehearsable rollback for immutable-artifact teams is reverting to the prior tag, because the old artifact is still in the registry:

# revert running environment to the last known-good tag
PREV=$(cat .releases/last-good.txt)
./scripts/deploy --tag "$PREV"
./scripts/smoke --env production

The command fails loudly if the old tag is missing, so you discover untested rollback paths in rehearsal, not in an incident.

Test rollback in CI on a staging environment

A rollback that only lives in docs has never run. The discipline is a CI pipeline that performs a full release-and-rollback cycle on a staging environment on a schedule. It deploys a new build, then runs the rollback script, then verifies the service is healthy again. If any step breaks, the pipeline is red — which is exactly what you want to discover before a real incident.

name: rollback-drill
on:
  schedule:
    - cron: "0 3 * * 1"
jobs:
  drill:
    runs-on: ubuntu-latest
    steps:
      - run: ./scripts/deploy --env staging
      - run: ./scripts/smoke --env staging
      - run: ./scripts/rollback --env staging
      - run: ./scripts/smoke --env staging  # healthy again?

Run it weekly. A drill that has been green for months is gold dust; a drill that has never been run is fiction.

Rehearse failure injection, not just the happy path

A drill that only ever rolls back a release that was fine is testing plumbing, not judgment. Where possible, inject a deliberate fault — a deliberately broken canary, a flag expected to misbehave — so the team practices recognizing a bad release and pulling it. This closes the loop between detection and response: the same metrics that should flag a canary as unhealthy are the ones the rollback trigger watches.

Log the rollback as a first-class event

Treat a rollback like the release it reverses: record it. A rollback log with timestamp, previous and current versions, trigger reason, and outcome turns an incident into a learning artifact. It also gives you a historical view of release quality — the pipeline data post in this series digs into that — and makes the messy reality of your release process visible rather than hidden in a war-room chat.

The human decision layer

Scripts cannot decide everything; at the edges a human chooses whether to revert or roll forward based on the shape of the data change. The discipline is to keep that decision minimal — a short matrix, not an open-ended debate — and to rehearse it so the on-call person greets an incident with a prepared motion instead of a blank screen.

Takeaway: a rollback you have never run is a hope, not a plan. Script the shapes, embed a weekly drill in CI, and rehearse fault injection so a bad release is undone in minutes with confidence. Keep your environments quick to stand up on a Netbay VPS from Lucknow DC01 — instant, reprovisionable, and ready at netbayhosts.in.