Idempotent Automation: Why Scripts Must Be Re-Runnable

The most common cause of production incidents at 3 AM is not a bad idea — it is a good script that was run twice. A setup script that appends config, creates users, or seeds data is usually written once, tested once, and trusted forever, and then someone re-runs it during an outage and it duplicates lines, errors on already-existing users, or doubles subscriptions. Idempotency is the property that fixes this: an operation is idempotent when running it twice produces the same end state as running it once.

Check-Then-Act Beats Fire-and-Forget

The mechanical fix is to make every step ask before it acts. Consider hardening SSH the classic (fragile) way, then the idempotent way:

# check-then-act style: safe to run again after a crash
if ! grep -q '^MaxTries 3' /etc/ssh/sshd_config; then
  printf 'MaxTries 3\n' | sudo tee -a /etc/ssh/sshd_config
  sudo systemctl reload ssh
fi

Without the grep guard, the same snippet appends a duplicate MaxTries directive on the second run, and the second occurrence wins in configuration files with last-value-wins parsing — meaning rerunning your own "fix" silently re-breaks the setting. The guard turns the script into a no-op when the state already matches.

Prefer Modules and Tools That Converge

The stronger move is to stop writing one-shot logic and hand the task to a tool whose modules are idempotent by design. Ansible modules describe their target state and do nothing when it already holds:

- name: Safe to run any number of times
  hosts: all
  become: true
  tasks:
    - name: Ensure the app user exists
      ansible.builtin.user:
        name: app
        shell: /bin/bash
        groups: sudo
        append: true

    - name: Ensure the nightly cron job exists
      ansible.builtin.cron:
        name: nightly backup
        minute: "0"
        hour: "2"
        job: /usr/local/bin/backup-db.sh

The user module creates the account once and is a no-op on the next pass; append: true adds sudo to existing groups without destroying them. The cron module manages the entry by name, so re-running neither spawns a second backup job nor orphans the old one. A playbook like this is safe to run every night from cron, after every deploy, and in the middle of any incident — which is exactly the schedule trust you want.

Convergence, Not Counting

The useful mental model is convergence: run number one brings things up to date, runs two through a hundred are polite no-ops, and a run after someone scatters the config rebuilds it. A non-idempotent script is indistinguishable, on the second run, from a race condition. Nothing about convergence is automatic — you still have to test the second run — but the workload shifts from "remember every script's history" to "verify a replay is clean", which a one-line dry-run can do.

Takeaway

Audit your existing scripts for the double-run hazard and fix each one the cheap way (guards) or the strong way (converging modules). Then adopt a discipline: every new automation job must pass a clean --check dry-run followed by a clean real replay before it earns a place in cron. Rehearse the whole loop on disposable VPSes — create a scratch instance, break its config, re-run the playbook, and confirm it repairs itself, all free to iterate on netbayhosts.in.