Restart Policies in systemd: Building Self-Healing Services

A service that pages you because of one transient crash is a policy failure, not an operations failure. systemd watches every process it starts and can revive it automatically with a backoff delay — but only if you configure a restart policy deliberately. The difference between an on-call nightmare and a self-healing service is about five lines of unit file.

The Restart Values That Matter

Restart= decides when systemd starts the process again after it exits. Five values cover almost every real case.

• no (the default): never restart. The wrong choice for anything user-facing.
• on-failure: restart when the process exits non-zero, is killed by a signal, or times out. The right default for daemons.
• always: restart on any exit, clean or not. Right for workers that should literally never stop.
• on-abnormal: restart only on signals, core dumps, and timeouts, not on clean non-zero exits. Useful when the application uses exit codes for control flow.
• on-watchdog: restart if the service stops answering its watchdog, described below.

[Service]
Restart=on-failure
RestartSec=5s

RestartSec= adds a delay between death and revival, which stops a crash loop from pinning the CPU and gives a failing dependency time to come back. Note that Restart= never fights you: if you run systemctl stop yourself, systemd stays stopped regardless of policy.

Rate Limiting: The Guardrail Under the Guardrail

Unbounded restart is its own failure mode, so systemd rate-limits by default: if the unit starts more than 5 times within 10 seconds, it stops retrying and enters the failed state. Tune that explicitly rather than discovering it in production.

[Service]
Restart=on-failure
RestartSec=5s
StartLimitIntervalSec=300
StartLimitBurst=10

This allows ten attempts across five minutes before giving up — enough to ride out a restarting database, few enough that something truly broken gets noticed. When the limit trips, the unit's Result becomes start-limit-hit, a distinct and greppable signature in journalctl. Recent systemd versions also add smooth backoff for free: RestartSteps= and RestartMaxDelaySec= grow the delay exponentially with each consecutive failure, so a crashing unit waits 5 seconds, then 10, then 20, instead of hammering the scheduler at a fixed interval.

Teaching systemd What Success Means

Applications often use exit codes in ways that confuse a naive policy. Two directives fix the mapping.

[Service]
SuccessExitStatus=143
RestartPreventExitStatus=78

SuccessExitStatus=143 tells systemd that exit 143 — 128 plus 15, what a process reports after a normal SIGTERM — counts as clean, which stops spurious restarts when something sends the process a routine termination signal. RestartPreventExitStatus=78 is the opposite move: 78 is the conventional exit code for a configuration error, and restarting a process with a broken config is pure noise. This directive says do not bother.

Watchdogs: Restarting Processes That Do Not Exit

The nastiest failures are hangs: the process is alive, holding the port, and serving nothing but timeouts. Restart=on-failure never fires because nothing ever exits. The watchdog closes that gap. Set WatchdogSec=30s together with Type=notify, and the service must ping systemd with WATCHDOG=1 more often than every 30 seconds; the sd_notify library call does this in a few lines of application code. A wedged process stops pinging, systemd kills it, and the normal restart policy takes over from there.

Restarts Are Not Monitoring

One honest caveat: automatic restarts mask symptoms. A service that quietly restarts itself every hour will never page you, even though something is genuinely wrong. Watch the restart count with systemctl show myapp -p NRestarts and alert on the rate, not just on the failed state.

Takeaway: Restart=on-failure with a sensible RestartSec, explicit rate limits, and a watchdog for hangs is the entire self-healing recipe — five lines that turn overnight pages into journal entries nobody has to read.

Deploy something crashable and practice the recovery: get a Netbay VPS online in under 60 seconds at netbayhosts.in.