Diagnosing Failed Units: A systemctl Troubleshooting Playbook

Every operator eventually meets the red failed state, usually at the worst possible moment. The good news is that failed units are unusually well instrumented: the exit code, the terminating signal, the exact log window, and the effective configuration are all one command away. This post is the playbook, in the order to run it, so the next failure is a ten-minute fix instead of an evening.

Step 1: Establish the Facts

systemctl --failed
systemctl status myapp.service -l --no-pager
systemctl show myapp.service -p Result -p ExecMainStatus -p ExecMainCode -p NRestarts

The Result property is the single most informative field in systemd troubleshooting. exit-code means the process ran and returned non-zero; signal means it died from a signal such as an OOM kill or a segfault; core-dump means it crashed hard enough to leave a core; timeout means it missed its TimeoutStartSec; start-limit-hit means the restart rate limiter gave up on it. ExecMainStatus carries the numeric exit code or signal number that goes with it.

Step 2: Interpret the Exit Code

Exit codes follow conventions worth memorizing. 1 is a generic error, 2 is usually command-line misuse, 126 means the file exists but is not executable, 127 means the command or interpreter was not found at all, and 128 plus N means the process was killed by signal N — so 137 is the OOM killer's signature (128 plus 9) and 143 is a clean SIGTERM. A 127 in the journal almost always means a wrong ExecStart path, the most common unit-file bug in existence.

Step 3: Read the Log Window, Not the Last Line

journalctl -u myapp.service -b --no-pager -n 200
journalctl -u myapp.service -p err --since today

The mistake everyone makes is reading the final traceback and missing the first error three hundred lines earlier — the dependency timeout, the DNS failure, the permission denial that started the cascade. Read from the first entry after the unit entered activating state and move forward.

Step 4: Run the Usual Suspects Checklist

Most failed units reduce to one of six causes:

• Wrong ExecStart path or missing interpreter: exit 127 in the journal; check what systemctl cat myapp actually points to.
• Permissions: EPERM or permission denied in the log; check User=, Group=, and ownership of every path the service touches, including the working directory.
• Port already bound: address already in use in the log; find the holder with ss -ltnp and decide who owns the port.
• Missing dependency: the service starts before its database or network is actually ready; add After= and Wants= and verify the dependency is itself enabled.
• Sandboxing: a hardening directive added last week now denies a needed path or syscall; look for denials in the journal and review recent drop-ins under /etc/systemd/system.
• Resources: OOM kills (exit 137) or file-descriptor exhaustion; check MemoryMax= and LimitNOFILE= and the kernel log for the kill decision.

Step 5: Validate Configuration Without Running

systemd-analyze verify /etc/systemd/system/myapp.service
systemctl cat myapp.service
ls -l /etc/systemd/system/myapp.service.d/

verify parses the unit and reports syntax errors, unknown directives, and executables referenced by ExecStart= that do not exist — most of row one on the checklist caught before a single start attempt. The other two lines catch the quieter failure mode: a stale drop-in overriding a key you just fixed in the main file. On any machine with history, check the drop-ins before trusting the main unit.

Step 6: Clear the State and Watch It Recover

After fixing, systemctl reset-failed myapp.service clears the failed state so monitoring stops shouting, then systemctl start myapp and follow journalctl -u myapp -f for the first minute. If it fails again, the loop restarts at Step 1 with strictly more information than you had the first time — that convergence is why the playbook works.

Takeaway: Result and ExecMainStatus tell you how it died, the journal window tells you why, and systemd-analyze verify proves the fix. Six steps, five commands, most failures resolved without a search engine.

Bounce a few units on purpose — get a Netbay VPS in under 60 seconds at netbayhosts.in and break something safely.