Node Health Checks That Fail Fast When Wedged

A wedged Node process still has a pid. systemd thinks it is active. nginx still proxy_passes to it. Clients time out. The failure mode is silence, which is worse than a crash, because a crash restarts and silence sits there until a human notices. Health checks exist to turn silence into a failed unit. This post wires three layers on Ubuntu: a cheap HTTP /health, a systemd watchdog that expects a notify ping, and a boot path that exits non-zero if a required dependency is missing.

Failing fast means the unit never reports ready when it cannot serve. It does not mean the health route runs a five-table join.

/health is a liveness probe, not a dashboard

The route must run on the same event loop as the rest of the app. If the loop is blocked, the route does not answer and the checker fails. That is the feature. Keep the handler tiny: return 200 and a short body when the process can accept work. Return 503 when a required dependency is known-down. Do not return 200 because the process exists.

const http = require('http');
const net = require('net');
let ready = false;
let dbOk = true;
function pingDb(cb) {
  const sock = net.connect({ host: '127.0.0.1', port: 5432 });
  const t = setTimeout(function () { sock.destroy(); cb(false); }, 400);
  sock.on('connect', function () { clearTimeout(t); sock.end(); cb(true); });
  sock.on('error', function () { clearTimeout(t); cb(false); });
}
const server = http.createServer(function (req, res) {
  if (req.url === '/health') {
    const ok = ready && dbOk;
    res.writeHead(ok ? 200 : 503, { 'Content-Type': 'text/plain' });
    res.end(ok ? 'ok\n' : 'fail\n');
    return;
  }
  res.end('app\n');
});
server.listen(3000, '127.0.0.1', function () {
  pingDb(function (ok) {
    if (!ok) {
      console.error('database unreachable at boot');
      process.exit(1);
    }
    dbOk = true;
    ready = true;
  });
});
setInterval(function () {
  pingDb(function (ok) { dbOk = ok; });
}, 5000).unref();

The boot path exits 1 if Postgres is down. systemd restarts with Restart=on-failure, and you get a failed unit after StartLimitBurst instead of an app that accepts HTTP and 500s every query. The interval ping updates dbOk so /health can go 503 without exiting; a blip should not SIGKILL the process. A missing database at start is a config error. A missing database at minute 40 might be a maintenance window. Those are different.

nginx can proxy /health to the same upstream or you can curl loopback from a timer. Do not expose /health with stack traces or config dumps. L3/L4 DDoS filtering will not stop someone from scraping an open debug route.

WatchdogSec catches a frozen event loop

Type=notify plus WatchdogSec asks the process to ping systemd on an interval. If the ping stops, systemd treats the unit as failed and kills it. A busy CPU or a blocked loop cannot JSON.stringify its way out; it simply misses the deadline. That is the check /health cannot run when the loop is stuck, because /health is on the same loop.

const fs = require('fs');
const path = process.env.NOTIFY_SOCKET;
function sdNotify(msg) {
  if (!path) return;
  const sockPath = path[0] === '@' ? '\0' + path.slice(1) : path;
  const client = require('dgram').createSocket('unix_dgram');
  const buf = Buffer.from(msg);
  client.send(buf, 0, buf.length, sockPath, function () { client.close(); });
}
sdNotify('READY=1\n');
const wdUsec = Number(process.env.WATCHDOG_USEC || 0);
if (wdUsec > 0) {
  const every = Math.max(1000, Math.floor(wdUsec / 2000));
  setInterval(function () { sdNotify('WATCHDOG=1\n'); }, every).unref();
}

WATCHDOG_USEC is microseconds. Ping at half that. READY=1 is what Type=notify waits for before it considers the service started. If you never send READY, systemd hits TimeoutStartSec and kills you. That is the correct outcome for an app that hangs during listen().

[Service]
Type=notify
NotifyAccess=main
WatchdogSec=30
TimeoutStartSec=20
TimeoutStopSec=20
ExecStart=/usr/bin/node /srv/app/server.js
ExecStartPost=/usr/bin/curl -fsS --max-time 3 http://127.0.0.1:3000/health
Restart=on-failure
RestartSec=2
StartLimitBurst=5
StartLimitIntervalSec=60

ExecStartPost runs after READY. curl -fsS fails the start if /health is not 200. WatchdogSec=30 is long enough for GC on a 2 GB box and short enough that a wedged loop is gone in under a minute. StartLimitBurst stops a crash loop from becoming a CPU spin.

What not to check

Do not call an upstream payment API from /health. Do not compile templates. Do not allocate a heap snapshot. A probe that is expensive becomes the outage. Ready versus live: live means the process should stay up, ready means nginx may send it traffic. systemd watchdog is liveness. /health can encode readiness with 503. Mixing them into one slow function is how you page yourself with false positives.

When the unit fails, journalctl -u nodeapp.service -b tells you whether it was a boot ping, a watchdog, or an exit(1). Fix that cause. Do not raise WatchdogSec to 5 minutes to hide a leak.

The takeaway: crash at boot if you cannot serve, answer /health cheaply, and let systemd abort a process that can no longer ping. A pid is not a healthy app.

Add the watchdog on a Netbay Ubuntu 24.04 VPS and wedge the loop on purpose once, so you trust the restart — netbayhosts.in.