Container Health and Self-Healing Beyond Docker Healthchecks

A single HEALTHCHECK in a Dockerfile tells the engine "is the process responding", but real availability needs more nuance. Your service can respond on a port while it is still warming up, or while it holds a broken dependency, or while it is degrading but not dead. Robust self-healing separates those cases: it restarts what is broken, drains what is temporarily unavailable, and never restarts what is simply slow.

This post builds a layered health model that works on a single host or a Swarm service.

Readiness vs Liveness: Two Different Questions

A liveness probe answers "is the process stuck and must be restarted?" A readiness probe answers "is this instance able to serve traffic right now?" In Swarm, the healthcheck gate for rolling updates acts as readiness, while the restart policy handles liveness. Mixing them up causes restarts of temporarily-busy services or, worse, rolling updates that consider a barely-started task healthy.

deploy:
  replicas: 3
  update_config:
    order: start-first
    monitor: 30s
  restart_policy:
    condition: any
    delay: 5s
    max_attempts: 3
  healthcheck:
    test: ["CMD-SHELL", "curl -fsS http://localhost/ready || exit 1"]
    interval: 10s
    timeout: 3s
    retries: 2
    start_period: 15s

The healthcheck above is a readiness-style probe tied into the update gate; keep a separate, stricter liveness condition inside the app's own process.

The three layers of a healthy small stack.

Give Systemd a Watchdog at the Host Level

The container engine restarts crashed containers, but a process that hangs while the host needs to notice should be caught even lower, by the service supervisor. systemd's WatchdogSec powers a software watchdog: the service must ping sd_notify or the unit is considered failed and restarted.

[Unit]
Description=app container wrapper

[Service]
ExecStart=/usr/local/bin/run-app
Restart=always
RestartSec=5
WatchdogSec=20

[Install]
WantedBy=multi-user.target

From inside the container, send the ready and watchdog signals so the wrapper can react:

#!/bin/sh
exec /usr/bin/python3 /app/server.py &
echo "READY=1" > /dev/initctl
# a tiny loop pings the watchdog; if the app hangs, the unit restarts it
while true; do
  printf "WATCHDOG=1
" | nc -U /run/systemd/notify
  sleep 10
done

Alternatively, rely on sd_notify from systemd's notify socket inside a properly configured container.

Layered Health in a Small Stack

Combine the layers for the most honest view: kernel and host metrics (memory, load) at the bottom, the supervisor's restart policy at the service layer, and an application health endpoint at the top that checks database connectivity, not just HTTP 200.

curl -s http://localhost:8080/health | jq .
# {"status":"ok","checks":{"db":"ok","cache":"ok"},"uptime_s":184}

When the database is briefly unavailable during a deploy, you want the app to report not-ready (traffic paused, no restart), not to be killed.

What Not to Overengineer

Do not add a probe for everything or set tiny restart timeouts, or your "self-healing" starts thrashing. Restart on failure, gate traffic on readiness, use a watchdog for hangs, and let one layer own each responsibility. Two replicas + a readiness gate already survive most single-container issues on one node.

Takeaway

True self-healing is layered: liveness restarts what is dead, readiness drains what is unavailable, and a host watchdog catches the hangs the engine misses. Apply this stack of probes to the containers you run on a Netbay VPS — spin one up at netbayhosts.in and watch a dead database cause graceful drain instead of a crash loop.