Health Checks and Draining a Local Model Process

A GGUF server is a long-lived process with a huge working set. Restarting it blindly drops in-flight generates and then spends seconds to minutes mmap-ing weights from High-Speed SSD. Health checks tell nginx and systemd whether the process can take a job. Draining tells you when it is safe to replace the binary or the model file. None of this is a managed inference feature. It is a unit, a probe, and a stop sequence on an Intel Xeon Platinum CPU VPS in Lucknow.

What healthy means for CPU inference

Listening on 8080 is not enough. A process can accept TCP while the GGUF is still loading, or while RSS is one prompt away from OOM. Prefer an HTTP probe that the server answers only after the model is mapped. llama-server exposes /health on current builds. If your build does not, probe GET /v1/models and treat a 200 with a model list as ready.

curl -sS -o /dev/null -w "%{http_code}
" http://127.0.0.1:8080/health
curl -sS http://127.0.0.1:8080/v1/models
ps -o pid,rss,stat,cmd -C llama-server

RSS should be stable at idle. If it climbs without traffic, you have a leak or you are looking at the wrong process. STAT D for long stretches during generate can be mmap I/O; on a box that is swapping, drain and shrink context before you call it healthy.

systemd can restart on failure. That is not a drain. A crash loop while you replace a GGUF will hit StartLimit and leave you with nothing listening. Set TimeoutStopSec long enough for a generate to finish, and use ExecStop that first marks the instance down.

Drain: stop new work, then SIGTERM

The order is: nginx out of rotation, wait for inflight, SIGTERM llama-server, wait, SIGKILL only if it ignores you, start the new process, wait for /health, nginx back. On a single VPS, "out of rotation" means returning 503 from nginx for /v1/ while still answering a local health URL.

# /usr/local/sbin/drain-llama
set -e
echo "down" > /var/run/llama-drain
nginx -s reload
for i in 1 2 3 4 5 6 7 8 9 10; do
  if ! pgrep -f llama-server >/dev/null; then
    break
  fi
  sleep 3
done
systemctl stop llama-server
rm -f /var/run/llama-drain
systemctl start llama-server
until curl -fsS http://127.0.0.1:8080/health >/dev/null; do
  sleep 1
done
nginx -s reload

Pair that with an nginx snippet: if the drain file exists, return 503 for /v1/. Clients retry. Your one-slot queue should already bound how long stop can wait. TimeoutStopSec=180 in the systemd unit covers a slow 7B Q4 completion. Killing mid-token is how you corrupt nothing on disk (weights are read-only) but waste the user's turn.

Do not delete the old GGUF until /health is green on the new file. A truncated download is the usual failed swap.

Probes from nginx and from cron

A local cron that curls /health and writes a line to a log is enough. If you front the API with nginx, use a separate location for 127.0.0.1 that does not require TLS client setup. External uptime checks should hit nginx on 443 with the Bearer secret, or you leak an unauthenticated generate path.

Failed probes should not SIGKILL immediately. A generate can block /health on some builds. Distinguish "busy" from "dead". If /health hangs for 30 seconds while a generate is running, lengthen the probe timeout or probe a lighter URL. If RSS is near RAM and /health is slow, you are unhealthy even if HTTP 200 returns.

Failure modes worth scripting

OOM-killer leaves no polite drain. journalctl will show killed process. systemd restarts, mmap happens again, and if the model still cannot fit, you loop. Detect that: if the unit restarted more than twice in five minutes, stop restarting and page yourself. The fix is a smaller GGUF or a remote API, not a tighter RestartSec.

Disk full during a pull is another false healthy. /health may still 200 on the old file while wget of the new file failed. Check ls -l of the GGUF against the expected size before you drain into it.

Takeaway

Probe after the model is mapped, drain nginx first, then stop the unit with a long TimeoutStopSec. Health checks on a CPU GGUF server are about RAM and load state, not just a TCP accept. There is no hidden orchestrator on Netbay; this script is the orchestrator.

You can spin up Ubuntu 24.04 on Netbay in under 60 seconds and rehearse a llama-server drain on Intel Xeon Platinum — netbayhosts.in.