Reverse Proxy, App, and Database Failure Domains

A failure domain is the set of things that go down together when one thing goes wrong. On a laptop that set is the whole machine. On a well-run VPS it is smaller: the reverse proxy can die without dropping the database, the app can restart without dropping TLS, and a locked table does not have to take nginx with it. Three processes on one kernel still share a disk and a NIC, so this is not isolation in the cluster sense. It is isolation you can actually operate: separate units, separate memory caps, separate logs, and a health check that names the guilty layer.

Name the three domains before you ship

The reverse proxy owns the public socket, certificates, and request routing. The app owns business logic, sessions it should not persist locally, and outbound calls. The database owns durable rows and the disk they live on. If those three share a user, a cgroup, and a log file, they are one domain wearing three hats. The goal is not more moving parts. The goal is that a restart of one does not imply a restart of the others.

Treat each domain as a question you can answer at 2 a.m. Is the edge answering 443? Is the app answering 127.0.0.1:3000? Is Postgres accepting connections on the local socket? Those three probes are the entire incident dashboard for a single-box product. Everything else — CPU graphs, disk graphs, slow-query logs — is detail you open after a probe fails.

Cap memory so one leak cannot eat the node

Linux will happily let a Node process fill RAM and then start killing whatever the OOM killer notices first, which is often Postgres. That is one failure domain pretending to be three. systemd MemoryMax on the app unit is the cheapest fence. Pair it with Restart=on-failure so the leaky process comes back small, and leave Postgres uncapped relative to the app so the killer has a smaller target.

# /etc/systemd/system/app.service.d/limits.conf
[Service]
MemoryMax=768M
TasksMax=256
TimeoutStopSec=20
Restart=on-failure
RestartSec=3

# /etc/systemd/system/nginx.service.d/limits.conf
[Service]
MemoryMax=256M

After a daemon-reload, systemctl show app -p MemoryMax should print the cap. If it prints infinity, the drop-in did not load. Do the same for CPUQuota only if you have a noisy neighbor inside the app (a report job, a thumbnail worker). Do not CPU-cap Postgres on a single-box product unless you have measured that the app is starving it; the database is usually the thing you want to win.

Probe each domain, not the homepage

A curl of the marketing page can return 200 while the database is wedged, because the homepage is static. Probe the three sockets independently and fail the check if any one of them is dark. Keep the script in /usr/local/bin so timers and humans run the same code.

#!/usr/bin/env bash
# /usr/local/bin/stack-health
set -euo pipefail

ss -ltn | grep -q "0.0.0.0:443" || { echo proxy-down; exit 1; }
curl -fsS --max-time 2 http://127.0.0.1:3000/healthz >/dev/null   || { echo app-down; exit 2; }
pg_isready -h 127.0.0.1 -p 5432 -q || { echo db-down; exit 3; }
echo ok

Wire that to a systemd timer every minute and to an external uptime check that hits a dedicated /healthz on the proxy, which itself calls the same three tests. The external check tells you the box is reachable from the internet. The local script tells you which domain failed. Together they stop you from restarting nginx because Postgres ran out of connections.

What to restart, and in what order

Proxy down, app up, database up: reload or restart nginx only. Certificates and upstreams live here. App down, others up: restart the app unit; do not touch Postgres. Database down: stop the app first so it stops holding locks and reconnecting in a loop, bring Postgres back, then start the app. Rebooting the VPS is the last resort, not the first, because a reboot is how you turn three domains back into one. L3/L4 DDoS filtering at the edge of the network does not replace this discipline; it only keeps volumetric noise off the NIC so your own processes are the ones you have to reason about.

You can bring up an Ubuntu 24.04 VPS on Netbay, drop these three units in place, and practice failing each one without the others noticing — netbayhosts.in.