Monitoring Proxy Health and Connection Queues

A proxy is a single choke point, which makes it the perfect place to measure the health of the whole stack. Connection counts, queued requests, upstream failures, and timeout rates all surface there first. You rarely need a heavy monitoring platform for one or two servers — Nginx and HAProxy both ship counters you can query with a single HTTP request, and the dashboards on top are a formality.

The numbers that matter

• **Active connections and their split** between reading, writing, and waiting tells you whether the proxy or the backend is the bottleneck.
• **Queued connections** in HAProxy (qcur) are the leading indicator of a slow backend: requests lining up faster than workers drain them.
• **Upstream failure counters** turn "the site felt slow" into "backend B failed 40 checks in the last hour."
• **Timeout rate** is what you expect from tuned proxies going wrong: if read timeouts climb, response times have drifted past your configured ceiling.

Nginx stub_status on a private port

Nginx's status module exports live counters over HTTP. Bind it to localhost so the world cannot scrape it.

server {
    listen 127.0.0.1:8080;
    server_name metrics.local;

    location /stub_status {
        stub_status;
        allow 127.0.0.1;
        deny all;
    }
}

curl -s http://127.0.0.1:8080/stub_status

The output is a few lines of plain numbers: active connections, the requested-to-handled totals, and the current reading, writing, and waiting counts. Waiting is the number worth watching — connections holding a socket while the backend thinks, which grows exactly when the upstream starts struggling.

HAProxy stats and the admin socket

HAProxy exposes the same idea through a statistics page and a socket you can also drive with commands.

global
    stats socket /run/haproxy/admin.sock mode 660 level admin

frontend ft_web
    bind :443 ssl crt /etc/haproxy/certs.pem
    default_backend bk_app

backend bk_app
    server app1 127.0.0.1:3000 check inter 2s
    server app2 127.0.0.1:3001 check inter 2s

echo "show info" | socat /run/haproxy/admin.sock stdio
echo "show stat" | socat /run/haproxy/admin.sock stdio

show info covers the process: uptime, total connections, connection rates, and the memory used. show stat is the per-server table; the columns that deserve dashboards are qcur for queued requests, scur for active sessions, and the status column carried earlier in this series.

Turning numbers into a warning

Raw counters are not alerts; derived rates are. Poll every thirty seconds and watch the deltas: stale too-slow readings are the difference between a ticket and an outage.

while true; do
  curl -s http://127.0.0.1:8080/stub_status
  sleep 30
done

Use a simple cron job or systemd timer that fetches the counters into a rolling buffer, then set three thresholds to start alerting on: active connections sustained above the profile peak, waiting connections above a modest fraction of active, and any sequence of upstream 5xx answers lasting more than a minute. Each threshold maps to an action: add capacity, look at the slowest route, or pull a backend before users notice.

Takeaway

The proxy is the cheapest health endpoint you have: it sees every request, every queue, and every dead backend before anyone else does. Capture the stub_status or HAProxy stats every thirty seconds, alert on sustained deviations rather than single spikes, and you will beat most incidents to the punch. Standing this up on a Netbay VPS is minutes of work and nothing extra to install — the counters are built in at netbayhosts.in.