Background Workers vs the HTTP Request Path

The request path has a clock the user can hear. nginx proxy_read_timeout, the browser, the mobile client, and the load balancer all agree that a few seconds is late. Background work has a different clock: minutes are acceptable, retries are required, and a crash must not lose the job. Mixing those clocks in one process is the most common operational architecture mistake on a VPS. The handler sends a mail, the SMTP handshake stalls, the worker slot is held, and checkout p95 follows the mail provider's outage.

The Request Path Is a Short List

If it is not required to produce the HTTP response the client will read, it does not belong in the handler. Required: authz, validate, write the source of truth, return the representation or 202. Not required: email, webhooks, PDF, image variants, search index, third-party GST, Slack, analytics. Those are jobs. The handler writes an outbox row or LPUSHes after COMMIT and returns. Status 202 with a Location of a job resource is a better contract than a 12-second 200.

Timeouts make the rule enforceable. Set the application server timeout below nginx, and nginx below the client. A handler that cannot finish in that window is a bug, not a capacity problem. Adding more processes will copy the stall.

var http = require('http');
var server = http.createServer(function (req, res) {
  if (req.url === '/v1/orders' && req.method === 'POST') {
    readBody(req, function (err, body) {
      if (err) { res.writeHead(400); res.end(); return; }
      db.createOrder(body, function (e, order) {
        if (e) { res.writeHead(500); res.end(); return; }
        queue.enqueue({ type: 'order.created', id: order.id });
        res.writeHead(202, { 'Content-Type': 'application/json', 'Location': '/v1/jobs/' + order.id });
        res.end(JSON.stringify({ id: order.id, status: 'accepted' }));
      });
    });
  }
});
server.setTimeout(8000);
server.listen(8000);

The enqueue call must be fast: Redis LPUSH on the local network, not a partner HTTP POST. If enqueue itself can block, you have only moved the problem.

Workers Own Failure

A worker is allowed to die. systemd restarts it. A handler that dies mid-request leaves a client with a reset connection and no idea whether the write happened. Workers retry with jitter. They dead-letter. They record processed event ids. They can run on a different VPS with a different memory limit so a PDF library leak does not take down the API.

# /etc/systemd/system/app-worker.service
[Service]
User=worker
EnvironmentFile=/etc/app/worker.env
ExecStart=/usr/bin/node dist/worker.js
Restart=on-failure
RestartSec=3
Nice=5
MemoryMax=512M
WatchdogSec=60

Nice=5 keeps a CPU-heavy renderer from starving the API if you have not split nodes yet. MemoryMax turns a leak into a restart instead of an OOM of the whole guest. WatchdogSec requires the worker to ping systemd; a deadlock then looks like a failure, which is what it is. When those limits fight the API on the same box, move the unit to its own VPS. That is the split the queue-age SLO will also demand.

Concurrency is a worker setting, not an HTTP one. A pool of 4 invoice workers is a statement about the GST portal. Twenty gunicorn workers calling that portal from the request path is a statement that you have not decided.

Idempotency on Both Sides

Clients retry POST when the connection resets. Without an Idempotency-Key, you will double-create. The request path must treat that key as a unique constraint. Workers must treat event ids the same way. These are different keys. The client's key protects the HTTP contract. The event id protects the side effect. Collapsing them into one string is how a retry of the HTTP call skips a legitimate second job.

Healthchecks must not run worker code. /healthz is process up, database ping, Redis ping. It is not "render a test invoice". A healthcheck that does real work will amplify load during an outage, which is the opposite of a probe.

Sync work still happens on the request path when the user is staring at the result: login, quote, stock reservation you must confirm before 200. The test is whether a 30-second partner timeout should block the response. If yes, it is a worker. If the browser tab cannot proceed without it, keep it in the handler and set a tighter timeout than the partner's default, then fail clearly.

Takeaway

Two clocks, two units, two failure stories. The request path stays under a hard timeout and returns 202. Workers retry on their own node with their own secrets and their own memory cap. The queue is the bridge, not a thread pool inside gunicorn.

Run the API and the worker as separate systemd units on Netbay VPS nodes in Lucknow (DC01) and keep the 8-second path actually 8 seconds — netbayhosts.in.