Queue-Based Decoupling With Redis on Linux VPS

The request path is a terrible place to send email, render invoices, call a slow partner API, or resize images. Those jobs take seconds, fail in ways HTTP cannot represent, and pile up when a downstream is sad. Queue-based decoupling is the operational pattern that keeps the API honest: accept the work, persist a job, return 202, and let a worker that is not bound to a client timeout finish it. Redis lists are enough for a surprising amount of this. You do not need a streaming product to stop blocking nginx workers.

Why the Queue Exists

A queue is not a database and not a log. It is a buffer with a consumer. The producer is your API process. The consumer is a systemd service that can crash, restart, and resume without dropping the HTTP listener. The contract is simple: if the job is on the list, it will be attempted; if it is not on the list, it never happened. That contract is why you push the job in the same process that committed the user-visible write, or you use an outbox. This post is the queue side.

Put Redis on its own VPS when the web nodes and the workers would otherwise fight for memory. Bind it to a private address. Set maxmemory and a policy that does not evict the queue by accident. A cache that forgets keys is fine. A queue that forgets jobs is data loss.

# redis.conf on the queue node (not 0.0.0.0)
bind 10.0.0.12
protected-mode yes
port 6379
maxmemory 512mb
maxmemory-policy noeviction
save 60 1000
appendonly yes
appendfsync everysec

noeviction is the important line. LRU on a queue node will drop the oldest jobs under memory pressure, which looks like a successful enqueue from the API's point of view.

Lists, Reliability, and the Stolen Job

LPUSH plus BRPOP is the happy path and the wrong path under crashes. BRPOP removes the item before the worker finishes. If the process dies after pop and before ack, the job vanishes. The operational fix is a working list: BRPOPLPUSH (or BLMOVE) from the main queue onto a per-worker processing list, do the work, then LREM the processing list. A sweeper requeues items that sit too long.

import json, time, redis

r = redis.Redis(host='10.0.0.12', port=6379, decode_responses=True)
QUEUE = 'jobs:invoice'
WORKING = 'jobs:invoice:working:worker-1'
STALE_SEC = 300

def enqueue(payload):
    r.lpush(QUEUE, json.dumps(payload))

def take_job():
    raw = r.brpoplpush(QUEUE, WORKING, timeout=5)
    if not raw:
        return None
    r.hset('jobs:started', raw, str(int(time.time())))
    return json.loads(raw)

def ack(payload):
    raw = json.dumps(payload)
    r.lrem(WORKING, 1, raw)
    r.hdel('jobs:started', raw)

def reclaim_stale():
    now = int(time.time())
    for raw in r.lrange(WORKING, 0, -1):
        started = int(r.hget('jobs:started', raw) or 0)
        if started and now - started > STALE_SEC:
            r.lrem(WORKING, 1, raw)
            r.lpush(QUEUE, raw)

Retries need a budget. Infinite retries turn a poison payload into a hot loop that burns CPU on Intel Xeon Platinum cores you are paying for. After N failures, LPUSH onto a dead-letter list and page a human. Record the exception, the payload id, and the attempt count. Do not log secrets that were inside the job.

Topology: API Nodes, Queue Node, Worker Nodes

Keep three roles even if two of them share a box at first. The API node enqueues and returns. The queue node is Redis with persistence on High-Speed SSD. The worker node pulls jobs and talks to the world. When you split them later, the code does not change; only the hostnames in EnvironmentFile do. Workers should be allowed to scale independently of web replicas. A burst of signups should add worker processes, not double your gunicorn count.

Do not run the worker inside the web process pool. A blocked invoice renderer will steal a request slot. systemd units with Restart=on-failure and a memory limit keep a runaway job from taking the node with it.

What to Measure

Queue depth is the first number. If LLEN grows while workers are alive, you are under-provisioned or the partner is slow. Age of the oldest job is the second. Worker heartbeat is the third: a unit that is running but not popping is worse than a dead unit, because systemd thinks it is healthy. Export these as gauges from a tiny sidecar or from the worker itself. Alert on depth and age, not on CPU, when the problem is a blocked SMTP relay.

Backpressure belongs at enqueue time. If the list is already 50,000 deep, reject new work with 503 and Retry-After rather than taking money you cannot process. A queue is not infinite RAM. High-Speed SSD AOF helps durability; it does not help a worker that cannot keep up.

Takeaway

Redis lists plus a working list and a dead letter are an architecture, not a library. Keep the HTTP process idle after enqueue, pin Redis with noeviction, and let workers fail independently. When the partner is down, your API can still say yes to the user and no to unbounded memory.

You can run the API, Redis, and workers on separate Netbay VPS nodes in Lucknow (DC01) with pay-as-you-go INR billing when traffic actually needs the split — netbayhosts.in.