Queue LLM Inference Jobs with Redis on Ubuntu

A chat request that calls a remote model inside the HTTP handler ties up a worker for the whole completion. Under a burst you run out of workers, nginx returns 502, and clients retry, which doubles the burst. Inference is a job. The HTTP path should accept a payload, enqueue it, and either wait on a result key or return a job id. Redis on localhost is enough for a single VPS. You do not need a cluster, and you do not need a managed queue product.

Bind Redis to 127.0.0.1. Give it a password or ACL. Persistence can be RDB on High-Speed SSD; the jobs are not your source of truth, the result store is. Intel Xeon Platinum cores in Lucknow DC01 will run both the API and a couple of workers if you cap concurrency. L3/L4 DDoS filtering stops junk on :443. It will not stop you from enqueueing 10,000 completions against a vendor rate limit.

Why the request path is the wrong place

Remote APIs have latency measured in seconds. Local 7B decode on CPU is slower still. Node and gunicorn workers are counted in the low tens on a small plan. One slow completion per worker and the accept queue fills. Health checks fail. systemd restarts the unit. You now have two copies draining the same in-memory work.

A Redis list makes the work durable across a restart of the HTTP process. BRPOP in the worker is blocking and cheap. LPUSH from the API is a few milliseconds. You can scale workers independently of HTTP. You can rate-limit by not starting the fifth worker. You can dead-letter jobs that fail three times instead of retrying in a request loop that the user already abandoned.

Idempotency matters. Clients retry. Use a job id from the client or a hash of (user, prompt, nonce). SET job:id NX before LPUSH. If the key exists, return the existing job. That is how you stop a double-click from billing twice.

API enqueue, worker dequeue

Keep Redis on the same host. The HTTP unit and the worker unit both read EnvironmentFile for the vendor key. Redis itself should not hold the vendor key.

sudo apt-get update
sudo apt-get install -y redis-server
sudo sed -i 's/^bind .*/bind 127.0.0.1/' /etc/redis/redis.conf
sudo systemctl restart redis-server
redis-cli ping

const Redis = require('ioredis');
const redis = new Redis({ host: '127.0.0.1', port: 6379 });

async function enqueue(job) {
  const id = job.id;
  const key = 'job:' + id;
  const created = await redis.set(key, JSON.stringify({ status: 'queued', prompt: job.prompt }), 'EX', 3600, 'NX');
  if (!created) {
    return { id: id, deduped: true };
  }
  await redis.lpush('queue:infer', id);
  return { id: id, deduped: false };
}

async function workerLoop() {
  for (;;) {
    const popped = await redis.brpop('queue:infer', 5);
    if (!popped) continue;
    const id = popped[1];
    const raw = await redis.get('job:' + id);
    if (!raw) continue;
    const job = JSON.parse(raw);
    try {
      const text = await callModel(process.env.CHEAP_MODEL, job.prompt, { maxTokens: 512 });
      await redis.set('job:' + id, JSON.stringify({ status: 'done', text: text }), 'EX', 3600);
    } catch (err) {
      const n = await redis.incr('job:' + id + ':fails');
      if (n >= 3) {
        await redis.lpush('queue:infer:dead', id);
        await redis.set('job:' + id, JSON.stringify({ status: 'dead' }), 'EX', 86400);
      } else {
        await redis.lpush('queue:infer', id);
      }
    }
  }
}

callModel is the same helper as the router post. The HTTP handler returns 202 with the id for async UIs, or polls GET /jobs/:id every 300 ms for a simple page. Streaming can still exist for interactive chat; the queue is for batch, tools, and anything that must survive a restart.

Concurrency is a worker count, not a hope

One worker per in-flight vendor request is the mental model. If the vendor allows 5 parallel requests on your key, run 4 workers. If you also run a local 7B on CPU, run 1 local worker, because decode is memory-bandwidth bound and two copies thrash. systemd templates make this obvious: ai-worker@1, ai-worker@2.

[Unit]
Description=AI inference worker %i
After=network-online.target redis-server.service
Wants=network-online.target

[Service]
Type=simple
User=aiapp
Group=aiapp
EnvironmentFile=/etc/ai-app.env
WorkingDirectory=/srv/ai-app
ExecStart=/usr/bin/node /srv/ai-app/worker.js
Restart=on-failure
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/srv/ai-app/var

[Install]
WantedBy=multi-user.target

Enable two instances, not twenty. Watch queue:infer length with LLEN. If it grows during the day and never drains, you need more workers, a cheaper model, or a 429 at enqueue time. Do not grow the list forever. Cap LPUSH by checking LLEN and rejecting with 429 when the queue is deeper than a few minutes of work.

What belongs in Redis, what does not

Put job ids, status, and short results in Redis. Do not put 50 MB of retrieved context in the list. Put a pointer to an on-disk file under /srv/ai-app/var. Do not use Redis as your only transcript store; expire job keys in an hour and write durable rows to Postgres if you need history.

When Redis restarts, RDB should have the list if you have persistence on. Jobs in flight at the crash may run twice. The NX job key plus a status of done is the guard: the worker should skip ids already marked done. That is the whole exactly-once story you will get on one box. It is enough.

The takeaway: enqueue inference, bound the workers, dead-letter failures, and keep Redis on loopback. The HTTP process should be fast even when the model is not.

You can spin up an Ubuntu 24.04 instance on Netbay in under 60 seconds and follow along — netbayhosts.in.