Celery or a Simple Queue When Python Needs Two Processes

A web worker that sends email inside the request is a timeout waiting to happen. SMTP stalls, PDF reports chew CPU, and webhook retries block a Gunicorn slot. The fix is a second process: the HTTP server stays request-shaped, a worker process pops jobs. Celery is the common Python answer. It is not the only one, and on a single VPS it is easy to overbuild.

The rule is simple. If the work can fail independently of the HTTP response, it belongs in another process. If the user must wait for the result, keep it in the request or use a job id plus polling. Do not thread it inside Uvicorn and hope.

When You Actually Need a Queue

You need a second process when any of these are true:

• The work talks to a slow third party (SMTP, SMS, a partner HTTP API).
• The work is CPU heavy (thumbnails, invoices, CSV export).
• The work must retry after the user already got 201 Created.
• The work should outlive a Gunicorn restart.

You do not need Celery to write a file, hash a password, or run a 20 ms SQL insert. Adding Redis plus a broker plus flower plus beat for that is how a 2 GB box dies. A management command on cron is still a second process, and often the right one.

The Smallest Celery Layout

Broker on the same VPS: Redis. Result backend optional; skip it until something reads task results. One worker unit, one web unit, same .env, same venv.

sudo -u app -H /srv/app/.venv/bin/pip install "celery[redis]==5.4.0" redis==5.0.8
# /etc/systemd/system/app-worker.service ExecStart:
/srv/app/.venv/bin/celery -A proj worker --loglevel=INFO --concurrency=2

# proj/celery.py
import os
from celery import Celery

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "proj.settings")
app = Celery("proj")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

@app.task(bind=True, max_retries=5, autoretry_for=(Exception,), retry_backoff=True)
def send_invoice_email(self, invoice_id):
    # load row, render, SMTP; raise to retry
    return invoice_id

concurrency=2 is a cap, not a goal. Each Celery child is a process with its own Django and its own DB connections. Two plus three Gunicorn workers is five Python interpreters. Prefork is the default pool; it is the right pool until you know you need gevent.

Beat is a third process if you have periodic tasks. Do not run beat inside the worker on Celery 5 unless you accept duplicate schedules when you scale workers later. For one box, a separate app-beat.service with celery -A proj beat is clearer. For one nightly report, cron plus a management command is clearer still.

A Simpler Queue When Celery Is Heavy

If the job list is "send this email" and "resize this image", a Redis list plus a tiny loop is enough.

import json, os, time, redis

r = redis.Redis.from_url(os.environ["REDIS_URL"], decode_responses=True)

def enqueue(name, payload):
    r.lpush("jobs", json.dumps({"name": name, "payload": payload}))

def run_worker():
    while True:
        item = r.brpop("jobs", timeout=5)
        if not item:
            continue
        job = json.loads(item[1])
        try:
            dispatch(job["name"], job["payload"])
        except Exception:
            r.lpush("jobs-dead", item[1])
            time.sleep(1)

You lose routing, chords, and a rich retry policy. You gain a worker you can read in one screen. Put run_worker() behind systemd Restart=always, same as Celery. Persistence is Redis AOF or accept that a flush loses jobs; for invoice emails, that is not acceptable, so use Celery with acks_late or store the job row in Postgres first and let the worker pick unfinished rows. A database-as-queue table (id, status, run_at) plus FOR UPDATE SKIP LOCKED is an excellent one-box pattern with no Redis at all.

The second process is the requirement. The brand of queue is a preference.

Isolation and Failure

Never run the worker in a thread of the web process. A Gunicorn HUP will kill in-flight emails. Memory leaks in a report renderer will take down HTTP. Deploy restarts should bounce web and worker independently: migrate, restart web, restart worker.

Watch the math. Redis maxmemory, Postgres connections, and RSS. A worker that imports WeasyPrint can add 200 MB. On 2 GB RAM with Intel Xeon Platinum you still only have one memory budget. High-Speed SSD helps Redis snapshots; it does not create RAM.

If Redis is on the same host, bind 127.0.0.1 and require a password even locally so a leaked web shell does not become an open broker. Lucknow VPS nodes are on a public IPv4; do not publish 6379. L3/L4 DDoS filtering will not save an open Redis.

Takeaway

The second process is mandatory once work can fail after the response. Celery plus Redis is the standard. A Redis list or a SKIP LOCKED table is enough for many one-box apps. Threads inside the web process are not a queue.

You can spin up an Ubuntu 24.04 instance on Netbay in under 60 seconds and run web and worker as two systemd units — netbayhosts.in.