Debug a Wedged Python Worker with py-spy and Logs

A wedged worker looks like a live site. systemd is active, nginx returns 502 on some requests and 200 on others, CPU is either pegged or suspiciously idle, and restarting Gunicorn "fixes" it until the next spike. The worker is stuck in a lock, a DNS call, a Postgres wait, or a CPU loop. py-spy reads the Python stacks without stopping the process. journalctl tells you what happened just before. Together they beat a reboot.

This is the one-box incident path for Gunicorn, Uvicorn, and Celery on Ubuntu. You need root or CAP_SYS_PTRACE, the venv, and the patience not to kill -9 first.

Confirm It Is a Worker, Not the Proxy

nginx 502 means the upstream did not answer. That can be a dead bind, a full listen queue, or a worker that accepted the request and never returned. Check the cheap things first.

systemctl is-active app
ss -lptn | grep 8000
curl -sS -m 2 -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8000/healthz
journalctl -u app --since "10 min ago" --no-pager | tail -n 80
ps -o pid,ppid,state,pcpu,rss,cmd -C gunicorn
ps -o pid,ppid,state,pcpu,rss,cmd -C uvicorn

State D is uninterruptible disk wait. State S with high RSS and no logs is often a lock. State R at 100% CPU is a loop. If /healthz is instant and a real endpoint hangs, the hang is in application code or the database, not in the bind.

If ss shows 127.0.0.1:8000 and curl hangs, you have the wedge. If curl is fine and public HTTPS hangs, look at nginx, TLS, or L3/L4 DDoS filtering dropping a path — the Python process is not the patient.

py-spy dump Is the Stack Trace You Want

Install py-spy in a root-accessible way. It attaches from outside the venv; putting it in the app venv is fine too.

sudo /srv/app/.venv/bin/pip install py-spy==0.3.14
PID=$(pgrep -P $(systemctl show -p MainPID --value app) | head -n 1)
sudo py-spy dump --pid $PID
sudo py-spy top --pid $PID
sudo py-spy record -o /tmp/worker.svg --pid $PID --duration 20

dump prints every thread's Python stack at that instant. You are looking for a familiar name: requests.adapters, smtplib, psycopg waiting, a lock.acquire, an accidental while True, or a numpy/pandas C loop that never returns to Python (py-spy will show the last Python frame). top is a live view. record writes a flame graph.

Gunicorn's master is the MainPID. Dumping the master is usually boring. Dump the children. pgrep -P MASTER lists them. Dump two or three; the wedged one has a stack that does not change between dumps. Take two dumps ten seconds apart. If the frames are identical and not in epoll, that worker is stuck.

Uvicorn workers are the same idea: skip the supervisor, dump the children. Celery prefork: dump the child, not beat.

ptrace can fail under some hardening. If py-spy says permission denied, check kernel.yama.ptrace_scope, then sysctl kernel.yama.ptrace_scope=0 for the incident (and put it back). On a typical Ubuntu VPS this is 0 or 1; 1 still allows root.

Logs, Postgres, and Locks

Stacks without logs are half a diagnosis. Correlated timestamps matter.

• journalctl -u app -u app-worker --since "15 min ago"
• sudo -u postgres psql -c "SELECT pid, state, wait_event_type, wait_event, query FROM pg_stat_activity WHERE datname = 'app';"
• ls -l /proc/$PID/fd | wc -l (file descriptor leak)
• cat /proc/$PID/status | grep -E "VmRSS|Threads|State"

idle in transaction plus a py-spy frame in cursor.execute is a leaked Django transaction or a missing connection.close(). Lock wait_event plus a stack in SELECT ... FOR UPDATE is a lock order bug. SMTP connect in the request stack is the missing queue from the Celery post; the worker is not wedged, the architecture is.

Do not enable DEBUG=true on a public box to get a traceback. That ships secrets. py-spy plus the journal is the production debugger.

If RSS grows without bound, dump then restart that child. Gunicorn will spawn a replacement. Killing the master drops everyone; kill a worker PID when you can. Timeout settings exist so this happens automatically: Gunicorn --timeout 30 is a wedge detector you already paid for. If timeouts are 0 or 300, you asked to wait forever.

After the Dump: Fix, Then a Smaller Timeout

A dump is not a fix. The usual fixes:

• Move SMTP and HTTP fan-out to a worker process.
• Close DB sessions in a finally or context manager.
• Pin a requests timeout= on every outbound call; the default is none.
• Stop hashing passwords or resizing images in the request.
• Add statement_timeout so Postgres, not the Python timeout, kills the query.

Then lower the worker timeout so the next wedge is a 504 and a recycled process, not a silent stall. Recycle with --max-requests 1000 --max-requests-jitter 50 to bleed memory leaks.

Intel Xeon Platinum will run a busy loop with enthusiasm. High-Speed SSD will not show up in py-spy if you are stuck in DNS. Lucknow latency to a far API is a timeout waiting to happen; set one. Restarting the VPS is last, not first.

Takeaway

Do not reboot a wedged Python worker until you have two py-spy dumps and a page of journal. The stack names the wait. Timeouts, closed sessions, and a second process for slow I/O are the actual fixes. Restart is a tourniquet.

You can spin up an Ubuntu 24.04 instance on Netbay in under 60 seconds and practice py-spy dump on a quiet box before you need it — netbayhosts.in.