Retry, Timeout, Circuit Breakers for LLM APIs

Every agent on the box shares one failure domain: the remote model API. When that API hangs, a naive worker hangs. When it returns 429, a naive worker retries in a tight loop and makes the 429 worse. When it returns 500 for ten minutes, a naive fleet of cron agents, webhook workers, and triage supervisors all sit on open sockets until systemd kills them out of order.

Timeouts, jittered retries, and a circuit breaker are not ML. They are the same HTTP hygiene you already use for payment providers, applied to a slower, flakier endpoint. Intel Xeon Platinum in Lucknow will happily hold thousands of stuck sockets if you let it. Do not let it.

Timeout is a number, not a feeling

Set a connect timeout and a read timeout. One number named timeout=60 on a client often means "the whole request," but some libraries treat it as only connect. Be explicit. For classification, 20 seconds is generous. For a long summary, 120 seconds. The systemd TimeoutStartSec on a cron agent must be higher than the HTTP timeout plus retries, or the unit will SIGTERM mid-retry and you will double-spend.

# /opt/agents/llmhttp.py
import json, os, random, time, urllib.error, urllib.request

URL = os.environ["LLM_URL"]
KEY = os.environ["LLM_API_KEY"]

def once(payload, read_timeout):
    data = json.dumps(payload).encode()
    req = urllib.request.Request(
        URL,
        data=data,
        method="POST",
        headers={
            "Authorization": "Bearer " + KEY,
            "Content-Type": "application/json",
        },
    )
    with urllib.request.urlopen(req, timeout=read_timeout) as resp:
        return json.loads(resp.read().decode())

urlopen's timeout covers connect and read. That is enough for a start. If you move to a session pool, set both. Never leave the default, which is "wait forever" on some stacks.

Retry only the retryable

Retry 408, 429, 500, 502, 503, 504, and connection errors. Do not retry 400, 401, 403, or 422 — those are your payload or your key. Do not retry a 200 with a JSON schema miss; that is the decoder's one repair, not HTTP.

Use exponential backoff with full jitter: sleep random.uniform(0, min(cap, base * 2 ** attempt)). Full jitter avoids a convoy of cron agents waking at the same millisecond after a provider blip. Cap attempts at 3 for interactive workers and 5 for nightly jobs. After the last miss, raise; do not return a fake object.

# /opt/agents/retry.py
import random, time, urllib.error

RETRY_STATUS = {408, 429, 500, 502, 503, 504}

class CircuitOpen(Exception):
    pass

class Breaker:
    def __init__(self, fail_max=5, cool=30):
        self.fail_max = fail_max
        self.cool = cool
        self.fails = 0
        self.opened_at = 0.0
        self.state = "closed"

    def allow(self):
        if self.state == "open":
            if time.time() - self.opened_at >= self.cool:
                self.state = "half"
                return True
            raise CircuitOpen("llm circuit open")
        return True

    def ok(self):
        self.fails = 0
        self.state = "closed"

    def bad(self):
        self.fails += 1
        if self.fails >= self.fail_max:
            self.state = "open"
            self.opened_at = time.time()

BREAKER = Breaker()

def call_llm(payload, read_timeout=20, attempts=4):
    BREAKER.allow()
    last = None
    for i in range(attempts):
        try:
            out = once(payload, read_timeout)
            BREAKER.ok()
            return out
        except urllib.error.HTTPError as exc:
            last = exc
            if exc.code not in RETRY_STATUS:
                BREAKER.bad()
                raise
        except urllib.error.URLError as exc:
            last = exc
        sleep = min(16.0, 0.4 * (2 ** i))
        time.sleep(random.uniform(0, sleep))
    BREAKER.bad()
    raise last

The breaker is process-local. That is fine for one worker. If you run four workers, they will each need five failures before opening — 20 hammering requests. Share state with a file or Redis if you already have Redis; do not add Redis only for this. A file with opened_at and fails, updated under flock, is enough on one VPS.

What a trip looks like from the rest of the agent

When CircuitOpen is raised, the worker nacks the envelope with reason=provider_down and does not repair-prompt. The queue holds. A human or the next timer retries. Interactive webhook agents should return a canned "provider unavailable, queued" comment rather than stall GitHub.

Log every attempt: status, latency, bytes, breaker state. The JSONL is how you learn that 429s cluster at :00 when everyone's cron fires. RandomizedDelaySec on the timer, from the cron-agent post, is the other half of this fix.

Half-open is a single probe, not a stampede. When cool expires, allow one call. If it succeeds, close. If it fails, open again and reset the cool clock. Do not let four workers each decide they are the probe; the shared opened_at file plus flock is the mutex. Idempotency of the downstream tool still matters: a retried classify is safe, a retried github_comment is not unless you keyed it on delivery id. Put a request id on every HTTP call so the provider's logs and yours can be joined.

Spend and stall are different budgets

A breaker protects latency and error rate. A spend cap, from the cron post, protects money. You want both. A provider that returns 200 slowly will not trip a failure breaker; it will eat TimeoutStartSec. Keep a p95 latency trip as well: if three calls exceed 15 seconds, open the circuit even though they succeeded. Slow is a failure mode for a webhook worker.

Takeaway: explicit timeouts, jittered retries on the retryable set, and a breaker that nacks instead of queuing sockets. Wrap the client once; every agent imports it. Put the wrapper on Ubuntu 24.04 in Lucknow DC01 from Netbay — netbayhosts.in.