Observe Tokens, Latency, and OOM on CPU Inference

CPU inference fails in slow motion, then all at once. Latency climbs as the run queue fills. RSS walks toward MemAvailable. The OOM killer picks the largest process, which is usually the model. If you only watch HTTP 200 rates, you will call the box healthy until it is not. This post is the smallest set of signals that tell you a local encoder and generator are in trouble: tokens, timers, and memory.

One JSON line per request

Emit a structured line when the request ends, including errors. Fields that matter: route, model, prompt_tokens, completion_tokens, encode_ms, retrieve_ms, rerank_ms, generate_ms, total_ms, rss_mb, inflight, cache_hit, status. That is enough to find whether you are tokenizer-bound, scan-bound, or decode-bound.

Count tokens with the real tokenizer when you can. bytes/4 is a fallback for logs when the tokenizer is expensive, not a billing number. If generate_ms / completion_tokens jumps, you are starving cores or swapping. If encode_ms jumps while batch size is constant, you are competing with another PyTorch runtime or with steal.

import json
import time
import resource

def rss_mb():
    usage = resource.getrusage(resource.RUSAGE_SELF)
    return round(usage.ru_maxrss / 1024.0, 1)

def log_infer(event):
    event["rss_mb"] = rss_mb()
    print(json.dumps(event, separators=(",", ":")), flush=True)

t0 = time.perf_counter()
# ... encode, retrieve, generate ...
log_infer({
    "route": "/generate",
    "model": "minilm-plus-local-llm",
    "prompt_tokens": 812,
    "completion_tokens": 140,
    "encode_ms": 28,
    "retrieve_ms": 17,
    "rerank_ms": 94,
    "generate_ms": 2100,
    "total_ms": int((time.perf_counter() - t0) * 1000),
    "inflight": 1,
    "cache_hit": False,
    "status": 200,
})

Ship these lines with journald. Do not invent a metrics mesh on day one. grep and jq on a Lucknow VPS will answer "what happened at 14:02" faster than a dashboard you have not built.

Memory: the OOM killer is a production event

Linux will cache model weights in page cache, which is healthy. Watch MemAvailable, not MemFree. Watch the RSS of the inference process and the RSS of Postgres. Set a systemd MemoryMax on the generator unit below the plan size so systemd stops it before the kernel picks a victim at random. MemoryMax around 60 to 70 percent of RAM on a mixed app+model box is a starting point.

[Service]
MemoryMax=3G
MemoryHigh=2500M
TasksMax=64
OOMScoreAdjust=200
Restart=on-failure
RestartSec=5
Environment=OMP_NUM_THREADS=3

OOMScoreAdjust positive makes the generator more likely to die than sshd. That is correct. Log systemd OOM kills from the journal. If you see them, your batch size, context length, or concurrency is wrong. Cutting max_tokens and inflight is the fix, not hoping High-Speed SSD swap will save you. Swap hides the problem until disk wait tanks generate_ms.

If you run embeddings and generation in one process, a large generate can fragment the allocator and the next embed batch fails. Prefer two units. Embedders are small and should stay up even when generation is cycling.

Latency histograms beat averages

p50 generate_ms can look fine while p95 is 20 seconds because one inflight request queued five others. Histogram in the log by bucketing, or keep three in-memory counters: under 500 ms, under 3 s, over 3 s. Alert on the over bucket, not on mean.

Correlate with vmstat: user, steal, wait. High wait during generate means you are paging. High steal means the hypervisor is busy; on a VPS that is a sizing or neighbor question, not a prompt-engineering question. CPU user at 100 percent with inflight=1 is a healthy decoder. CPU user at 100 percent with inflight=4 and climbing total_ms is a queue.

Expose /healthz that checks: process up, last request under N seconds ago or idle, rss_mb under a watermark, disk for the model path writable. Do not load the model in the health check.

Bind metrics, if you add them later, to localhost. The public HTTPS port already has L3/L4 DDoS filtering. Prometheus on 0.0.0.0 is a free load test for strangers.

When generate_ms is high and rss_mb is flat, you are CPU-bound. When both climb, you are spilling to swap or allocating state you cannot afford. When rss_mb climbs across requests and never returns, you have a leak in the Python worker: graphs, tokenizer caches, or a list of past prompts you forgot to drop. Restarting the unit is a mitigation. Finding the list is the fix. Track rss_mb at request start and end; a 200 MB step on a tiny prompt is the smoking gun.

Takeaway

Log tokens and stage timers, cap MemoryMax on the generator, and treat OOM as a deploy-worthy incident. CPU inference observability is RSS plus a stopwatch. You can spin up an Ubuntu 24.04 instance on Netbay in Lucknow in under 60 seconds and read those JSON lines from journalctl the same afternoon — netbayhosts.in.