Rate-Limit Your Own Inference API on a VPS

CPU inference is a single queue wearing an HTTP badge. One client that sends 50 concurrent generates will swap the box, trip the OOM killer, and take search down with it. Rate limiting is not a product feature you buy. It is nginx limit_req plus an application token bucket that counts prompt tokens, not only requests. The VPS in Lucknow is finite: Intel Xeon Platinum cores, a RAM cap, High-Speed SSD. Protect those, then talk about fairness.

Limit requests at the proxy, tokens in the app

nginx limit_req is the first gate. It stops connection floods and silly loops. It does not know that one /generate with 8k prompt tokens costs fifty times a /search. Put a tight request limit on /generate and a looser one on /search and /embed. Burst small. Return 429 with Retry-After.

The application then counts tokens. Estimate prompt tokens as bytes/4 if you do not want a tokenizer on the hot path, or run the real tokenizer and cache the count. Each API key gets a bucket: N tokens per minute for generate, M embeddings per minute, C concurrent in-flight generates. Concurrent is the one that saves you. Two long generations on a 4 vCPU plan is a policy decision, not a default of unlimited.

limit_req_zone $binary_remote_addr zone=search:10m rate=5r/s;
limit_req_zone $binary_remote_addr zone=gen:10m rate=20r/m;
limit_req_status 429;

server {
    listen 443 ssl;
    server_name rag.example.com;

    location /search {
        limit_req zone=search burst=10 nodelay;
        proxy_pass http://127.0.0.1:8088;
    }

    location /generate {
        limit_req zone=gen burst=2 nodelay;
        proxy_read_timeout 120s;
        proxy_pass http://127.0.0.1:8088;
    }
}

Keep the inference process on 127.0.0.1. L3/L4 DDoS filtering at the edge drops junk packets. It will not stop a valid API key from stacking generates. That is your bucket.

Token buckets that match CPU cost

A leaky bucket per key is enough. Store remaining tokens and last refill in memory if you have one API process, or in SQLite if you have a few workers. Do not introduce a network cache just for counters on one box.

import time

class TokenBucket:
    def __init__(self, rate_per_min, burst):
        self.rate = rate_per_min / 60.0
        self.burst = burst
        self.tokens = burst
        self.ts = time.monotonic()

    def take(self, n):
        now = time.monotonic()
        self.tokens = min(self.burst, self.tokens + (now - self.ts) * self.rate)
        self.ts = now
        if self.tokens < n:
            return False
        self.tokens -= n
        return True

# one generate at a time per process, second waits or 429
INFLIGHT_MAX = 1

Reject oversize prompts before you tokenize the world. A 100 kB JSON body is not a RAG question. nginx client_max_body_size 32k on /search and a larger but finite cap on /generate. Application-side, max_prompt_tokens is a hard 2048 or 4096 depending on the model.

On 429, send a JSON body with error, limit, and retry_after_ms. Clients that retry immediately are why you have a burst of 2, not 50. Document that. If you issue API keys, put the quota in the key row so you can give a CI job more embed throughput than a browser widget.

Fairness, auth, and failure mode

Rate limit after you authenticate. Limiting by IP only will punish a NAT and ignore a leaked key from elsewhere. Still keep the IP limit as a floor for anonymous search if you offer it. Fail closed on generate when the bucket store is unavailable. Fail open on search if you must, but log it.

Semaphore for inflight generates belongs in the worker, not only in nginx. nginx knows connections, not tokens in the decoder. If RSS hits a watermark, stop taking new generates even if the bucket has tokens. That is load shedding. A 503 with Retry-After is better than the OOM killer picking systemd.

Do not rate limit ingest. Ingest is local, locked, and already serial. Do not put the embedder on the public server block.

Watch 429 count, inflight, and p95 generate latency. If 429s climb while CPU is idle, the limit is too tight. If latency climbs with no 429s, the limit is too loose. Tune against the box you have in Lucknow, not against a paper.

Separate quotas for embed and generate. A docs rebuild that calls embed 20k times should not share a bucket with interactive generate. Issue an ingest key that is only valid on localhost or a Unix socket. Public keys get the nginx limit and the token bucket. Internal timers skip nginx entirely. That split keeps a legitimate re-embed from 429ing the site, and keeps a scraped key from encoding the whole internet through your MiniLM process.

Takeaway

Cap requests at nginx, cap tokens and inflight in the app, and shed when RAM is the bottleneck. Your inference API is a scarce CPU queue. You can spin up an Ubuntu 24.04 instance on Netbay in Lucknow in under 60 seconds and put both gates in front of a localhost worker — netbayhosts.in.