LLM Concurrency: One User at a Time vs a Queue

CPU inference is a single expensive worker, not a dozen cheap ones. llama-server can open parallel slots, and each slot wants its own KV cache. On an 8 GB Intel Xeon Platinum VPS that second slot is how a comfortable 7B Q4 process becomes an OOM. The honest architecture is one in-flight generate and a queue for everyone else. Netbay does not offer a managed inference scheduler. You own the queue, the timeout, and the 503.

Why parallel slots are not free threads

A web app can handle hundreds of JSON requests on four vCPUs because each request is microseconds of CPU. A 7B generate is seconds to tens of seconds and saturates memory bandwidth. Two generates at once often run slower than one-then-the-other, and they need roughly two KV caches. --parallel 2 is not a 2x feature on this hardware.

Keep --parallel 1 on llama-server and OLLAMA_NUM_PARALLEL=1 on Ollama. If a second HTTP request arrives while a generate is running, the server may block, queue internally, or overlap if you raised slots. Do not guess. Prove it with two curls started together and watch RSS.

./build/bin/llama-server   -m $HOME/models/model-q4_k_m.gguf   --host 127.0.0.1 --port 8080   -t 4 -c 2048 --parallel 1

If you experiment with --parallel 2, take an RSS snapshot at idle first. If idle RSS already ate 6.5 GB, there is no room for a second slot.

Put a queue in front of the model

A small local proxy is enough: one worker, a FIFO, a max wait, and a 429 or 503 when the queue is full. Clients retry with jitter. This is better than letting gunicorn stampede llama-server.

import json, queue, threading, urllib.request
from http.server import BaseHTTPRequestHandler, HTTPServer

work = queue.Queue(maxsize=8)
UPSTREAM = "http://127.0.0.1:8080/v1/chat/completions"

def worker():
    while True:
        handler, body = work.get()
        try:
            req = urllib.request.Request(
                UPSTREAM, data=body,
                headers={"Content-Type": "application/json"},
            )
            raw = urllib.request.urlopen(req, timeout=180).read()
            handler.reply(200, raw)
        except Exception as exc:
            msg = json.dumps({"error": str(exc)}).encode()
            handler.reply(504, msg)
        finally:
            work.task_done()

class Handler(BaseHTTPRequestHandler):
    def reply(self, code, raw):
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(raw)

    def do_POST(self):
        length = int(self.headers.get("Content-Length", "0"))
        body = self.rfile.read(length)
        try:
            work.put_nowait((self, body))
        except queue.Full:
            self.reply(429, b'{"error":"queue full"}')

threading.Thread(target=worker, daemon=True).start()
HTTPServer(("127.0.0.1", 8081), Handler).serve_forever()

Bind the queue to localhost and put nginx in front of 8081, not 8080. The model process never sees a stampede. Tune maxsize to how long a user will wait. Eight pending jobs at 20 seconds each is almost three minutes at the tail. A smaller queue and a fast 429 is usually kinder.

One user, many users, and batch jobs

A personal SSH-tunnel chatbot needs no queue. A Slack bot for a team of ten does. A nightly batch can run sequentially from a systemd timer and skip HTTP entirely by calling llama-cli. Do not mix an interactive API and a huge batch on the same process if the batch can wait; you will wreck tail latency.

Fairness is FIFO unless you add priorities. Do not start there. Timeouts matter more: drop a job that waited 60 seconds, return 504, and let the client retry. Holding a 2k context while the user is gone wastes the only slot you have.

CPU worker pools in gunicorn or a Node cluster do not help here. Those tools multiply HTTP handlers, which is useful for JSON APIs and harmful when each handler calls the same one-slot model. You will pile up blocked workers, hit file descriptor limits, and still only complete one generate at a time. Cap the app outbound concurrency to one, or force every model call through the queue process so the application never opens a second upstream socket. That single constraint is the whole scaling story on 8 GB.

When a second process is the wrong scale-out

Two llama-server processes with the same 7B Q4 file double the weights in RAM. That only works on 16 GB-plus, and even then the SSD and memory bus are shared. Horizontal scale for large models is a remote API. Vertical scale on one VPS is a smaller model or a shorter context, not more copies.

Ollama's parallel setting is the same trap with friendlier names. Leave it at 1 until RSS and tok/s prove otherwise.

Takeaway

Serve one user at a time on CPU. Queue the rest, fail fast when the queue is full, and keep parallel slots at one on RAM-limited hosts. Concurrency without RAM is just a coordinated crash.

Spin up Ubuntu 24.04 on Netbay in under 60 seconds and put a one-slot llama-server behind a tiny queue — netbayhosts.in.