Plan LLM Capacity as Tokens Per Second on Xeon

Capacity planning for LLMs is not "how many users" and it is not "how many vCPU". It is tokens per second at a given context length, split into prefill (reading the prompt) and decode (emitting new tokens). Remote APIs hide this behind a rate limit and a bill. A local 7B on Intel Xeon Platinum does not. If you promise chat to 50 concurrent people on one VPS, you need to know that decode might be 8 tokens per second on one thread group, that two concurrent jobs share memory bandwidth, and that a 4k prompt can stall the box for seconds before the first token.

Netbay does not sell GPU SKUs or a managed LLM. Lucknow DC01 is Xeon Platinum, High-Speed SSD, L3/L4 DDoS filtering. Plan local inference as CPU. Plan remote inference as vendor RPM and spend. The VPS still has to hold nginx, Redis, and the app without swapping.

Measure, do not guess

Sit on the box. Run a fixed prompt at three lengths: 128, 1024, and 4096 input tokens. Ask for 64 output tokens. Record wall time to first token and wall time to last token. Decode tok/s is 64 divided by (last minus first). Prefill is first-token time as a function of input length. Repeat with one job and with two jobs. If two jobs are each slower than half, you are bandwidth bound and the second worker is a lie.

python3 /srv/ai-app/bench/toks.py --url http://127.0.0.1:8080/v1/chat/completions --in 128 --out 64 --n 5
python3 /srv/ai-app/bench/toks.py --url http://127.0.0.1:8080/v1/chat/completions --in 1024 --out 64 --n 5
python3 /srv/ai-app/bench/toks.py --url http://127.0.0.1:8080/v1/chat/completions --in 4096 --out 64 --n 5
vmstat 1 20
awk '/MemAvailable|Cached|SwapFree/' /proc/meminfo

import argparse
import json
import time
import urllib.request

def once(url, n_in, n_out):
    prompt = ' '.join(['alpha'] * n_in)
    body = json.dumps({
        'model': 'local-7b',
        'messages': [{'role': 'user', 'content': prompt}],
        'max_tokens': n_out,
        'stream': False,
    }).encode('utf-8')
    req = urllib.request.Request(url, data=body, headers={'Content-Type': 'application/json'})
    t0 = time.time()
    with urllib.request.urlopen(req, timeout=300) as res:
        data = json.loads(res.read().decode('utf-8'))
    dt = time.time() - t0
    usage = data.get('usage') or {}
    print('sec', round(dt, 3), 'prompt', usage.get('prompt_tokens'), 'out', usage.get('completion_tokens'))

def main():
    p = argparse.ArgumentParser()
    p.add_argument('--url', required=True)
    p.add_argument('--in', dest='n_in', type=int, required=True)
    p.add_argument('--out', dest='n_out', type=int, required=True)
    p.add_argument('--n', dest='n', type=int, default=3)
    args = p.parse_args()
    for _ in range(args.n):
        once(args.url, args.n_in, args.n_out)

if __name__ == '__main__':
    main()

Non-streamed bench understates TTFB. For interactive UI, stream and timestamp the first SSE event. The number you put on a slide is the worse of the two.

Translate tok/s into jobs and users

Suppose decode is 10 tok/s and a typical answer is 200 tokens. One local worker produces one answer every 20 seconds plus prefill. That is 3 completions a minute. A Redis queue with LLEN cap of 15 is five minutes of work. Beyond that, 429. Interactive chat at that rate feels like a batch job. Use local 7B for classification, redaction, and short summaries. Use a remote cheap model for chat unless the prompt is private.

Remote APIs need a different budget. If gpt-4o-mini allows N requests per minute on your key, your worker count must stay under N, and your monthly token spend must stay under the watermark from the router post. Measure actual prompt plus completion tokens from logs, not guesses from the UI.

vCPU count is a ceiling for thread flags, not a multiplier for tok/s. Four threads on four Xeon Platinum vCPU may beat two threads. Eight threads on four vCPU will not. Leave cores for the rest of the system. CPU steal in vmstat means you are not getting the cores you thought; plan on the measured tok/s, not the plan page.

Memory and disk are part of throughput

Weights plus KV cache plus OS must fit in RAM. If MemAvailable collapses and SwapFree starts moving, tok/s will fall off a cliff and the rest of the VPS will too. MemoryMax on the runner unit is how you keep sshd alive. High-Speed SSD matters when you load or swap; it will not save you from a working set that does not fit.

Context length is a knob. A 8k context is not free. If your product never sends more than 1k tokens of prompt, bench at 1k, not at the model maximum. Cap max_tokens in the router so a runaway completion cannot occupy the only local worker for minutes.

Queue depth is the user-visible SLA

Users do not see tok/s. They see wait time. Wait time is (queue length plus one) times (prefill plus decode) for a single worker. Publish that. If wait exceeds 15 seconds, do not take more local jobs. Offer remote fallback for non-private traffic. For private traffic, fail closed with a message, not a hang.

The takeaway: write down prefill, decode tok/s, concurrent jobs that still hold that rate, and the LLEN cap. That is the capacity plan. Everything else is hope.

You can spin up an Ubuntu 24.04 instance on Netbay in under 60 seconds and follow along — netbayhosts.in.