Context Length vs RAM vs Tokens per Second

Context length is not a free quality knob. Every extra token in the window needs KV cache in RAM and extra work on every prompt eval. On a GPU host you might shrug at 32k. On an Intel Xeon Platinum CPU VPS with High-Speed SSD in Lucknow, 2048 is a good default, 4096 is a conscious spend, and 32k is how you OOM a 7B Q4 model that otherwise fit. llama.cpp will let you set -c to a large number. RAM will not.

Where the memory goes

Weights are roughly constant. KV cache scales with layers, heads, context, batch, and parallel slots. Doubling context from 2048 to 4096 can add hundreds of megabytes to more than a gigabyte on 7B, depending on type and implementation. Four parallel slots multiply that again. People raise -c to "support long PDFs" and then wonder why the process died while idle with a single empty slot allocated up front.

Prompt eval is also compute. The server must read the whole prefix. A 6k-token paste on CPU can sit for tens of seconds before the first output token. Tokens per second for generation may look fine in the log footer while the user already left. Measure time to first token separately from generation speed.

./build/bin/llama-server   -m $HOME/models/model-q4_k_m.gguf   --host 127.0.0.1 --port 8080   -t 4 --parallel 1   -c 2048
ps -o rss= -p $(pgrep -n llama-server)

Restart with -c 4096 and compare RSS at idle. If the jump is larger than your free memory, you do not have 4096. There is no SSD trick that stores KV cache cheaply enough for interactive use.

A fair benchmark of 2k versus 4k versus 8k

Use one GGUF, one thread count, and two prompt files: a 200-token question and a 1500-token paste. Record RSS, time to first token, and tokens per second. Keep max_tokens fixed so generation length does not confuse the picture.

python3 - << 'PY'
import json, time, urllib.request
url = "http://127.0.0.1:8080/v1/chat/completions"
body = {
    "model": "local-q4",
    "messages": [{"role": "user", "content": "Explain mmap in 80 words."}],
    "max_tokens": 80,
    "stream": False,
}
data = json.dumps(body).encode()
t0 = time.time()
req = urllib.request.Request(url, data=data, headers={"Content-Type": "application/json"})
raw = urllib.request.urlopen(req, timeout=120).read()
print("seconds", round(time.time() - t0, 2), "bytes", len(raw))
PY

Repeat after changing only -c. If 4096 adds two seconds of eval on the short prompt, you are paying for a larger graph even when the user did not use it. Some builds allocate the max graph up front. That is a reason to keep -c near the real workload.

For long documents, chunk outside the model. Embed or grep, then send a 1k-token slice. A CPU 7B Q4 model is a reasoner over a clip, not a 100-page buffer. Remote APIs with long context exist when that product requirement is non-negotiable.

Tokens per second will drop as eval work grows

Generation tok/s can stay similar while the request wall time explodes, because eval dominates. Users experience wall time. Cap request size at nginx or in the app. Reject prompts over N characters with a 413. That is kinder than an OOM.

• 1024–2048: interactive tools, command help, short chat.
• 4096: few-shot plus a small file, on 16 GB or a 3B model.
• 8192+: usually the wrong default on 8 GB CPU.

Do not advertise 128k context because the base model card does. The card assumed hardware you are not renting. Netbay plans are CPU VPS, not GPU inference nodes.

Product choices that respect the hardware

If the app needs a large window once a day, run that job against a remote API and keep the VPS for the 2k path. If every request is short, lower -c and recover RAM for the OS. If you must stay local, a 3B Q4 model at 4096 can be cheaper in RAM than a 7B Q4 at 2048, and sometimes faster. Benchmark your prompts; do not copy a cloud vendor's 128k banner.

Takeaway

Context is a RAM and latency budget. Set -c to what you measure, not what the model card brags about. On an 8 GB CPU VPS, 2048 is the honest starting window. Long documents should be chunked or sent to an API.

Follow along on a Netbay Ubuntu 24.04 VPS in Lucknow — you can be generating with llama-server in under 60 seconds after checkout at netbayhosts.in.