Q4 vs Q5 vs Q8 GGUF on a RAM-Limited CPU VPS

Quantization is how a 7B model fits a CPU VPS. GGUF files ship as Q4, Q5, Q8, and a handful of K-variants that shuffle which tensors stay more precise. The quality gap is real but smaller than the RAM gap. On an Intel Xeon Platinum plan with High-Speed SSD in Lucknow you should almost always start at Q4_K_M. Q8 is for boxes with spare memory or for 3B models. Q5 is the middle that still surprises people by OOM-killing llama-server when context grows.

Netbay does not sell GPUs. You cannot offload Q8 layers to VRAM you do not have. If Q4 of a 7B file still does not fit, you need a 3B model or a remote API, not a different quant name.

What the numbers mean in gigabytes

Weights dominate idle RSS. KV cache dominates the jump when a long prompt arrives. Approximate on-disk sizes for a dense 7B-class model:

• Q4_K_M: about 4.0–4.5 GB
• Q5_K_M: about 5.0–5.6 GB
• Q8_0: about 7.0–8.0 GB

A 3B model is roughly half of that. An 8B file is a shade above 7B. Ubuntu plus llama.cpp want around 1 GB. A 2048-token context on 7B adds hundreds of megabytes to a gigabyte depending on implementation and batching. That is why Q8 7B on 8 GB RAM is a bad idea even before the first user types a long paste.

ls -lh $HOME/models/*.gguf
python3 - << 'PY'
import os
for name in sorted(os.listdir(os.path.expanduser("~/models"))):
    if name.endswith(".gguf"):
        path = os.path.expanduser("~/models/") + name
        gb = os.path.getsize(path) / (1024 ** 3)
        print(name, round(gb, 2), "GiB on disk")
PY
free -h

If the GGUF is 7.2 GiB and the plan is 8 GB, stop. Do not start the server to "see if mmap saves you". mmap still needs resident pages for the working set. The process will grow, steal cache from the OS, and then die.

Quality versus speed on CPU

Q4_K_M is the default for a reason. It is small, it is fast on CPU because less memory moves per token, and for coding or summarization it is usually good enough. Q5_K_M recovers some perplexity at a RAM tax you feel on 8 GB. Q8_0 is close to fp16 for many tasks and is the wrong default on a RAM-limited VPS.

Tokens per second often improve when you drop from Q8 to Q4 on CPU, because inference is memory-bandwidth bound. People upgrade quant expecting a quality win and get a slower, fatter process. Measure both: a short eval prompt and RSS from ps.

./build/bin/llama-cli -m $HOME/models/model-q4_k_m.gguf -p "Summarize systemd in one paragraph." -n 64 -t 4 -c 1024
ps -o rss,cmd -p $PID

Replace the file with Q5 and Q8 if they fit. Record tokens per second from the llama.cpp footer. If Q5 is only slightly better on your prompts and adds 1 GB RSS, stay on Q4. If you are classifying short strings, even a 3B Q4 model may beat a 7B Q8 that thrashes.

Decision rule for 4 GB, 8 GB, and 16 GB

On 4 GB, only tiny 1B–3B Q4 files belong. On 8 GB, prefer 3B Q4 or Q5, or 7B/8B Q4 with context 2048 and parallel 1. On 16 GB you can run 7B Q5 or Q8 with a longer context, still without a GPU. None of these plans make a 70B Q4 file a good idea; that weight file is tens of gigabytes.

Do not mix two models in RAM to A/B quants. Run one process, one file. Store the other GGUF on High-Speed SSD and switch during a drain.

Practical picks

Internal chatbot on 8 GB: 7B or 8B Q4_K_M, context 2048, one slot. Classification and extraction: 3B Q4 or Q5, faster turns, more RAM left for nginx and the app. Anything that must match a hosted 70B: do not fake it with Q8 7B; call the API.

Keep the Q4 file even if you experiment with Q5. Rolling back is a systemd restart, not a rebuild.

Takeaway

Q4_K_M is the RAM-limited default. Q5 is optional when you have headroom. Q8 7B does not belong on 8 GB. Quantization does not create a GPU. Measure RSS and tokens per second, then keep the smallest file that is good enough.

You can spin up an 8 GB Ubuntu VPS on Netbay in under 60 seconds and A/B Q4 versus Q5 on Intel Xeon Platinum — netbayhosts.in.