Picking 3B vs 7B vs 8B for an 8 GB CPU VPS

An 8 GB CPU VPS is a small envelope. Ubuntu, nginx, and llama.cpp want about a gigabyte. That leaves roughly 7 GB for weights plus KV cache. A 3B Q4 GGUF is comfortable. A 7B or 8B Q4 file can work with a short context and one slot. Q8 7B cannot. 13B and 70B cannot. Netbay does not sell GPUs or H100s, and there is no managed inference SKU that will stretch this RAM. The decision is which small GGUF you load on Intel Xeon Platinum in Lucknow, and when you should send work to a remote API instead.

RAM first, parameters second

Parameter count is a label. Quantization and context decide RSS. Approximate idle plus short context on 8 GB:

• 3B Q4_K_M: about 2–3 GB total. Room for nginx, a small app, context 4096.
• 7B Q4_K_M: about 5–6 GB. Context 2048, parallel 1, little else on the box.
• 8B Q4_K_M: similar to 7B, often a bit fatter. Same rules, less margin.
• 7B Q5: often the straw that OOMs after a long paste.
• 7B Q8: weights alone fight the OS for 8 GB. Skip it.

If you also run Postgres, Docker, or a JVM, treat the machine as a 3B host. Do not stack a 7B model on a busy app server and hope mmap will share pages politely.

free -h
ls -lh $HOME/models/*.gguf
./build/bin/llama-cli -m $HOME/models/3b-q4_k_m.gguf -p "ping" -n 8 -t 4 -c 2048
./build/bin/llama-cli -m $HOME/models/7b-q4_k_m.gguf -p "ping" -n 8 -t 4 -c 2048

If the 7B process is killed, you have your answer. Stay on 3B or rent more RAM. More vCPUs will not save an OOM.

Quality and speed on CPU

3B Q4 models are surprisingly good at classification, extraction, short rewrites, and routing. They generate more tokens per second, which matters when the hardware is Xeon Platinum cores without a GPU. 7B and 8B Q4 models follow instructions better and hallucinate less on multi-step tasks, at a cost of RAM and latency. Llama 3.1 8B Q4 is not automatically better than Mistral 7B Q4 on your prompts; A/B them.

A slow 8B that swaps is worse than a 3B that stays resident. Tokens per second on CPU for 3B Q4 can be interactive. 7B Q4 is often usable for internal tools. Neither matches a hosted large model. If the product copy needs GPT-class reasoning, call a remote API from the same VPS and keep the local GGUF for the cheap path.

python3 - << 'PY'
import json, time, urllib.request

def once(model):
    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": "List three systemd Restart values."}],
        "max_tokens": 80,
    }).encode()
    req = urllib.request.Request(
        "http://127.0.0.1:8080/v1/chat/completions",
        data=body,
        headers={"Content-Type": "application/json"},
    )
    t0 = time.time()
    raw = urllib.request.urlopen(req, timeout=180).read()
    print(model, "sec", round(time.time() - t0, 2), "bytes", len(raw))

once("local-3b")
once("local-7b")
PY

You can only load one of those aliases at a time on 8 GB if they are 7B-class. Sequential tests mean restarting llama-server with a different -m. Record RSS with each.

A simple picker

Use 3B when the VPS also hosts the app, when you need 4k context, or when latency matters more than nuance. Use 7B Q4 when the VPS is dedicated to the model, context stays at 2048, and you have measured that 3B fails your eval set. Use 8B Q4 only if it beats 7B on that eval set and still leaves ~1 GB free at peak. If none of the three pass the eval, do not reach for Q8 or a 13B file on this plan. Use an API.

Ollama tags that look like 8B often unpack to Q4 GGUF under the hood. Check ollama list for size. A 4.9 GB tag on 8 GB is the 7B/8B Q4 case. A 15 GB tag is a refusal.

Operational notes

Load one model. Switching 3B to 7B is a drain and restart, not a hot swap on 8 GB. Keep both GGUF files on disk if you A/B; High-Speed SSD can hold extra weights you are not mapping. Watch swap. If si/so is non-zero during generate, the larger model is a lie.

For a first deploy, ship 3B Q4. Raise to 7B only when you have eval failures and RSS headroom. That order avoids a week of OOM tickets.

Takeaway

On 8 GB, 3B Q4 is the default, 7B/8B Q4 is the dedicated-box upgrade, and everything larger is an API. Pick with free -h and a prompt eval, not with a leaderboard that assumed a GPU.

You can spin up an 8 GB Ubuntu 24.04 VPS on Netbay in under 60 seconds and load a Q4 GGUF on Intel Xeon Platinum — netbayhosts.in.