Run a Small LLM on a CPU VPS with llama.cpp

A CPU VPS can run a small language model if you stay honest about RAM and pick a quantized GGUF file. llama.cpp is the practical path: a C++ runtime that loads GGUF weights, uses the CPU, and does not require a GPU. Netbay does not sell GPUs or H100-class cards. The fleet is Intel Xeon Platinum CPU VPS with High-Speed SSD in Lucknow. That is enough for 3B to 8B models at Q4 quantization, and it is the wrong place to host a 70B dense model. If you need a large model, call a remote API.

What actually fits in RAM

The hard limit is resident memory, not CPU brand. A 7B Q4_K_M GGUF is roughly 4.4 GB on disk and a similar size once mapped. Add Ubuntu, llama.cpp itself, and the KV cache for the prompt, and an 8 GB plan is tight. A 3B Q4 model leaves headroom for the OS and a short context. An 8B Q4 model often works only if you keep context short and avoid extra processes. Q8 weights for a 7B model want around 8 GB for weights alone, which will not sit next to a running system.

Swap is not a workaround. If the model pages to disk, tokens per second collapse and the box feels frozen. Check free -h before you download a file, and leave at least a gigabyte free after the weights load. If a GGUF is larger than half your RAM, pick a smaller parameter count or a heavier quantization drop, or send the request to a hosted API.

Build llama.cpp on Ubuntu 24.04

You need a compiler, cmake, and git. Stay on CPU: do not enable CUDA. OpenBLAS can help matrix multiplies on Intel Xeon Platinum, but a plain Release build is enough to prove the path.

sudo apt-get update
sudo apt-get install -y build-essential cmake git wget
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=OFF -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc)
ls build/bin/llama-cli

The -j flag matches vCPU count. On a 4 vCPU plan the compile finishes in a few minutes and writes llama-cli plus llama-server into build/bin. If cmake cannot find a compiler, install build-essential again and retry. Do not copy binaries from a laptop with a different glibc; build on the VPS.

Download a Q4 GGUF and run a prompt

Use a well-known 3B or 7B Q4_K_M file from Hugging Face. Store it on the High-Speed SSD, not in /tmp. Replace the URL with the model you actually trust; the flags below are what matter.

mkdir -p ~/models
wget -O ~/models/model-q4_k_m.gguf "https://huggingface.co/example/model-Q4_K_M.gguf"
./build/bin/llama-cli   -m $HOME/models/model-q4_k_m.gguf   -p "Explain systemd unit files in two short paragraphs."   -n 128 -t 4 -c 2048 --no-display-prompt

-t should match vCPU count. -c 2048 is a safe first context on 8 GB. -n 128 caps the answer so you can see tokens per second without waiting. ngl stays at zero because there is no GPU to offload layers to. The first run mmaps the file and may look slow; later runs reuse the page cache.

Watch RSS while it loads. If llama-cli is killed with code 9, you ran out of RAM. Drop to a 3B Q4 file, cut context to 1024, or stop other services. That failure is a sizing signal, not a llama.cpp bug.

CPU tuning that actually moves the needle

Threads, batch size, and memory pressure dominate CPU inference. Oversubscribing threads past vCPU count usually hurts. mmap lets the kernel pull weights from SSD on demand, which is why a High-Speed SSD helps cold start but does not replace RAM. Once the working set is resident, tokens per second are limited by memory bandwidth and core count.

• Pin threads to vCPU count with -t.
• Keep --batch-size modest (512 is a reasonable start).
• Prefer Q4_K_M over Q8 on 8 GB and below.
• Leave headroom: do not run Docker, MySQL, and a 7B model on the same 8 GB box.

A 3B Q4 model on four Xeon Platinum vCPUs often lands in a usable interactive range. A 7B Q4 model is slower and still fine for internal tools. A 13B or 70B GGUF belongs on a remote API, not on this VPS.

Takeaway

llama.cpp on a CPU VPS is a real local runtime for small GGUF models, not a substitute for GPU inference. Start with Q4, a 3B or 7B file, a short context, and thread count equal to vCPUs. When the weights no longer fit, stop fighting the box and use an API.

You can spin up an Ubuntu 24.04 instance on Netbay in under 60 seconds and compile llama.cpp on Intel Xeon Platinum — netbayhosts.in.