Serve GGUF Models with llama-server on a VPS

llama-cli is for a prompt on a terminal. llama-server is for a process that stays up, loads one GGUF, and answers HTTP. That is the serving path on a CPU VPS. There is no managed inference control plane on Netbay, and there is no GPU SKU. You run a binary under systemd on Intel Xeon Platinum with High-Speed SSD in Lucknow, you pick a Q4 3B–8B file that fits RAM, and you keep the listen address on loopback until a reverse proxy is ready.

Pick flags that match RAM, not marketing

The server maps the GGUF and allocates a KV cache sized by context and parallel slots. Those two numbers dominate RSS after the weights. -c 4096 with --parallel 4 can exceed an 8 GB plan even when the Q4 file itself is 4 GB. Start with -c 2048 and --parallel 1. Raise context only after free -h still shows a buffer.

Threads should equal vCPU count. More threads than cores add contention. --mlock can pin weights in RAM so the kernel does not reclaim them; only use it when you have spare memory. Without mlock, mmap plus High-Speed SSD still cold-starts reasonably, then the working set stays hot if the box is not under memory pressure.

./build/bin/llama-server   -m $HOME/models/model-q4_k_m.gguf   --host 127.0.0.1 --port 8080   -t 4 -c 2048 --parallel 1   --batch-size 512   --alias local-q4

Confirm with ss -lntp and a GET to /v1/models. If RSS is over 90 percent of RAM at idle, you will OOM on the first long prompt. Cut context or switch to a 3B file.

systemd unit for a single model process

A long-lived server needs a unit, a dedicated user, and a working directory that can read the GGUF. Do not run it as root. Restart on failure, but put a StartLimitBurst so an OOM loop does not hammer the disk.

# /etc/systemd/system/llama-server.service
[Unit]
Description=llama-server GGUF CPU API
After=network.target

[Service]
User=llama
Group=llama
WorkingDirectory=/opt/llama.cpp
ExecStart=/opt/llama.cpp/build/bin/llama-server -m /var/lib/llama/model-q4_k_m.gguf --host 127.0.0.1 --port 8080 -t 4 -c 2048 --parallel 1 --alias local-q4
Restart=on-failure
RestartSec=4
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

systemctl enable --now llama-server then journalctl -u llama-server -f. The first request logs prompt eval time and tokens per second. Those numbers are CPU tokens per second, typically single digits to low tens for 7B Q4, higher for 3B. If you need GPU-class throughput, this host cannot provide it; send that traffic to a remote API.

Routes you will actually use

llama-server exposes a small Web UI on / if you want a browser smoke test from an SSH tunnel. Production traffic should use /v1/chat/completions and /completion. /health is the probe for load balancers and systemd Type=notify setups that you wrap yourself. Do not publish the Web UI on a public IP.

Metrics in logs beat guesswork. Watch prompt eval (time to ingest context) versus token generation. Long system prompts hurt CPU servers more than hosted APIs because every token of context is paid for on every request unless you enable caching features your build supports. Keep system prompts short.

Swap still kills serving. If the unit is alive but tokens per second drop by an order of magnitude, check si/so in vmstat 1. Move to a larger RAM plan, a smaller GGUF, or an API.

Reloading a new GGUF

There is no hot-swap that keeps the HTTP port and replaces weights in place in a way you should depend on. Drain, stop the unit, replace the file, start the unit. Clients will see connection refused during the window; put a queue or a 503 in front if that window matters. Keep the old GGUF until the new process answers /health so you can roll back with a rename.

Checksum the file. A truncated wget looks like a mysterious load error. Compare bytes to the Hugging Face listing. Store models outside the git repo.

Takeaway

llama-server is a single-model CPU daemon. Size context and parallel slots for RAM, run it under systemd, and keep it on localhost. Small Q4 GGUF files belong here. Large models belong on a remote API.

Deploy Ubuntu 24.04 on Netbay in under 60 seconds and run llama-server next to your app in Lucknow — netbayhosts.in.