An OpenAI-Compatible Local API for llama.cpp

Most application code already speaks the OpenAI Chat Completions shape: a POST to /v1/chat/completions with model, messages, and a token budget. llama.cpp ships llama-server, which implements a useful subset of that API on localhost. You do not need a managed inference product, and you do not need a GPU. You need a Q4 GGUF that fits RAM, a process bound to 127.0.0.1, and clients that can set a custom base URL. On a Netbay Intel Xeon Platinum CPU VPS with High-Speed SSD in Lucknow, that is a 3B or 7B model, not a 70B hosted replica.

What compatibility actually means

llama-server answers chat completions, completions, and a models list. It will not implement every billing field, every tool-calling corner, or vision. If your SDK sends tools, logprobs, or response_format json_schema, test that path. Many SDKs only need base_url and an API key string they never validate on the wire.

Bind to loopback. The server does not replace nginx, TLS, or auth. A dummy key in the client keeps the SDK happy; check the key in the reverse proxy if you expose the port at all. One in-flight generation is the default you should keep on 8 GB. Parallel slots multiply KV cache.

Start llama-server with the OpenAI routes

Build llama.cpp the same way as a CLI install. Then run llama-server against your GGUF. --alias is the model name clients will send.

./build/bin/llama-server   -m $HOME/models/model-q4_k_m.gguf   --host 127.0.0.1 --port 8080   --alias local-7b   -c 2048 -t 4 --parallel 1   --ctx-size 2048
curl -s http://127.0.0.1:8080/v1/models

/v1/models should list local-7b. If the process dies at start, the GGUF is too large. Drop to Q4 or a 3B file. --parallel 1 keeps a single slot. Raising it without RAM is how you get OOM during a second request.

Call it with curl and with a small client

The JSON body matches what you already send to OpenAI, minus features the server ignores. Keep max_tokens modest so a slow CPU turn cannot run for minutes.

curl -s http://127.0.0.1:8080/v1/chat/completions   -H "Content-Type: application/json"   -H "Authorization: Bearer local-key"   -d '{"model":"local-7b","messages":[{"role":"user","content":"Say hello in one sentence."}],"max_tokens":64}'

In application code, set the base URL to http://127.0.0.1:8080/v1 and keep using your existing SDK. In Python that is OpenAI(base_url=..., api_key="local-key"). In Node it is new OpenAI({ baseURL: "http://127.0.0.1:8080/v1", apiKey: "local-key" }). Avoid template literals in that client if you copy snippets around; string concatenation is enough.

Timeout the HTTP client. CPU generation of 64 tokens on a 7B Q4 model may take several seconds. A 3B Q4 model is snappier. A 70B model will not load. If the product needs low latency at large size, send those calls to a remote API and keep the VPS for the small local path.

Streaming works over SSE on the same route when stream is true. Reverse proxies must not buffer that response. If you only need batch jobs, keep stream false and parse one JSON object.

Wire it behind a real app without lying to users

Set timeouts, max_tokens, and a clear error when the local server is down. A 502 from nginx is better than a hung worker. Health-check GET /health if your llama-server build exposes it, or GET /v1/models. Do not retry a cancelled generate blindly; you may double the CPU load.

Keep the model name stable. If you swap a 3B file for a 7B file, keep --alias the same only when prompt quality is acceptable. Applications cache model strings. Document that this endpoint is CPU-bound and slower than a hosted 70B. Product copy that claims GPU speed on a CPU VPS will be disproved on the first request.

Auth belongs at nginx or an internal network. The Bearer token in the example is a stub. If the VPS has a public IP, do not publish port 8080.

Takeaway

llama-server gives you an OpenAI-shaped local API in front of a GGUF. Point the SDK at localhost, run a 3B–8B Q4 model that fits RAM, and send oversized work to a remote API. That is the whole architecture.

You can spin up an Ubuntu 24.04 instance on Netbay in under 60 seconds and run llama-server on Intel Xeon Platinum — netbayhosts.in.