Host Private LLM Inference on a Single VPS

Some prompts should not go to a remote vendor: payroll questions, unpublished source, customer identifiers, internal tickets. A local 7B-class model on one VPS is a reasonable private path if you treat it as a loopback daemon, not as a public API. This is not an installer for a frontend or a model runner. Assume you already have a binary that speaks an OpenAI-shaped HTTP API on a port you choose. The production problem is isolation, memory, and who is allowed to talk to it.

Netbay does not offer a managed LLM and does not publish GPU SKUs for this. You are on Intel Xeon Platinum CPU, High-Speed SSD for the weight file, and L3/L4 DDoS filtering on :443. CPU decode of a 7B quantized model is slow compared with a vendor API. Use it for private and cheap-local traffic. Fall back to a remote model when the task is hard and the data is allowed to leave.

Bind to loopback or do not bother

The moment the model port is on 0.0.0.0, you have shipped an unauthenticated completion API to the internet. Prompt injection against a public local model is a side quest you do not need. Listen on 127.0.0.1 only. nginx never proxies to that port. The app on 127.0.0.1:3000 is the only client. Firewall the port anyway.

Weights live on disk. A 4-bit 7B is a few gigabytes. Put them under /srv/models owned by a dedicated user, mode 0750. Do not put them in the git tree. Do not share /srv/models with the web user beyond what the runner needs. High-Speed SSD matters at load time; after the model is resident in RAM, you are in CPU and memory bandwidth.

sudo useradd --system --home-dir /srv/models --shell /usr/sbin/nologin llm
sudo mkdir -p /srv/models /run/llm
sudo chown -R llm:llm /srv/models /run/llm
sudo ss -tulpn | awk '/127.0.0.1:8080|0.0.0.0:8080/'
sudo nft list ruleset | awk '/8080/'

If ss shows 0.0.0.0:8080, stop. Fix the runner flags to 127.0.0.1 and confirm again. Do not open 8080 in the firewall "temporarily".

systemd limits are the product

A 7B quantized model can pin every vCPU and grow RSS until the OOM killer picks nginx. That is an outage of the whole box for a chatbot. Pin CPUQuota, MemoryMax, and thread count. One generation at a time is the default on a small plan. Parallel decode on CPU rarely pays; you want one in-flight local job and a Redis queue in front.

[Unit]
Description=Local 7B OpenAI-shaped server
After=network.target

[Service]
Type=simple
User=llm
Group=llm
WorkingDirectory=/srv/models
ExecStart=/usr/local/bin/llm-server --host 127.0.0.1 --port 8080 --model /srv/models/7b-q4.gguf --threads 4
Restart=on-failure
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/srv/models /run/llm
MemoryMax=8G
CPUQuota=400%
LimitNOFILE=4096
OOMScoreAdjust=200

[Install]
WantedBy=multi-user.target

CPUQuota=400% means four cores of a Xeon Platinum guest, not every core you bought. Leave headroom for nginx, Redis, and the app. MemoryMax must exceed weights plus KV cache for your max context. If it is too tight, the unit fails instead of taking down sshd. OOMScoreAdjust=200 makes the runner the preferred victim if something still goes wrong.

The binary name above is a stand-in. Use whatever runner you already installed. The flags that matter are host, port, threads, and the model path. Do not add a public CORS origin. Do not enable an admin UI on that port.

The app is the only client

Talk to http://127.0.0.1:8080/v1/chat/completions from the app. Treat it as another model name in the router: local-7b. Health-check it on a timer. If the unit is down, fail private tasks closed; do not silently send payroll text to OpenAI.

async function completeLocal(prompt) {
  const res = await fetch('http://127.0.0.1:8080/v1/chat/completions', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      model: 'local-7b',
      messages: [{ role: 'user', content: prompt }],
      max_tokens: 256,
    }),
  });
  if (!res.ok) throw new Error('local model ' + res.status);
  const body = await res.json();
  return body.choices[0].message.content;
}

async function privateComplete(prompt, allowRemote) {
  try {
    return { text: await completeLocal(prompt), model: 'local-7b' };
  } catch (err) {
    if (!allowRemote) throw err;
    return { text: await callModel(process.env.CHEAP_MODEL, prompt, {}), model: 'remote-fallback' };
  }
}

allowRemote must default false for classified traffic. Log model=local-7b. Never log the prompt body for that path.

Capacity is honest tokens per second

On Xeon CPU, expect single-digit to low tens of tokens per second for a 7B depending on quantization and context, not vendor-like streaming. Prefill of a long prompt will stall the one worker. Cap context. Cap concurrent local jobs at 1. Put overflow in Redis. If that is too slow for the product, the private path is for summaries and classification, and the interactive chat stays on a remote API with redaction.

The takeaway: a private model is a loopback service with memory caps, one client, and a closed fail if it dies. Public ports and unbounded threads turn a 7B into an outage.

You can spin up an Ubuntu 24.04 instance on Netbay in under 60 seconds and follow along — netbayhosts.in.