Agent Architecture: Local 7B Plus API Fallback

The rest of this series is pieces. This post is the box you actually run: one Ubuntu VPS in Lucknow DC01, public TLS at nginx, an app that owns tools and routing, a local 7B on loopback for private and cheap work, Redis for jobs, and a remote API for tasks the 7B cannot finish. No managed LLM. No GPU SKU. Intel Xeon Platinum, High-Speed SSD, L3/L4 DDoS filtering on the public NIC.

If a request is classified or contains identifiers, it may not leave the machine. If it is a public FAQ, it may go to a cheap remote model. If it is a long plan, it may go to a dear remote model after the router says so. Tools never run from free-form model text. That is the architecture. Everything else is flags.

One public port, four internal ones

The guest listens on :443 only. nginx terminates TLS and proxies to 127.0.0.1:3000. The app talks to Redis on 6379, to the local model on 8080, and to api.openai.com (or another vendor) on 443 outbound. Nothing else is open. ssh is keys only, not a product path.

Units:

• nginx.service
• redis-server.service
• ai-app.service (HTTP, EnvironmentFile=/etc/ai-app.env)
• ai-worker@.service (one or two instances)
• llm-server.service (127.0.0.1:8080, MemoryMax, CPUQuota)

Config lives in /srv/ai-app/config. Weights live in /srv/models. Vectors live in /srv/vectors. Secrets live in /etc/ai-app.env. Backups copy config and vectors, not weights.

Classify, then route, then maybe tool

The HTTP handler does not call a model and hope. It labels the request: private, interactive, or batch. Private goes to local-7b with allowRemote=false. Interactive goes through the cheap-first router and may stream through nginx. Batch is LPUSH onto queue:infer.

function classify(req) {
  if (req.headers['x-data-class'] === 'private') return 'private';
  if (req.body && req.body.async) return 'batch';
  return 'interactive';
}

async function handle(req) {
  const kind = classify(req);
  const prompt = req.body.prompt;
  if (kind === 'private') {
    return privateComplete(prompt, false);
  }
  if (kind === 'batch') {
    return enqueue({ id: req.body.id, prompt: prompt });
  }
  const routed = await routeCompletion(prompt, { requireJson: !!req.body.json });
  if (req.body.tools) {
    const call = parseTool(routed.text, req.session);
    return runTool(call, req.session);
  }
  return routed;
}

That is glue. The functions were defined in earlier posts. Keep them in separate modules so the handler stays this short. Fail private closed if llm-server is down. Do not "helpfully" send the payroll prompt to OpenAI because the local unit crashed.

Streaming stays on the interactive path only. Queue workers call the same router. Workers that must stay private set allowRemote false in the job payload.

Fallback policy is data policy

Write the policy in config, not in a prompt.

{
  "cheap_model": "gpt-4o-mini",
  "dear_model": "gpt-4o",
  "local_model": "local-7b",
  "private_fail_closed": true,
  "interactive_remote": true,
  "max_local_context": 1024,
  "max_queue": 15,
  "tools_default": ["lookup_order", "search_docs"],
  "tools_never": ["run_shell", "fetch_url", "write_file"]
}

private_fail_closed means a 503 with a generic error, plus a metric. interactive_remote means the cheap-first router may use vendor APIs. max_local_context keeps prefill inside the tok/s you measured. max_queue is LLEN. tools_never is enforced in parseTool even if a prompt asks for a shell.

When the cheap remote API is 429, do not dump interactive traffic onto the local 7B unless the prompt is short and you have spare tok/s. The local worker is for private work. When the local 7B is saturated, private traffic 429s. Do not steal the local worker for a public chat.

Health, deploy, restore

/health on the app checks: nginx is not the check, the app is. Probe Redis PING, probe 127.0.0.1:8080 with a one-token completion or a runner /health if it has one, and confirm EnvironmentFile loaded by testing that the key prefix redacts. systemd OnFailure can ping you. Do not block deploys on a vendor outage if local private still works; do block deploys if gold.jsonl eval fails.

Deploy is git pull of the app, restart ai-app and workers, leave llm-server running so you do not reload weights on every prompt tweak. Restore is the backup post: unpack config and vectors, hash-check weights, restore env, start units in order redis, llm-server, workers, app, nginx.

The whole guest should still ssh if the model OOM-kills. That is why MemoryMax sits on llm-server and OOMScoreAdjust prefers it.

The takeaway: one VPS, one public port, a loopback 7B for private text, a remote cheap-first API for the rest, Redis for bursts, tools as code. That is production AI infrastructure you can actually operate.

You can spin up an Ubuntu 24.04 instance on Netbay in under 60 seconds and follow along — netbayhosts.in.