Build a Multi-Model Router Cheap Model First

Most product traffic is not a hard reasoning problem. Classification, rewriting, extracting a field, answering from a short snippet: a small remote model does that well at a fraction of the price. Teams still pin every request to the most expensive chat model they have access to, then wonder why the vendor invoice jumped. A production router on a VPS is a few hundred lines: pick a default cheap model, escalate on explicit signals, and record which path you took.

This is not a managed LLM product. You call remote APIs yourself from an Ubuntu service in Lucknow DC01. The VPS has Intel Xeon Platinum cores, High-Speed SSD, and L3/L4 DDoS filtering. The router is just your process. Cheap-first only works if the cheap path is the default and the expensive path is an exception you can count.

Default cheap, escalate on signals

Name two models in config, not in prompt text. cheap_model is something like gpt-4o-mini or claude-haiku. dear_model is the large chat model you actually mind paying for. Every request starts on cheap_model unless a rule fires.

Useful rules are boring. Escalate when the user asks for multi-step planning, when the cheap response fails a JSON schema, when a confidence field you asked the cheap model to emit is below a threshold, or when the prompt is longer than a token budget you set. Do not escalate because the user typed please use the smart model. That is a cost attack.

Keep the classifier tiny. Either a keyword and length check in your code, or one extra cheap completion that returns a single token: CHEAP or DEAR. The second option costs a little and is more robust. Cache the decision per prompt hash for a few minutes so retries do not double-charge.

Implement the router as one function

The HTTP handler should not know vendor URLs. It calls routeCompletion(prompt, opts) and gets text plus metadata. Metadata is what you log: model used, escalate reason, latency, token counts.

const CHEAP = process.env.CHEAP_MODEL || 'gpt-4o-mini';
const DEAR = process.env.DEAR_MODEL || 'gpt-4o';

function wantsDear(prompt, opts) {
  if (opts.forceDear) return 'forced';
  if (prompt.length > 8000) return 'long-prompt';
  if (opts.requireJson && opts.lastParseFailed) return 'json-retry';
  const plan = /\b(plan|architecture|trade-?off|step by step)\b/i;
  if (plan.test(prompt) && prompt.length > 400) return 'planning';
  return null;
}

async function callModel(model, prompt, extra) {
  const res = await fetch('https://api.openai.com/v1/chat/completions', {
    method: 'POST',
    headers: {
      Authorization: 'Bearer ' + process.env.OPENAI_API_KEY,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      model: model,
      messages: extra.messages || [{ role: 'user', content: prompt }],
      max_tokens: extra.maxTokens || 512,
    }),
  });
  if (!res.ok) throw new Error('vendor ' + res.status);
  const body = await res.json();
  return body.choices[0].message.content;
}

async function routeCompletion(prompt, opts) {
  const reason = wantsDear(prompt, opts || {});
  const first = reason ? DEAR : CHEAP;
  const text = await callModel(first, prompt, opts || {});
  if (!reason && opts && opts.requireJson) {
    try {
      JSON.parse(text);
    } catch (err) {
      const retry = await callModel(DEAR, prompt, Object.assign({}, opts, { lastParseFailed: true }));
      return { text: retry, model: DEAR, reason: 'json-retry' };
    }
  }
  return { text: text, model: first, reason: reason || 'cheap-default' };
}

Object.assign is used instead of object spread so the sample stays easy to paste. There is no template literal in that file. Keys still come from EnvironmentFile as in the secrets post.

Measure mix or you will not keep the default

If 80 percent of requests already hit DEAR, the router is decoration. Log a line per request: request_id, model, reason, prompt_tokens, completion_tokens, ms. Grep for reason=cheap-default versus reason=planning. Once a week, sample twenty cheap responses and score them. If cheap quality is fine, tighten escalate rules. If cheap is failing a task type, add a dedicated rule for that type rather than sending everything upstairs.

Put a monthly token budget in the process. When cheap plus dear spend crosses a watermark, stop escalating except for forceDear from an internal admin path. Serving a slightly worse answer is better than a surprise invoice. Return 429 with a Retry-After when you shed load. The public NIC already has L3/L4 DDoS filtering; application-level spend is your problem.

journalctl -u ai-app --since '1 hour ago' | awk '/model=/ {print}'
awk 'BEGIN{c=0;d=0} /reason=cheap-default/{c++} /reason=/{d++} END{print c, d}' /srv/ai-app/var/router.log

Keep router.log on High-Speed SSD under ReadWritePaths. Rotate it. Do not log the full prompt if it contains customer PII; log a hash and the first 80 characters.

Failures and fallbacks

Cheap model 5xx should retry once, then escalate or fail, depending on the product. Escalating every blip trains you to pay for vendor brownouts. Dear model 429 should not fall back to cheap for a task you already decided was hard; return 503 and let the client retry. Circuit-break a vendor that has failed three times in 30 seconds. Talk to a second vendor only if you actually have that key and a mapping of model names.

Do not hide the model name from internal logs. Do hide it from end users unless the product needs to disclose it. Users who see the name will try to force the expensive path.

The takeaway: cheap-first is a default plus a short list of escalate reasons, not a pile of embeddings. Count the mix, cap the spend, and keep the HTTP handler ignorant of vendor URLs.

You can spin up an Ubuntu 24.04 instance on Netbay in under 60 seconds and follow along — netbayhosts.in.