Fine-Tuning Is Usually the Wrong First LLM Step

Fine-tuning is the first idea many teams reach for when a chat model sounds generic. It is almost never the first thing that will move a production metric. A fine-tune needs a clean labeled set, an eval that you already trust, a training budget, and a plan to host or call the resulting weights. On a single VPS in Lucknow you already have the pieces that usually win instead: a better system prompt, a router that uses a cheap model, retrieval over your own documents, and a golden set you score every week.

Netbay does not sell a managed LLM or a training cluster. You will call remote APIs from Ubuntu, or run a small local model on Intel Xeon Platinum CPU. High-Speed SSD holds your eval JSONL. L3/L4 DDoS filtering protects the public app. None of that is a reason to start by mutating weights.

What fine-tuning actually buys

Fine-tuning changes the prior over tokens. It is good at format (always emit this JSON shape), tone (always sound like this support brand), and narrow domain jargon that never appears in the base model. It is weak at facts that change, at policy that you must be able to edit this afternoon, and at tool use you have not specified. If your bug is "the model invented a refund policy", a fine-tune will memorize this week's policy and lie next month. Put the policy in a retrieved snippet and instruct the model to quote it.

If your bug is "the model rambles", cut max_tokens, add a one-line system instruction, and add a stop sequence. Measure. If your bug is "the model cannot see our SKUs", that is retrieval or a SQL tool, not a weight update.

The cost that people skip is dataset work. You need hundreds of input-output pairs that match production traffic, not twenty blog examples. You need a holdout. You need to keep collecting because product copy changes. Until you can score a prompt change on that holdout, you cannot score a fine-tune either. Build the eval first. Then you may never need the fine-tune.

Build an eval before you touch weights

A golden set is a JSONL file of prompts, allowed facts, and a grader. The grader can be exact match for extraction, a schema check for JSON, or a second cheap model asked "does this answer violate the policy snippet". Run it in CI or from a systemd timer on the VPS. Store scores next to git SHAs of the prompt pack.

import json
import os
import urllib.request

GOLD = '/srv/ai-app/eval/gold.jsonl'

def complete(prompt):
    body = json.dumps({
        'model': os.environ['CHEAP_MODEL'],
        'messages': [{'role': 'user', 'content': prompt}],
        'max_tokens': 256,
    }).encode('utf-8')
    req = urllib.request.Request(
        'https://api.openai.com/v1/chat/completions',
        data=body,
        headers={
            'Authorization': 'Bearer ' + os.environ['OPENAI_API_KEY'],
            'Content-Type': 'application/json',
        },
    )
    with urllib.request.urlopen(req, timeout=60) as res:
        data = json.loads(res.read().decode('utf-8'))
    return data['choices'][0]['message']['content']

def main():
    ok = 0
    n = 0
    with open(GOLD, 'r', encoding='utf-8') as fh:
        for line in fh:
            row = json.loads(line)
            n += 1
            text = complete(row['prompt'])
            if row['must_include'] in text and row['must_not'] not in text:
                ok += 1
    print('pass', ok, 'of', n)
    if n == 0 or ok / float(n) < 0.85:
        raise SystemExit(1)

if __name__ == '__main__':
    main()

must_include and must_not are blunt, which is the point. You want a gate that fails a deploy, not a vibe check. Add schema validation for tool-calling tasks. Add a token-cost column so a "better" prompt that triples spend is visible.

Try these levers in order

Work the list in this order and stop when the eval moves.

1. System prompt and output contract. Specify JSON keys, refuse conditions, and the citation format.
2. Cheap-first routing so easy tasks stop paying for a large model that still rambles.
3. Retrieval of current policy and product text, with the model instructed to say "not in context" when the snippet is missing. That is not a chunking tutorial; it is a lookup you already have in Postgres or a vector directory.
4. Tool allowlists and confirmation for side effects, covered later in this series.
5. A larger remote model for the slice of traffic the eval still fails.
6. Fine-tune, only if the remaining failures are format and tone on a stable distribution, and you can host or call the result.

python3 /srv/ai-app/eval/run.py
git -C /srv/ai-app log -1 --format=%H
install -m 0644 /srv/ai-app/eval/gold.jsonl /var/backups/ai-app/gold.last.jsonl

Keep gold.jsonl in git. Keep vendor keys out of git. When a fine-tune eventually happens, it is a new model name in the router config plus a new row in the eval log. If the score does not beat the prompt-only baseline on the same gold set, do not ship it.

When you should fine-tune

Fine-tune when you have a stable task, a gold set of at least a few hundred rows, a baseline you cannot beat with prompt and retrieval, and a serving plan. Serving on Xeon CPU is realistic for a small adapter or a small dense model; it is not realistic for a huge dense model. If you cannot name the hardware and the latency budget, you are not ready.

Do not fine-tune to "add knowledge". Knowledge belongs in data you can edit. Do not fine-tune to paper over prompt injection. That is a security boundary, not a style.

The takeaway: eval, prompt, retrieve, route, then maybe fine-tune. Weights are the expensive way to store instructions you should have kept as text.

You can spin up an Ubuntu 24.04 instance on Netbay in under 60 seconds and follow along — netbayhosts.in.