Reranking RAG Results on a Small CPU VPS

Bi-encoders are fast because they never see the query and the document together. That is also why they miss. A cross-encoder reads the pair and outputs one relevance score. On a small VPS you cannot cross-encode the whole corpus. You can rerank the 20 or 40 chunks hybrid search already found. ms-marco-MiniLM-L-6-v2 is tens of millions of parameters and runs on Intel Xeon Platinum at interactive latency if the shortlist is short. That is the whole trick.

Retrieve wide, rerank narrow

Hybrid search should over-fetch. If you want 8 chunks in the prompt, retrieve 30. The reranker will drop near-misses that cosine liked because they shared a heading word. Cap the pair length. Truncate documents to the first 1500 characters. Truncate the query to a few hundred. A cross-encoder with a 512 token limit will silently drop the tail if you do not.

Load the reranker once in the API process or in a tiny localhost worker. Do not load it in the ingest job. Ingest only needs the bi-encoder. Two models in RAM at once is the memory question: MiniLM embedder plus MiniLM reranker is still a few hundred MB, which fits beside Postgres on a 4 GB plan if you are not also hosting a 7B generator on the same box. If you are, put the reranker next to the API and keep generation as a second process with a hard RSS cap.

from sentence_transformers import CrossEncoder

rerank = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2", max_length=512)

def rerank_hits(query, hits, keep=8):
    pairs = [(query, h["content"][:1500]) for h in hits]
    scores = rerank.predict(pairs, batch_size=8)
    ordered = sorted(zip(scores, hits), key=lambda row: float(row[0]), reverse=True)
    out = []
    for score, hit in ordered[:keep]:
        item = dict(hit)
        item["rerank"] = float(score)
        out.append(item)
    return out

Batch 8 pairs. On CPU, a batch of 8 at 512 tokens is the difference between 80 ms and a multi-second stall. Do not rerank 200 hits to look thorough.

When the reranker is wrong

Cross-encoders trained on MS MARCO like web-search relevance. They can demote a perfect code citation because it looks unlike a Bing passage. For a code-heavy corpus, add a cheap rule before the model: if the query token appears as a whole word in the chunk path or in a function name, floor the score so it cannot fall out of the top 8. Rules plus a small model beat a large model you cannot run.

Log query, hit ids before, hit ids after, and scores. When a user complains, you want to see the reranker swap. If it always swaps toward longer chunks, you have a length bias. Cap body length as above. If it always swaps toward the first heading in a file, your chunks still include too much shared preamble.

curl -sS http://127.0.0.1:8090/rerank   -H "Content-Type: application/json"   -d "{"query":"busy_timeout sqlite","hits":[{"id":1,"content":"PRAGMA busy_timeout = 5000"},{"id":2,"content":"generic database notes"}]}"

Keep that worker on 127.0.0.1. The public API already sits behind L3/L4 DDoS filtering. The reranker does not need a second public port.

Latency budget on a small box

Budget the request. Embed query 20 to 40 ms. FTS plus vector scan 10 to 80 ms. Rerank 30 hits 80 to 250 ms. Generation is whatever your LLM costs. If rerank exceeds 300 ms p95, cut max_length, cut shortlist size, or pin threads. OMP_NUM_THREADS=2 for the reranker if the generator is on the same VPS. High-Speed SSD only matters at load; after that the weights should be in page cache.

Do not send the reranker to a remote API. The shortlist includes private source. Lucknow DC01 is the trust boundary. If CPU cannot meet the budget, shrink the shortlist before you add RAM.

Fail open. If the reranker process is down, return the hybrid list. Search that is a bit noisier beats search that 502s.

Rerankers belong after hybrid fusion, not instead of it. If you rerank FTS-only results you will still miss paraphrase. If you rerank vectors-only you will still miss filenames like pg_hba.conf. The cross-encoder is a judge of a shortlist, not a search engine. Keep it off the ingest path. The weights are small enough to live under /var/lib/rag/models next to MiniLM, loaded at API start, and never downloaded on a live request.

Takeaway

Over-fetch with hybrid search, score the shortlist with a small cross-encoder, and keep that model on localhost. Reranking is a CPU loop over 30 pairs, not a new platform. You can spin up an Ubuntu 24.04 instance on Netbay in Lucknow in under 60 seconds and rerank private hits on that box — netbayhosts.in.