CPU Embeddings with Sentence Transformers

Embedding models turn text into fixed-length vectors so you can retrieve similar passages without a keyword match. On a single VPS you do not need a remote embedding API. Intel Xeon Platinum cores run MiniLM-class models at tens of milliseconds per short chunk, and llama.cpp can emit embeddings from a GGUF you already keep for generation. This post is about producing those vectors on CPU, normalizing them, and writing them to a file you control. It is not an install walkthrough for llama.cpp or Ollama.

Why CPU embeddings belong on one VPS

A managed embedding endpoint bills per token, adds a network hop, and ships your corpus off the box. For RAG over a private git repo, that is the wrong default. all-MiniLM-L6-v2 is 384 dimensions and roughly 80 MB on disk. It fits in RAM beside Postgres or SQLite. A 4 vCPU Lucknow VPS with High-Speed SSD will encode a few hundred short chunks per minute on CPU alone.

The real constraint is memory bandwidth and batch size, not marketing FLOPs. Keep the model on local SSD, pin one long-lived process, and batch 16 to 32 sentences. Do not spawn a new Python interpreter per HTTP request. Do not embed the same blob twice. Hash the model id plus the exact UTF-8 bytes and skip work you already did.

L3/L4 DDoS filtering on the public NIC does not belong in this path. Bind the embedder to 127.0.0.1. Your search API is the only process that should face the internet. The embedding worker talks Unix sockets or localhost HTTP.

Sentence Transformers as the default encoder

sentence-transformers wraps a Hugging Face encoder, mean pooling, and L2 normalization. For English docs, start with all-MiniLM-L6-v2. For mixed English and code comments, try all-mpnet-base-v2 at 768 dimensions, or a small E5 model. Load the model once at process start. Encode with normalize_embeddings set to True so cosine similarity is a plain dot product.

Set a thread budget. On a 4 vCPU plan, leave one core for Postgres, nginx, and the ingest cron. Torch will happily eat every core if you let it. OMP_NUM_THREADS=3 and MKL_NUM_THREADS=3 are enough. If RSS climbs after a few batches, you are holding tokenizers or graphs you do not need. Encode, convert to float32, drop the batch.

import os
from sentence_transformers import SentenceTransformer

os.environ["OMP_NUM_THREADS"] = "3"
os.environ["MKL_NUM_THREADS"] = "3"

model = SentenceTransformer("all-MiniLM-L6-v2")
texts = [
    "Chunk markdown by heading, not by a blind 512-token window.",
    "Store 384-d float32 vectors next to the source path and git sha.",
]
vecs = model.encode(texts, batch_size=16, normalize_embeddings=True)
print(vecs.shape[1], float(vecs[0].dot(vecs[1])))

Write the output as raw float32, not JSON arrays of decimals. A 384-d vector is 1536 bytes. JSON will 4x that and slow every scan. SQLite BLOB or a Postgres bytea column is the right on-disk form. Keep the model name and dimension in a sidecar table so you never mix MiniLM rows with a 768-d experiment.

llama.cpp embeddings without a new install story

If you already run a GGUF for generation, some of those weights expose an embedding endpoint. Point your ingest job at localhost. Send one string, receive a float array, L2-normalize in Python if the server does not. Treat the server as a black box you already operate. Do not compile it in this post. Do not install Ollama here. The contract is HTTP in, vector out, same dimension every time.

Keep generation and embedding on different ports if both run. A 30-second generate must not block a batch of 200 chunk embeddings. One worker, one listen address, one timeout. If the GGUF is large, embedding a short string still has to load layers. MiniLM is usually cheaper for retrieval. Use llama.cpp embeddings when you need the same tokenizer as the generator, not because it is fashionable.

curl -sS http://127.0.0.1:8080/embedding   -H "Content-Type: application/json"   -d "{"content":"Chunk markdown by heading, not by a blind window."}"

Parse the JSON array, convert to float32, divide by the L2 norm, store. Fail the job if the length is not the dimension you recorded at first run. A silent dimension change poisons every later cosine.

Batching, dimensions, and failure modes

Encode at ingest, not at query time for the corpus. Queries are one vector. Documents are thousands. A nightly job that walks new git commits is the right shape. If a batch OOMs, cut batch_size before you add RAM. MiniLM on short chunks is a few hundred MB. A 7B embedding GGUF is a different machine.

Watch for empty strings, BOM prefixes, and chunks that are only a code fence marker. They cluster together and pollute nearest-neighbor results. Skip chunks under a few dozen characters unless they are titles you explicitly want. Log encode_ms per batch and RSS after each 1000 rows. CPU steal on a busy node will stretch encode_ms; that is scheduling, not a bad model.

Pick one model and stick with it until you re-embed the whole corpus. Mixing models in one table is how you ship a search box that looks random.

Takeaway

Load one small encoder, normalize every vector, store float32 next to the source path, and keep the worker on localhost. Re-embed the corpus when the model changes, not when a blog post tells you to try a new name. You can spin up an Ubuntu 24.04 instance on Netbay in Lucknow in under 60 seconds and encode a private repo on the same box — netbayhosts.in.