Caching Prompts and Embeddings on a Linux VPS

Embedding and generation are the expensive steps. The same question hits a docs search box dozens of times. The same chunk is re-embedded after every ingest that forgot a content hash. A local cache keyed by model id and exact bytes removes that work. Redis is optional. SQLite on High-Speed SSD with WAL is enough, lives next to the index, and backups with the rest of the data directory.

Two caches, two TTLs

The embedding cache is content-addressed and nearly immortal. Key = sha256(model_name + NUL + utf8_bytes). Value = float32 blob + dim + created_at. If the text did not change and the model did not change, the vector cannot change. TTL is optional; eviction is LRU when the file grows past a cap, not time.

The prompt cache is different. Key = sha256(model_name + sampler params + prompt bytes). Value = completion text + token counts + created_at. TTL in hours. Docs change. A cached answer about a flag that moved last week is a bug. Store the git sha of the index in the key if the prompt includes retrieved chunks, so an ingest busts the answer cache automatically.

CREATE TABLE IF NOT EXISTS embed_cache (
  cache_key TEXT PRIMARY KEY,
  model TEXT NOT NULL,
  dim INTEGER NOT NULL,
  embedding BLOB NOT NULL,
  created_at TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS prompt_cache (
  cache_key TEXT PRIMARY KEY,
  model TEXT NOT NULL,
  git_sha TEXT NOT NULL,
  answer TEXT NOT NULL,
  prompt_tokens INTEGER NOT NULL,
  completion_tokens INTEGER NOT NULL,
  created_at TEXT NOT NULL,
  expires_at TEXT NOT NULL
);

CREATE INDEX IF NOT EXISTS prompt_cache_exp ON prompt_cache(expires_at);

Prune expires_at on a timer. Vacuum only after large deletes. Do not vacuum on every request.

Hashing rules that prevent silent hits

Normalize the UTF-8. NFC vs NFD will fork keys. Strip a trailing newline if your chunker is inconsistent, or do not strip and make the chunker consistent. Include the model name and revision, not just "minilm". all-MiniLM-L6-v2 from two dates should not share a key if you care; pin the revision in the string.

For prompts, include temperature, max_tokens, and the ordered list of chunk ids, not only the final concatenated string. Two different shortlists can stringify similarly. You want a miss when retrieval changed.

import hashlib
import sqlite3
import numpy as np

def embed_key(model, text):
    h = hashlib.sha256()
    h.update(model.encode("utf-8"))
    h.update(b"\x00")
    h.update(text.encode("utf-8"))
    return h.hexdigest()

def get_or_embed(conn, model, text, encode_fn):
    key = embed_key(model, text)
    row = conn.execute(
        "SELECT dim, embedding FROM embed_cache WHERE cache_key = ?",
        (key,),
    ).fetchone()
    if row:
        return np.frombuffer(row[1], dtype=np.float32).copy()
    vec = encode_fn(text)
    conn.execute(
        "INSERT INTO embed_cache(cache_key, model, dim, embedding, created_at) "
        "VALUES (?, ?, ?, ?, datetime('now'))",
        (key, model, int(vec.shape[0]), vec.astype(np.float32).tobytes()),
    )
    return vec

Use IMMEDIATE transactions around insert. Two workers encoding the same new chunk is wasteful but correct if both inserts use the same key. OR IGNORE if you prefer.

Where to put the files and what to skip

Keep caches under /var/lib/rag/cache/ on the same VPS as the models. Do not put them in /tmp. Do not put them in the git working tree. A 2 GB embedding cache is a win; a 20 GB prompt cache of unique one-off questions is a leak. Cap prompt_cache with DELETE of oldest rows when count exceeds N.

Cache GET-style search. Do not cache a POST that includes user secrets. If a query string can hold a token, hash it for the key but do not log the raw key material. Bind the API to localhost or put it behind your public HTTPS with the usual L3/L4 DDoS filtering. The cache is not a CDN. Lucknow DC01 is one site; there is no second-region replica to invalidate.

Skip cache for empty queries, for queries under three characters, and for admin endpoints that rebuild the index. Hit rate is the metric. Log cache=hit or cache=miss on every embed and every generate. If embed hit rate is under 30 percent during ingest, your chunker is unstable. If prompt hit rate is under 5 percent on a public search box, your key includes noise such as a timestamp.

Intel Xeon Platinum plus High-Speed SSD makes the SQLite lookup trivial next to a 40 ms encode. The cache is worth it even at modest QPS.

Measure hit rate per route, not a single global counter. Embed during ingest should climb toward 80 percent after the first full pass. Generate on a public search box may sit at 15 percent and still save your p95. A cache that stores unique one-line questions forever is a disk leak dressed as an optimization. Cap it. The embed cache can grow with the corpus; the prompt cache should not grow with traffic. If hit rate falls after a deploy, you probably put a timestamp or a random request id into the key.

Takeaway

Hash bytes, pin the model name, and keep two SQLite tables with different lifetimes. The cache is part of the RAG data directory, not a separate product. You can spin up an Ubuntu 24.04 instance on Netbay in Lucknow in under 60 seconds and keep those files on High-Speed SSD next to the index — netbayhosts.in.