RAG Ingest Pipeline: Git Pull, Embed, Upsert

An index that is rebuilt from scratch every night is easy to write and expensive to wait for. The useful pipeline is: git pull, list files that changed since the last sha you indexed, chunk those files, embed the new chunks, upsert, and delete rows for paths that vanished. The unit of work is a commit range, not the whole tree. On a Lucknow VPS that job is a systemd timer, a lock file, and a SQLite or Postgres table you already trust.

Record the sha you last indexed

Keep a small state table: repo, last_sha, last_run_at, last_error. If last_sha is empty, walk the tree once. After that, git diff --name-status last_sha HEAD. A, M, D, and R are the only statuses you care about. Renames are a delete of the old path plus an insert of the new one. Do not invent a content-hash cache until this diff path is correct. Git already hashed the files.

Clone or fetch into a dedicated directory such as /var/lib/rag/repos/docs.git. Bare or a working tree both work. A working tree is easier to open in an editor when you debug a bad chunk. Run git as a dedicated user. Do not run ingest as root.

set -euo pipefail
REPO=/var/lib/rag/repos/docs
STATE=/var/lib/rag/state/last_sha
cd "$REPO"
git fetch --quiet origin
git merge --ff-only origin/main
NEW=$(git rev-parse HEAD)
OLD=$(cat "$STATE" 2>/dev/null || true)
if [ -z "$OLD" ]; then
  git ls-files
else
  git diff --name-status "$OLD" "$NEW"
fi
echo "$NEW" > "$STATE.tmp"
mv "$STATE.tmp" "$STATE"

The snippet prints the work list. Your Python job should do the same with subprocess and then chunk. If fetch fails, leave last_sha alone and exit non-zero so systemd records it. A half-updated sha is how you skip files forever.

Upsert is delete-then-insert per path

When a file changes, every chunk index may shift. The safe approach: DELETE FROM chunks WHERE path = ? AND model = ?, then insert the new chunks. Trying to patch chunk 3 in place will desync headings. Vectors are cheap compared to a wrong citation.

Embed after you have the new bodies. Hash model + body bytes. If that hash already has a vector in an embedding cache table, reuse it. Files that only change comments still need a path upsert, but you may skip the encoder.

import hashlib
import sqlite3
import subprocess

def changed_paths(repo, old_sha, new_sha):
    if not old_sha:
        out = subprocess.check_output(["git", "-C", repo, "ls-files"], text=True)
        return [("A", line) for line in out.splitlines() if line]
    out = subprocess.check_output(
        ["git", "-C", repo, "diff", "--name-status", old_sha, new_sha],
        text=True,
    )
    rows = []
    for line in out.splitlines():
        status, path = line.split("\t", 1)
        rows.append((status[0], path))
    return rows

def upsert_file(conn, path, chunks, model):
    conn.execute("DELETE FROM chunks WHERE path = ? AND model = ?", (path, model))
    for i, ch in enumerate(chunks):
        conn.execute(
            "INSERT INTO chunks(path, chunk_index, git_sha, model, dim, content, embedding) "
            "VALUES (?, ?, ?, ?, ?, ?, ?)",
            (path, i, ch["sha"], model, ch["dim"], ch["body"], ch["blob"]),
        )

Wrap one file in a transaction. Crash mid-file and you retry that file, not the repo. Crash mid-repo and last_sha stays old, so the next run repeats the range. Idempotent upserts make that safe.

Timers, locks, and what not to parallelize

Use a systemd timer every 5 or 15 minutes if the repo is busy, hourly if it is docs. flock a lock file so two timers never embed the same range. One encoder process. Intel Xeon Platinum cores do not enjoy two PyTorch runtimes fighting for RAM.

Skip binaries by extension and by git attributes. If git says a file is binary, do not chunk it. Cap file size. A 40 MB markdown dump is a data export, not a page. Log files_seen, files_changed, chunks_upserted, encode_ms, and rss_mb. That one JSON line per run is your ingest dashboard until you care about Prometheus.

Bind nothing public for ingest. The timer is local. L3/L4 DDoS filtering on the web API does not apply here. High-Speed SSD matters when you re-embed a large diff; sequential writes of BLOBs should be quiet. If encode_ms explodes, you pulled a huge commit, not a disk failure.

Do not push embeddings to a hosted vector product. Upsert locally. The git objects, the chunk table, and the model files all live on this VPS. That is the backup unit.

If origin/main rewinds, stop. A force-push is not a name-status you want to apply blindly. Compare last_sha to merge-base and refuse to delete the world. Human review is cheaper than an empty index. The same caution applies to a submodule bump that rewrites thousands of paths. Log the count of D statuses and abort if it exceeds a threshold such as 30 percent of the corpus. You can always re-run after you look. Keep the lock held across that abort so a second timer does not start a competing walk.

Takeaway

Index a commit range, upsert by path, and only then move last_sha. Ingest is git plumbing plus a local encoder, not a platform. You can spin up an Ubuntu 24.04 instance on Netbay in Lucknow in under 60 seconds and timer this job against a private clone — netbayhosts.in.