Chunk Markdown and Code for RAG on a VPS

Retrieval quality is mostly chunking. A 384-d MiniLM vector of a half function and a leftover heading will not match a real question. Blind 512-token windows with 50-token overlap are a demo default, not a docs default. On a VPS corpus of markdown and source, split on the structure the authors already wrote: ATX headings, fenced code blocks, and language-aware function bounds. Then embed those units.

Markdown is a tree, not a bag of tokens

Walk the file once. Keep a heading stack. A chunk is the heading path plus the paragraphs until the next heading of the same or higher level. Include the heading text in the chunk body so the embedding sees "Install Postgres" not just the commands underneath. Drop YAML front matter from the body but keep title and tags as metadata columns.

Fenced code inside a markdown section stays with that section if it is short. If a fence is longer than your token budget, split it as code using the rules below, and attach the same heading path as metadata. Never split through a fence because a character counter said so. The reader of the answer will see a broken block.

def markdown_chunks(text, max_chars=1800):
    lines = text.splitlines()
    stack = []
    buf = []
    out = []

    def flush():
        body = "\n".join(buf).strip()
        if body:
            title = " > ".join(stack) if stack else ""
            out.append({"title": title, "body": (title + "\n\n" + body).strip()})
        buf.clear()

    for line in lines:
        if line.startswith("#"):
            level = len(line) - len(line.lstrip("#"))
            if level <= 6 and line[level:level + 1] == " ":
                flush()
                stack[:] = stack[: level - 1]
                stack.append(line[level:].strip())
                continue
        buf.append(line)
        if sum(len(x) + 1 for x in buf) >= max_chars:
            flush()
    flush()
    return out

Cap on characters, not tokens, if you do not want a tokenizer in the splitter. MiniLM is comfortable well under 256 word pieces. 1200 to 1800 characters of prose is a safe first cap. Overlap is optional when headings already stitch context.

Code wants functions, not line windows

For Python, split on top-level def and class using the ast module. Emit one chunk per function with the module docstring prefixed. For Go, use go/parser if you will take the dependency, or a conservative regex on ^func with a brace counter. For languages you will not parse, split on blank lines and keep chunks between 40 and 200 lines. Discard import-only hunks.

Always store path, start_line, end_line, and language. A citation that cannot open the file at a line is trivia. Do not embed minified vendor bundles, lockfiles, or generated protobufs. A skip list of glob patterns belongs next to the ingest config, not in folklore.

import ast

def python_functions(src, path):
    tree = ast.parse(src)
    chunks = []
    for node in tree.body:
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
            start = node.lineno
            end = getattr(node, "end_lineno", start)
            lines = src.splitlines()[start - 1 : end]
            body = "\n".join(lines)
            chunks.append({
                "path": path,
                "start_line": start,
                "end_line": end,
                "body": path + "\n" + body,
            })
    return chunks

If a class is huge, emit the class signature as one chunk and each method as another. Retrieval then answers "where is the constructor" and "what does save do" as separate hits. That is what you want.

Metadata you will filter later

Every chunk row should carry: path, language, heading path or symbol name, start_line, git_sha, byte_len, token_est. You will filter to language=markdown when the question is a how-to, and to language=python when it is a stack trace. Embeddings are bad at that distinction by themselves.

Skip chunks that are only a table of contents, a license header, or a repeated copyright. They form a tight cluster and steal top-k slots. Skip files over a few MB. A sqlite dump in the repo is not documentation.

On Intel Xeon Platinum with High-Speed SSD in Lucknow, walking a 5000-file repo is I/O plus parse time, not a cluster job. Do it in one process. Keep the skip list in git next to the ingest script so a bad glob is reviewable.

Typical denylist entries: node_modules, vendor, dist, *.min.js, lockfiles, LICENSE, and any path that is a data dump. Re-read that list when a search result cites a generated file. Chunking is the last filter before you spend CPU on MiniLM. Review the top 50 largest chunks after every ingest. If any of them is a concatenated changelog or a vendored library, tighten the list before you celebrate recall. Store the splitter version next to the model name. If you change heading rules, re-chunk the corpus. Mixed old and new splits look like a bad encoder.

What to measure before you celebrate

Sample 20 real questions. For each, inspect the top 8 chunks before any generator runs. If the right heading is missing, your splitter dropped it or the cap was too small. If you see half a function, fix the code splitter. If you see license files, fix the skip list. Embedding model swaps will not fix those.

Keep a golden file of expected path plus heading for those 20 queries. Run it after ingest. This is cheaper than arguing about cosine thresholds.

Chunking is the last place to be clever with tokenizers and the first place to respect files. You can spin up an Ubuntu 24.04 instance on Netbay in Lucknow in under 60 seconds, clone a docs repo, and split it on disk you control — netbayhosts.in.