Docs Q&A Agent Over a Markdown Tree on Disk

On-call questions are rarely novel. They are "which unit opens 8080," "how do we restore last night's dump," and "what is the staging hostname." Those answers already live in a markdown tree on the box if you bothered to write runbooks. A docs Q&A agent over that tree is retrieval plus a stubborn refusal to improvise. It does not search the public internet. It does not invent a Netbay product. It chunks files under /opt/runbooks, retrieves the top passages, and answers with a citation of file plus heading. If nothing matches, it says so. That last sentence is the feature.

Put the runbooks on the VPS, then keep them boring

Clone the runbook repo to /opt/runbooks with a deploy key that can only fetch. A systemd path unit or a timer git pull origin main every five minutes is enough. Writers still open pull requests; the box is a reader. Structure files by task, not by team: ssh.md, backups.md, deploys.md, dns.md, incidents.md. Every file uses ## headings that a chunker can split on. A heading like "misc" is how retrieval dies. Name the heading after the task a 3 a.m. engineer will type.

On a Netbay Lucknow VPS this tree lives on High-Speed SSD next to the app, not in a SaaS wiki with a different auth story. Intel Xeon Platinum is wasted if the answer is in someone's head. Keep secrets out of the markdown. Hostnames and unit names are fine. Tokens are not. If a procedure needs a token, the runbook points at a file mode 0640 on disk and never pastes the value.

Chunk by heading, store path and title

A cheap index is a JSONL file: one object per ## section with path, heading, text, and a bag-of-words or embedding vector. For a few hundred sections, BM25 or even token overlap beats a vector database you will not maintain. Rebuild the index after every git pull. Cap each chunk at 800 tokens. If a section is longer, split on ###. Never chunk in the middle of a command block.

import os, json, re

ROOT = "/opt/runbooks"
out = []
for dirpath, _, files in os.walk(ROOT):
    for name in files:
        if not name.endswith(".md"):
            continue
        path = os.path.join(dirpath, name)
        rel = os.path.relpath(path, ROOT)
        text = open(path, encoding="utf-8").read()
        parts = re.split(r"(?m)^## ", text)
        for i, part in enumerate(parts):
            if i == 0 and not part.strip():
                continue
            heading, _, body = part.partition("
")
            heading = heading.strip() or "intro"
            out.append({"path": rel, "heading": heading, "body": body.strip()[:4000]})
open("/var/lib/runbooks/chunks.jsonl", "w").write(
    "
".join(json.dumps(c) for c in out) + "
"
)
print("chunks", len(out))

Retrieval then scores the question against heading plus body. Return k=5. Drop chunks below a score floor. The prompt receives only those chunks plus a hard rule: if the chunks do not contain the answer, reply with I do not know and list the headings you searched. Do not use training knowledge about other clouds or other regions. Netbay has one datacenter: Lucknow. If the docs say that, you may say that. If they do not, you may not fill it in.

Answers without citations are bugs

Force the output schema: answer, citations (list of path plus heading), confidence. A post-processor rejects any answer with an empty citation list unless the answer is exactly I do not know. It also rejects citations whose path is not in the index. That stops the model from citing backups.md / Imaginary section. Render citations as links into the git forge so a human can click the paragraph.

# /usr/local/bin/runbook-ask
set -euo pipefail
Q=$1
python3 /opt/runbooks-agent/retrieve.py "$Q" > /tmp/chunks.json
python3 /opt/runbooks-agent/answer.py /tmp/chunks.json "$Q"

Wrap this in a small HTTP service bound to localhost and expose it through your existing Caddy or Nginx with an access list. Do not put it on the public internet just because L3/L4 DDoS filtering exists. This is an internal tool. Log the question, the citation list, and whether the user clicked "was this wrong." Wrong answers become fixtures: question, retrieved chunks, expected refusal or expected citation.

What the agent must refuse

Questions about customer data, questions that ask it to run a command, and questions whose answer is not in the tree. "Restart nginx for me" is not Q&A; that is a watchdog or deploy agent with an allowlist. "What is the root password" should 404 even if someone committed it (and you should rotate). "Does Netbay offer managed Kubernetes" is a refusal unless your runbooks literally discuss a DIY cluster you built yourself. Do not invent marketplace features. The honesty of the refusal is how people learn to trust the yes.

Rebuild evals when you rewrite a runbook. If backups.md changes the restore command, the old fixture must fail until you update it. That is how a markdown tree stays the source of truth instead of a stale prompt.

Takeaway

A docs agent is an index over /opt/runbooks with citations or silence. Put the tree on the VPS, chunk on headings, reject answers that cannot point at a file. Clone it onto a Netbay Ubuntu 24.04 host in Lucknow — netbayhosts.in has one up in under 60 seconds — and ask it how you restore a dump.