Agent Identity, API Keys, and Secret Isolation

An agent without an identity is root with a prompt. The model will eventually follow a poisoned ticket, a hostile web page, or a confused supervisor and call a tool you intended for a different worker. Isolation is not a guardrail paragraph. It is a Unix user, a systemd EnvironmentFile, a scoped third-party key, and a filesystem the process cannot walk out of.

This post ties the series together on one Lucknow VPS. The supervisor, the billing worker, the rag worker, the GitHub hook, and the cron reporter must not share a uid or a .env. Intel Xeon Platinum does not care how many users you add. A single leaked LLM_API_KEY that can also push to GitHub does.

One role, one user, one unit

Create system users that cannot log in. Give each a home under /var/lib/agents/ROLE for state, not for secrets. Secrets live in /etc/agents/ROLE.env, mode 0400, owner root, group ROLE, and the unit sets Group=ROLE so the process can read the file via EnvironmentFile. The process cannot write the file.

sudo useradd --system --home /var/lib/agents/rag --shell /usr/sbin/nologin agent-rag
sudo useradd --system --home /var/lib/agents/triage --shell /usr/sbin/nologin agent-triage
sudo mkdir -p /var/lib/agents/rag /var/lib/agents/triage /etc/agents
sudo chown agent-rag:agent-rag /var/lib/agents/rag
sudo chown agent-triage:agent-triage /var/lib/agents/triage
sudo install -m 0440 -o root -g agent-rag /dev/null /etc/agents/rag.env
sudo install -m 0440 -o root -g agent-triage /dev/null /etc/agents/triage.env

Put LLM_API_KEY, GITHUB_WEBHOOK_SECRET, and the GitHub App PEM in different files. The rag worker never needs the webhook secret. The hook never needs the billing provider token. If a tool is not imported by that role, the key for that tool must not be in the env.

Scope the third-party keys as if they will leak

They will. Treat leak as the common case.

• Model provider: a key per role, with a spend cap in the vendor console, not one org key in every unit.
• GitHub: an App installation with issues write, no contents, no administration. PEM on disk, installation tokens at runtime.
• Helpdesk: read-only for the supervisor, comment-only for the worker.
• Object or DNS APIs: do not put them on an agent box unless that agent is the thing that rotates records.

Rotate by writing a new file and restarting the unit. Do not overwrite in place and hope the process re-reads. systemd EnvironmentFile is read at start. After rotation, grep the audit log for the old key's hash (you hashed it on ingest, right) and confirm silence.

# /etc/systemd/system/agent-rag.service
[Unit]
Description=rag worker
After=network-online.target

[Service]
Type=simple
User=agent-rag
Group=agent-rag
EnvironmentFile=/etc/agents/rag.env
WorkingDirectory=/opt/agents
ExecStart=/opt/agents/.venv/bin/python /opt/agents/rag_worker.py
Restart=on-failure
RestartSec=5
ProtectHome=yes
ProtectSystem=strict
PrivateTmp=yes
NoNewPrivileges=yes
ReadWritePaths=/var/lib/agents/rag /var/log/agents
CapabilityBoundingSet=
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6

ProtectSystem=strict plus ReadWritePaths is the local equivalent of "this worker cannot rewrite /opt/agents and cannot read /etc/agents/triage.env." Combine it with the circuit breaker and the audit wrap from earlier posts.

What the model is allowed to know

Never load env into the prompt "for debugging." Never log os.environ. The redactor from the audit post must run on tool args before the model sees results, too: a lookup_customer tool that returns a PAN or a session cookie needs a projection. Identity is also data minimization.

If two roles must share a queue file, put the queue in a group-readable directory with a dedicated group agent-queue, and add both users to that group. Do not chmod 0777 /var/lib/agents. The lock file should be the same group. This is the only shared surface the handoff post needs.

A small proof you can run after deploy

As root, confirm the rag unit cannot read the triage env and cannot write /opt/agents:

sudo systemd-run --uid=agent-rag --gid=agent-rag   --property=ProtectSystem=strict   --property=ReadWritePaths=/var/lib/agents/rag   /usr/bin/test -r /etc/agents/triage.env; echo rag_read_triage:$?
sudo -u agent-rag test -w /opt/agents && echo writable || echo opt_readonly

You want a non-zero on the first and opt_readonly on the second. If either fails, the unit is a shared account in disguise. L3/L4 filtering on the public interface does not save you from a worker that can read every env file on disk.

Takeaway: identity is a Unix user plus a scoped key plus a unit that cannot wander. Split secrets by role the same way you split envelopes by kind. You can lay this out on Ubuntu 24.04 in Lucknow DC01 from Netbay in under 60 seconds — netbayhosts.in.