Fixture-Based Evals for Production Ops Agents
Score ops agents against frozen fixtures for SQL safety, command allowlists, and citation honesty so a model change cannot silently break production habits.
NetBay journal
30 practical AI Agents articles from Netbay Engineering — written for operators running Linux and Windows VPS in India.
30 articles in this topic
Score ops agents against frozen fixtures for SQL safety, command allowlists, and citation honesty so a model change cannot silently break production habits.
Choose a cron script over an AI agent when the steps are known, the failure is binary, and a Linux VPS should not spend tokens to restart nginx.
Give each agent its own Unix user, env file, and scoped API keys so a leaked worker key cannot drain the billing account or rewrite git history.
Watch disk and CPU on a VPS and let the agent run only allowlisted fixes such as journal vacuum or nginx reload, logging every action it takes.
Record who called which tool with which args, hash the result, and keep a replayable JSONL trail so incidents have evidence rather than guesses.
Build a server-admin agent that never gets a raw shell: allowlisted binaries, a cwd jail, timeouts, and no pipes, on a Linux VPS in Lucknow.
Prove last night dump is restorable: check age, size, archive headers, and a throwaway restore so backup success is a test, not a file timestamp.
Wrap every model HTTP call with timeouts, jittered retries, and a circuit breaker so one slow provider cannot stall every agent process on the box.
Cap steps, tokens, and daily spend so a long-running ops agent on a Linux VPS cannot loop itself into an unbounded model-API bill overnight.
Force models to emit JSON Schema, then validate locally before any tool runs, so a stray fence or trailing sentence never crashes your agent loop.
Run a review agent on pull request diffs that comments on systemd, firewall, and secret leaks instead of style nits, then posts one GitHub review.
Stop judging ops agents by one lucky demo. Score JSONL traces against golden tasks so regressions show up before they restart the wrong systemd unit.