auditd: Auditing Events on Production Linux Servers

When something critical happens on a server — a config file silently changed, a binary replaced, an account suddenly added — you need an answer to "what happened and when." auditd is the Linux audit subsystem that answers it. It runs in the kernel, records events to an append-only log, and is immune to the application under attack modifying its own footprint. This guide sets up auditd on a production VPS, defines rules that watch the things that matter, and shows how to read the evidence without drowning in noise.

When auditd Earns Its Keep

auditd is not a monitoring dashboard and it is not fail2ban. It is forensic evidence. The moment you need to reconstruct an incident — a config checkout you did not make, a cron job that appeared, a setuid binary that changed — the audit trail is the record you need. Its value sits precisely where discretionary logging fails: it is kernel-level, so a compromised web service that deletes its own logs cannot erase what auditd already recorded. The cost is overhead and log volume, which means the skill is not enabling it but scoping it. Watch too much and you are overwhelmed; watch the right invariants and the trail is genuinely useful.

Install and Start the Service

auditd is a base package on Debian and RHEL families alike. Install it, verify the daemon is running, and confirm the kernel has audit support compiled in. On cloud images the audit daemon is usually present but not active, so enabling it at boot is the first real action.

sudo apt install -y auditd audispd-plugins
sudo systemctl enable --now auditd
sudo auditctl -s

The auditctl -s output tells you the enabled status and backlog limit. On a busy host, the backlog is where audit records buffer before writing to disk; if the backlog limit is too small, events get dropped under load, silently weakening your evidence. Raise it and monitor the drop counter, because a transcript with holes is not forensic evidence at all.

Write Rules That Watch the Right Things

The audit ruleset is where the judgment lives. Start with the two highest-value targets: watching identity and privilege changes, and watching specific sensitive files for writes. Identity changes are cheap to record and directly answer "was an account added or a sudo rule edited"; file watches on /etc/passwd, /etc/shadow, and your application config catch tampering at the source.

sudo auditctl -w /etc/sshd_config -p wa -k sshd_config
sudo auditctl -w /etc/passwd -p wa -k identity
sudo auditctl -w /etc/sudoers.d/ -p wa -k sudoers
sudo auditctl -a always,exit -F arch=b64 -S execve -k exec

The first three are file watches: -w names the path, -p wa records writes and attribute changes, and -k tags a searchable key. The execve rule captures every executed binary as a process event tagged exec — expensive, but invaluable for incident reconstruction because it records who ran what. Persist these in /etc/audit/rules.d as an .rules file so they survive reboots instead of living only in the live ruleset.

Persist Rules So They Survive Reboot

Live auditctl rules vanish on reboot. The persistent form lives in /etc/audit/rules.d as an .rules file and is applied by the audit service at startup. Write your rules there, and keep the syntax identical so what runs now is what runs on the next boot — an audit policy that decays on reboot is a policy you cannot rely on the morning of an incident.

sudo tee /etc/audit/rules.d/security.rules > /dev/null <<'EOF'
-w /etc/sshd_config -p wa -k sshd_config
-w /etc/passwd -p wa -k identity
-w /etc/sudoers.d/ -p wa -k sudoers
-a always,exit -F arch=b64 -S execve -k exec
EOF
sudo augenrules --load
sudo auditctl -l

augenrules compiles the rule fragments in /etc/audit/rules.d into the active ruleset, and auditctl -l lists what the kernel is actually enforcing. Keeping the live and persistent rulesets identical — and verifying with -l — is the whole difference between a policy you think you have and one you have.

Read the Evidence Without Drowning

The raw audit log is verbose; the skill is summarizing it. The ausearch tool queries by key, user, or time, and is the primary way to slice the record. The most useful query is by key, because good tagging turns "is there anything interesting" into "who touched this file."

sudo ausearch -k ident audited
sudo ausearch -k sshd_config -i --start today
sudo aureport -au

The -i flag converts numeric UIDs into names, and --start today narrows a day of noise to the events that matter. aureport -au summarizes authentication activity. The pattern to look for is the unexpected: a write to /etc/passwd outside a package update, an execve by a normally-silent service account, a login outside your known IP set. When you spot one, ausearch with a key and a time window gives you the exact reconstructable sequence.

Rotate and Protect the Audit Trail

An audit trail you cannot retain or does not survive the service being cleared is moot. Configure log rotation so records are kept long enough to cover an incident timeline, and protect the log directory from the very accounts you are auditing. auditd writes to /var/log/audit; keep its mode tight and, on a serious host, forward records to a separate sink where a local compromise cannot delete them.

sudo systemctl status auditd
sudo ls -la /var/log/audit/
sudo tail -n 5 /var/log/audit/audit.log

The append-only nature of the log, combined with filesystem protections and rotation tuned to your retention requirement, keeps the evidence intact exactly when an attacker would most want it gone. Relying on a local audit log is a solid first line; shipping copies off-host is the difference between a story and proof.

Takeaway

auditd gives production servers a memory: watch identity, privilege, and the sensitive files you depend on; persist the rules; and query by key when something looks wrong. Scope it tightly enough to stay usable and the audit trail turns any future incident into an answerable question. Netbay VPS plans hand you a clean Ubuntu or Rocky base where auditd can be installed and scoped before your workload ever goes live — netbayhosts.in.