journald as the Source of Truth, Not Log Files

On a modern Ubuntu VPS, services already talk to the journal. nginx, sshd, your unit files, cron, sudo, kernel lines: they land in journald whether you also write a text file or not. The mistake is to treat /var/log/*.log as the source of truth and the journal as a weird binary cache. Reverse that. The journal is the indexed store. Files are optional exports for the one tool that still cannot read journalctl.

This is an observability choice. If you grep four files and miss the unit that failed because it never opened a log file, you do not have logs. You have leftovers. Make the journal persistent, cap it, query it by unit and priority, and only then decide whether Promtail ships it.

Make the journal survive reboot

The default on some images is volatile: logs live in /run/log/journal and die on reboot. That is useless for incidents you notice after a crash. Persistent journals live in /var/log/journal. Create the directory or set Storage=persistent in journald.conf and restart systemd-journald. Confirm with journalctl --list-boots that boot minus one exists.

[Journal]
Storage=persistent
Compress=yes
SystemMaxUse=500M
SystemKeepFree=1G
MaxRetentionSec=14day
ForwardToSyslog=no

SystemMaxUse keeps the journal from eating High-Speed SSD space on a small plan. SystemKeepFree refuses to write the last gigabyte so a noisy unit cannot take the OS down. MaxRetentionSec is a second ceiling. Vacuum is not optional on a VPS where the root filesystem is also the log filesystem.

Query by fields, not by filenames

journalctl is a database client. -u nginx.service, -u ssh, -p err, --since "1 hour ago", _COMM=python3, _PID= are filters, not grep flags you hope match a line format. -o json-pretty is how you feed a script. -o short-iso is how you paste into a ticket. -f is tail. --disk-usage is the budget.

journalctl --list-boots
journalctl -u nginx.service -p err --since "1 hour ago"
journalctl _COMM=sshd -o short-iso --since today
journalctl --disk-usage
journalctl --vacuum-size=300M

When a deploy fails, start with journalctl -u myapp.service -e, not with ls /var/log. The unit file StandardOutput=journal is the contract. If you set StandardOutput=append:/var/log/myapp.log you just opted out of fields, rotation policy, and the one place you already vacuum. Dual logging (journal plus a file) is how disks fill twice.

Units you should always be able to name

sshd, cron or systemd timers, nginx or caddy, the application unit, prometheus, grafana-server, and docker if you use it. Practice pulling 20 lines from each. Add a drop-in if the application logs to stdout: that is already the right design. Do not invent a log directory because a tutorial from 2012 did.

Kernel lines (journalctl -k) explain OOM kills, I/O errors, and NIC flaps. After an OOM, the application unit shows an exit and the kernel shows which PID was chosen. Both belong in the incident note. If you only read the app file, you will blame the framework.

Rate limiting in journald will drop floods. That is a feature on a VPS under a noisy debug flag. If you need every line of a debug session, raise RateLimitBurst temporarily, then put it back. A debug flood that fills 500 MB in ten minutes is a self-inflicted outage.

One pipeline out, not five formats

If you later run Promtail, scrape the journal directly. Do not scrape /var/log/syslog and the journal and docker json-file. Three pipelines of the same stdout is how you pay for Loki three times and still cannot tell which copy is late. rsyslog can still write a file if a compliance tool demands text; keep that as an export, not as the live path.

Permissions: the systemd-journal group can read the journal. Add your operator user to that group so you are not root just to tail logs. Backup of /var/log/journal is usually wasted; logs are not the restore target, configs and data are. Keep journald.conf in the same Git repo as prometheus.yml.

On Intel Xeon Platinum with a 2 GB plan, a 500 MB journal cap is plenty for two weeks of a quiet web app. Raise it only after --disk-usage shows you actually wrap in days, not because bigger feels safer. Bigger logs you never query are a disk risk, not observability.

Takeaway

Persistent journald with a size cap is the log store a VPS already has. Query by unit, vacuum on purpose, and do not dual-write files that you will forget to rotate. Follow this on a Netbay Ubuntu 24.04 instance in Lucknow — netbayhosts.in.