journalctl Deep Dive: Querying Logs Like a Professional

On any systemd system, every service log, kernel message, and boot record lands in one indexed, binary journal, and journalctl is the query language for it. Most people use it like tail with extra steps. Used well, it answers in seconds questions that used to mean grepping half-rotated text files in the middle of the night. This post builds the query habits that separate the two.

Start From a Time Window

The fastest way to shrink any log problem is to bound it in time, and journalctl takes human-friendly arguments to do exactly that.

journalctl --since "2026-04-09 08:00" --until "2026-04-09 09:30"
journalctl --since today
journalctl --since -2h
journalctl -n 100                # last 100 lines
journalctl -f                    # follow, like tail -f

Every filter journalctl accepts combines with AND, so --since today -u nginx -p err returns today's errors from nginx and nothing else. Get the window right first and the rest of the query usually writes itself.

Slice by Boot, Unit, and Priority

The -b flag scopes output to a boot, which is usually the first question in an incident: has this ever worked on this machine?

journalctl -b                    # this boot
journalctl -b -1                 # previous boot
journalctl -b -1 -u nginx        # nginx from the previous boot
journalctl -u nginx -u php8.3-fpm
journalctl -p err..alert --since today

-b -1 is the killer feature people discover last: when a server came up broken after a reboot, the previous boot's logs are still in the journal and directly comparable to the current one. Priorities run from emerg (0) to debug (7) following the classic syslog levels, and -p err shows priority 3 and above — a quick way to hide the noise without hiding the signal.

Two more filters round out the toolbox. -g takes a pattern and greps server-side, so journalctl -u nginx -g "upstream.*timed out" skips the pipe to grep entirely and is noticeably faster on large journals. And when you need a stable pointer to a specific place in the log, --show-cursor prints a cursor string that --after-cursor accepts, giving you a resume point that survives rotation and reboots — the right tool for handing an incident from one engineer to the next.

Field Matches: Query the Journal Like a Database

Everything in the journal is stored as key-value fields, and any field can be a filter. These are the ones worth memorizing.

journalctl _PID=4412
journalctl _SYSTEMD_UNIT=nginx.service
journalctl _COMM=sshd
journalctl SYSLOG_IDENTIFIER=kernel
journalctl _UID=0 _COMM=sudo --since today

Fields prefixed with an underscore are trusted metadata journald captured from the kernel: _PID, _UID, _GID, _COMM, _EXE. Names without the underscore come from the log message itself, such as SYSLOG_IDENTIFIER. Tab completion works here too: type journalctl _COMM= and press tab twice to list every value present in the journal. That turns what-is-even-writing-to-my-logs mysteries into a one-keystroke discovery.

Output Modes and Housekeeping

journalctl -u nginx -n 5 -o json-pretty
journalctl -u nginx -o short-iso
journalctl -k                    # kernel messages via the journal
journalctl --disk-usage
sudo journalctl --vacuum-time=30d
sudo journalctl --vacuum-size=2G

The -o json-pretty mode emits one JSON object per entry, which is what you want when piping logs into a script or a log shipper. short-iso gives ISO timestamps that sort correctly in files. On the housekeeping side, journald caps itself by default (commonly around 10 percent of the filesystem), and --disk-usage plus --vacuum-size or --vacuum-time let you enforce your own ceiling on a small VPS disk without editing journald.conf.

One caveat that trips up newcomers: if journalctl shows almost nothing, the machine may be running with a volatile journal. Check journalctl --disk-usage first, and make sure /var/log/journal exists so logs persist across reboots.

One more flag earns a place in muscle memory: --utc prints every timestamp in UTC, which matters the moment an incident spans machines in different timezones. Pair it with -o verbose when you want to see every field an entry carries — it is also the fastest way to discover which field names exist for the matches from the previous section.

Takeaway: bound the time, slice by boot and unit, drill into fields, and let the -o modes do the parsing for you. That is the entire skill — everything else is remembering these flags exist.

SSH into a Netbay VPS and try these against a live journal — your first instance is up in under 60 seconds at netbayhosts.in.