systemd Units and journalctl: Inspecting Services Properly

systemd has become the default init system on nearly every mainstream Linux distribution, including Ubuntu, Debian, Fedora, and openSUSE. It is responsible for starting services, managing their lifecycles, and providing a unified logging system. Two concepts dominate day-to-day administration: units (how a service is defined) and journalctl (how you inspect its output).

What Is a systemd Unit?

A unit is a configuration file that describes a service, socket, timer, mount, or target. Service units end with .service. They live in /etc/systemd/system/ for local customizations and /usr/lib/systemd/system/ for package-provided units. When you override a unit, you normally create a drop-in directory to avoid clobbering the packaged file.

A basic unit file looks like this:

[Unit]
Description=My custom application
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/myapp --port 8080
Restart=on-failure
RestartSec=3
User=deploy
WorkingDirectory=/opt/app

[Install]
WantedBy=multi-user.target

The After and Wants directives set the dependency order, ensuring networking is available before the app starts. Restart=on-failure automatically restarts the service if it exits with a non-zero code. User=deploy runs the process as an unprivileged user instead of root, which is a crucial security hardening step.

Controlling Services with systemctl

The systemctl command controls units at runtime:

systemctl daemon-reload
systemctl start myapp
systemctl enable --now myapp
systemctl status myapp
systemctl restart nginx

Always run daemon-reload after editing a unit file so systemd picks up the changes. enable --now both creates the dependency to start at boot and starts the service immediately. status shows the process state, the main PID, memory usage, and a slice of the recent log output.

To check whether a service is active and enabled:

systemctl is-active nginx
systemctl is-enabled nginx
systemctl list-units --type=service --state=running

These commands are the backbone of every health check and monitoring script.

journalctl: Reading the Journal

journald stores structured logs from services, the kernel, and the boot process. The journalctl command queries them:

journalctl -u myapp
journalctl -u myapp --since "1 hour ago"
journalctl -u nginx -f
journalctl -p err -b

The first shows all logs for the myapp unit. The second filters to the last hour. The -f flag follows the log in real time, like tail -f. The last shows only error-level messages from the current boot with -p err combining priority filtering and -b restricting to the current boot.

To see logs from the previous boot, especially useful after a crash:

journalctl -b -1
systemctl list-boots

Querying Logs by Field

journald indexes logs by fields. You can query by unit, executable, priority, and more:

journalctl _PID=1234
journalctl _COMM=nginx --since today
journalctl -u myapp -o json-pretty

The -o json-pretty flag outputs logs as structured JSON, which makes it trivial to feed them into log aggregation pipelines or parse them programmatically.

Writing Logs from Your Own Scripts

Any output your scripts send to stdout or stderr is captured by the journal when the script runs as a systemd service. You can also write directly to the journal:

logger -t myapp "Backup completed successfully"
journalctl -t myapp --since "5 minutes ago"

The logger command sends a message to the journal with the tag myapp. You can then filter for that tag. This is the idiomatic way to add structured logging to shell-based services.

Takeaway

Master systemctl and journalctl and you will move from blindly restarting services to methodically diagnosing them. Run a service on a Netbay Linux VPS, inspect its journal, and build confidence with real systemd units at netbayhosts.in.