Maintaining a Rolling-Release Server: Arch Update Strategies That Work

Arch has no version numbers because the version is always now. That is precisely why some teams choose it for servers: there are no release migrations, no distro upgrades every two years, no graveyard of old apt sources. It is also why neglected Arch boxes fail: nothing in the default install stops you from ignoring the system for eight months, and the bill arrives all at once. What keeps a rolling-release server healthy is a small set of boring policies applied consistently, and this post is that set of policies.

Pick a Cadence and Enforce It

Choose a maintenance window, weekly works well for most fleets and biweekly is acceptable, and update every box inside it. The dangerous pattern is not low frequency, it is drift: one box misses three windows and suddenly its toolchain is months behind the repositories. Put the window in the calendar and treat a missed window as an incident review question, not a rounding error. Then measure staleness instead of trusting memory.

# When did this machine last get a real upgrade?
grep 'starting full system upgrade' /var/log/pacman.log | tail -3

# What would an upgrade do right now? Dry run against current databases
pacman -Sup

The pacman log also records every transaction, which makes it a decent audit trail when you need to know what changed before an incident.

Snapshot, Then Upgrade

Before big merges, meaning kernel series jumps, openssl majors, anything that touches a database layer, take a snapshot from your provider's panel or API. It is a thirty-second insurance policy that beats any in-guest rollback tooling you could configure in the remaining hours. For single-package regressions, pacman's own cache plus the Arch Linux Archive are usually enough. Keep the snapshot list pruned too, because a rollback artifact you cannot actually restore from is just storage.

# Package-level rollback from the local cache
pacman -U /var/cache/pacman/pkg/openssl-3.3.1-1-x86_64.pkg.tar.zst

# Older builds live in the Arch Linux Archive if the cache was trimmed
pacman -U https://archive.archlinux.org/packages/o/openssl/openssl-3.3.1-1-x86_64.pkg.tar.zst

Read the News Before You Click Y

Arch publishes breaking-change notes at archlinux.org/news, and significant migrations get an entry there before they hit the mirrors. PostgreSQL major versions need a dump and restore before the new server starts. Keyring rotations sometimes need an explicit pacman-key step. Two minutes of reading prevents the classic 3 a.m. surprise where an upgrade finishes cleanly and a service still refuses to start. Keep a bookmark to the news feed and read it during the window, before the transaction, not after. Then run the upgrade and merge whatever pacman left behind.

# Interactive .pacnew review: vimdiff, sdiff, or plain diff
pacdiff

# Or just list them if you prefer to review by hand
find /etc -name '*.pacnew' -o -name '*.pacsave'

Automate the Boring Half, Not the Judgment

An upgrade transaction is fully scriptable; the judgment around it is not. Unattended full upgrades on Arch are controversial because pacman will happily apply a migration that expects a human to intervene afterwards. A pragmatic middle ground exists: automate the transaction inside the maintenance window, reboot only when a new kernel is actually waiting, and page yourself if the transaction fails.

# /etc/systemd/system/nightly-upgrade.timer
[Unit]
Description=Run nightly upgrade inside the maintenance window

[Timer]
OnCalendar=*-*-* 03:30:00
Persistent=true

[Install]
WantedBy=timers.target

# /usr/local/bin/nightly-upgrade.sh
#!/bin/bash
set -eu
pacman -Syu --noconfirm
# Reboot only when the installed kernel is newer than the running one
newest=$(ls /usr/lib/modules | sort -V | tail -1)
if [ "$newest" != "$(uname -r)" ]; then
  systemctl reboot
fi

If you would rather keep humans in the loop, flip the script into report-only mode: run pacman -Sup, mail the output, and let the on-call person trigger the real upgrade during the window. Both designs work; what does not work is having neither.

Rebuild Instead of Resurrect

If a box has skipped more than a couple of months, or has drifted into hand-edited mystery state, do not nurse it through a mega-upgrade. Rebuild it. Export the explicit package list, stand up a fresh instance, and let your config management or setup scripts take it from there. Arch makes this cheap because there is no release version to match and no migration path to follow. Write the rebuild procedure before you need it; the night you need it is the wrong time to discover what was hand-configured.

# Export what was explicitly installed, then replay it on a fresh box
pacman -Qqe > /root/pkglist.txt
pacman -S --needed - < /root/pkglist.txt

Takeaway

A rolling-release server stays healthy when the update loop is small, scheduled, and boring: check the news, snapshot, upgrade, merge configs, restart services, reboot if the kernel changed. Missing one window occasionally is fine; missing two in a row is how rescue-ISO stories start.

Netbay's Linux VPS plans boot standard distro images in under 60 seconds, which makes rebuild-instead-of-rescue a genuinely cheap decision — netbayhosts.in.