Alertmanager: Page Yourself Without Alert Fatigue

Dashboards do not wake you. Alerts do. The failure mode on a single VPS is not missing a graph; it is a phone that screams all night until you mute everything. Alertmanager exists to take firing alerts from Prometheus, group them, wait out flaps, and send one message you will actually read. If you pipe every PromQL threshold straight to email, you will ignore the mailbox within a week.

The design for one node is small: Prometheus evaluates rules, Alertmanager receives them on localhost:9093, and a single receiver sends to email or a chat webhook. No HA pair. No paging vendor. Group by alertname. Inhibit disk-full child alerts when the node is down. Use for: 5m or 10m so a 30 second CPU spike is not a page.

Write few rules, each with a human summary

Start with four alerts and no more: instance down, disk almost full, memory almost gone, probe failed (once blackbox exists). Each rule needs a for duration, a severity label, and an annotation that states what to do. An alert named HighCPU with no for clause is how fatigue starts. CPU is bursty. Disk fill is not. Page on the latter, graph the former.

groups:
  - name: node
    rules:
      - alert: InstanceDown
        expr: up{job="node"} == 0
        for: 2m
        labels:
          severity: page
        annotations:
          summary: node_exporter scrape failed for 2 minutes
      - alert: DiskAlmostFull
        expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.10
        for: 10m
        labels:
          severity: page
        annotations:
          summary: root filesystem under 10 percent free
      - alert: MemoryLow
        expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.10
        for: 10m
        labels:
          severity: ticket
        annotations:
          summary: MemAvailable under 10 percent for 10 minutes

Severity page versus ticket is the whole routing model on a one-person team. Pages go to the phone. Tickets go to morning email. If everything is a page, nothing is.

Route, group, and wait

Alertmanager.yml should group by alertname and instance, wait 30s to batch, and repeat pages every 4h not every 5m. group_wait lets a disk alert and an inode alert arrive as one notification if they trip together. repeat_interval that is too short trains you to swipe away.

global:
  resolve_timeout: 5m
route:
  receiver: email
  group_by: ['alertname', 'instance']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity = page
      receiver: pager
      repeat_interval: 1h
    - matchers:
        - severity = ticket
      receiver: email
      repeat_interval: 12h
inhibit_rules:
  - source_matchers:
      - alertname = InstanceDown
    target_matchers:
      - severity = ticket
    equal: ['instance']
receivers:
  - name: pager
    webhook_configs:
      - url: 'http://127.0.0.1:8081/alert'
  - name: email
    email_configs:
      - to: 'ops@example.com'
        from: 'alertmanager@example.com'
        smarthost: 'localhost:25'

The inhibit rule is the fatigue killer: when the node is down, do not also page about memory, disk, and every other timeseries that went stale. You already know the box is unreachable. Fix the scrape, then let the rest re-fire if they are still true.

Silence from the CLI, not from hope

Silences have a start, an end, and a matcher. Use them for planned reboots and backups that fill the disk for 20 minutes. Do not delete rules because a backup is noisy; silence DiskAlmostFull for the backup window. amtool from the Alertmanager package can add a silence over SSH so you do not need the UI exposed.

Keep a runbook next to each alert name. InstanceDown means check systemd status for node_exporter, then Prometheus, then whether the VPS itself answers ping. DiskAlmostFull means df -h, then journalctl vacuum or logrotate, then find the directory that grew. If the annotation cannot name the first command, the alert is unfinished.

On a Lucknow VPS with L3/L4 DDoS filtering, do not alert on public NIC drops. The filter is supposed to drop junk. Alert on your application accept queue and on exporter UP, not on raw drop counters that move every time someone scans the IPv4 internet. Write the first command for each alert in the annotation so 2 a.m. you does not open a wiki. If you cannot name that command, the rule is not ready to page anyone.

Bind it to localhost and unit-test the rules

promtool check rules and amtool check-config catch YAML mistakes before a restart. Point Prometheus at Alertmanager with alerting.alertmanagers static_configs 127.0.0.1:9093. Restart both under systemd. Trigger a fake InstanceDown by stopping node_exporter for three minutes in a maintenance window and confirm one page, not twelve. That drill is the only proof the pipeline works.

Takeaway

Four rules, a for duration, grouping, and one inhibit against InstanceDown will page you without teaching you to ignore the phone. Run Alertmanager next to Prometheus on a Netbay VPS and keep both on localhost — netbayhosts.in.