Advanced PromQL Patterns: SLI/SLO Calculations and Burn Rate Alerting

Traditional threshold alerting (e.g., paging on-call engineers when CPU exceeds 85% for 5 minutes) is the primary cause of on-call alert fatigue. High CPU utilization is often intentional during batch processing and rarely correlates directly with user-perceived downtime. Modern Site Reliability Engineering (SRE) practices replace static thresholds with Service Level Objective (SLO) burn rate alerting.

Prometheus 3.0 introduces significant PromQL enhancements, native high-resolution histograms, and improved subquery caching. By computing multi-window multi-burn-rate alerting rules, your monitoring system pages only when your application is actively consuming its error budget fast enough to breach customer service agreements.

SRE Fundamentals: SLI, SLO, and Error Budgets

To configure meaningful alerts, define your Service Level Indicators (SLI):

1. **Availability SLI**: The ratio of successful HTTP requests (non-5xx) divided by total valid requests over a rolling 30-day window.
2. **SLO Target**: 99.9% availability (an error budget of 0.1%).
3. **Burn Rate**: The rate at which your application consumes its error budget.

- 1x Burn Rate = Error budget exhausts exactly in 30 days. - 14.4x Burn Rate = Error budget exhausts in 2 days (requires an immediate page within 1 hour). - 6x Burn Rate = Error budget exhausts in 5 days (requires a warning ticket within 6 hours).

Writing Advanced PromQL Multi-Window Rules

A naive 1-hour burn rate alert triggers false alarms during temporary network blips that immediately recover. Conversely, a 6-hour alert responds too slowly to catastrophic outages. SRE best practice uses multi-window alerting: both a short window (to detect immediate burn) and a long window (to confirm sustained consumption) must trigger simultaneously.

Create /etc/prometheus/rules/slo_alerts.yml:

# /etc/prometheus/rules/slo_alerts.yml
groups:
  - name: api_slo_alerts
    rules:
      # Recording rules for 5m, 30m, 1h, and 6h error rates
      - record: job:api_requests_errors:rate5m
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

      - record: job:api_requests_errors:rate30m
        expr: sum(rate(http_requests_total{status=~"5.."}[30m])) / sum(rate(http_requests_total[30m]))

      - record: job:api_requests_errors:rate1h
        expr: sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h]))

      - record: job:api_requests_errors:rate6h
        expr: sum(rate(http_requests_total{status=~"5.."}[6h])) / sum(rate(http_requests_total[6h]))

      # Critical Page: 14.4x burn rate (2% budget consumed in 1 hour)
      - alert: ApiHighErrorBudgetBurnPage
        expr: |
          (job:api_requests_errors:rate1h > (14.4 * 0.001))
          and
          (job:api_requests_errors:rate5m > (14.4 * 0.001))
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "API is burning 2% of error budget in 1 hour."
          description: "Current error rate is {{ $value | humanizePercentage }}. At this rate, monthly error budget exhausts within 48 hours."

      # Warning Ticket: 6x burn rate (5% budget consumed in 6 hours)
      - alert: ApiModerateErrorBudgetBurnTicket
        expr: |
          (job:api_requests_errors:rate6h > (6 * 0.001))
          and
          (job:api_requests_errors:rate30m > (6 * 0.001))
        for: 15m
        labels:
          severity: ticket
        annotations:
          summary: "API is burning 5% of error budget in 6 hours."
          description: "Sustained error rate is {{ $value | humanizePercentage }}. Inspect services before the monthly budget is exhausted."

Notice the and clause: the alert fires only when both the 1-hour and 5-minute rates exceed 14.4x the target error budget (0.1%), eliminating transient spikes.

Expand

Latency SLOs Using Native Histograms

In Prometheus 3.0, native high-resolution histograms replace clumsy exponential bucket boundaries. Calculate p99 latency SLOs with precision:

# Calculate 99th percentile latency over 15 minutes
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[15m])) by (le))

# Ratio of requests faster than 250ms threshold (Latency SLI)
sum(rate(http_request_duration_seconds_bucket{le="0.25"}[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))

Validating Alert Rules with promtool

Always validate your alerting syntax and run unit tests using promtool prior to deploying changes:

# Verify Prometheus syntax
promtool check rules /etc/prometheus/rules/slo_alerts.yml

# Reload Prometheus configuration without restarting
curl -X POST http://localhost:9090/-/reload

SLO burn rate alerting ensures on-call engineers are paged only when real user experience and contractual commitments are endangered.

Deploy scalable Prometheus monitoring infrastructure on Netbay Linux VPS in under 60 seconds with High-Speed SSD in Lucknow DC01 — netbayhosts.in.