Automated Canary Deployments with Caddy Reverse Proxy and Prometheus Metrics

Deploying a major release directly to 100% of production users in an all-or-nothing cutover is high risk. Even comprehensive automated testing suites can miss edge cases triggered by real-world user payloads, exotic browser headers, or production database concurrency.

Canary deployments minimize release risk by exposing new code to a small, controlled fraction of live user traffic (e.g., 5% or 10%). By combining Caddy's dynamic HTTP reverse proxy with Prometheus metric observation, engineering teams can implement automated canary validation and instant rollback on a single VPS host without the complexity of a Kubernetes cluster.

Fractional Upstream Routing Architecture

In our lightweight canary architecture:

• Caddy Edge Proxy: Receives all incoming HTTPS requests. It uses weighted round-robin upstream balancing to direct 90% of requests to the stable baseline service (Port 3001) and 10% to the canary candidate (Port 3002).
• Prometheus Telemetry: Measures real-time HTTP response status codes, latencies, and error rates per upstream backend.
• Rollback Watchdog: A simple deployment script monitors canary error metrics for 5 minutes. If canary 5xx error frequency exceeds baseline by more than 1%, Caddy weight is immediately set back to 0%.

Configuring Weighted Upstreams in Caddyfile

Caddy supports weighted round-robin load balancing directly in its configuration:

# /etc/caddy/Caddyfile
api.example.in {
    reverse_proxy {
        to 127.0.0.1:3001 127.0.0.1:3002
        lb_policy weighted_round_robin 90 10
        lb_try_duration 2s
        fail_duration 5s
    }

    log {
        output file /var/log/caddy/access.log
        format json
    }
}

Canary Evaluation Script

During CI deployment, a lightweight script polls Prometheus and validates health:

#!/usr/bin/env bash
set -euo pipefail

CANARY_UPSTREAM="127.0.0.1:3002"
PROMETHEUS_URL="http://127.0.0.1:9090"

echo "Beginning 3-minute canary verification window..."
sleep 60

# Query Prometheus for canary error percentage
ERROR_RATE=$(curl -sG "$PROMETHEUS_URL/api/v1/query"   --data-urlencode 'query=sum(rate(http_requests_total{upstream="'"$CANARY_UPSTREAM"'",status=~"5.."}[1m])) / sum(rate(http_requests_total{upstream="'"$CANARY_UPSTREAM"'"}[1m]))'   | jq -r '.data.result[0].value[1] // 0')

echo "Current canary error rate: $ERROR_RATE"

# If error rate exceeds 1% (0.01), execute instant rollback
if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
    echo "CRITICAL: Canary error threshold breached! Reverting Caddy to 100% baseline..."
    curl -X PATCH "http://localhost:2019/config/apps/http/servers/srv0/routes"       -H "Content-Type: application/json"       -d '{"revert": true}'
    exit 1
fi

echo "Canary verification passed! Scaling canary to 100% production traffic."

Expand

Video Reference on Continuous Delivery & Canary Deployments

Explore continuous delivery patterns, canary deployments, and automated testing pipelines:

Canary deployments transform software releases from nail-biting cutover events into automated, calm, and mathematically safe operations.

Deploy agile web services on Netbay high-speed cloud VPS instances with instant IP provisioning at netbayhosts.in.