Blackbox Exporter HTTP and TLS Probes on a VPS

node_exporter tells you the box is alive. It does not tell you the website answers 200, that TLS handshakes, or that the certificate has 12 days left. Blackbox exporter is the probe: Prometheus asks it to hit a URL or a TCP port the way a client would, then scrapes the result as metrics. On a VPS you operate, this is how you notice nginx is up while the application behind it is not.

Run blackbox_exporter on localhost:9115. Do not let the public internet use it as an open proxy. A probe exporter that accepts arbitrary targets is an SSRF gift. Static configs in Prometheus list the only URLs it may hit.

Modules for HTTP and TLS, nothing clever

The module file defines how to probe. http_2xx is the default: GET, follow redirects, succeed on 2xx. A TLS module should fail if the chain is invalid or the name does not match. Add a second HTTP module that checks a specific header or a JSON path only if you already need it. Most sites need status code, TLS, and latency.

modules:
  http_2xx:
    prober: http
    timeout: 8s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200]
      method: GET
      follow_redirects: true
      fail_if_not_ssl: true
      tls_config:
        insecure_skip_verify: false
  tcp_connect:
    prober: tcp
    timeout: 5s
  icmp_ping:
    prober: icmp
    timeout: 5s

ICMP needs CAP_NET_RAW and is often blocked anyway. Prefer TCP connect to 443 over ping. Ping succeeding while HTTPS fails is the common lie. Probe the protocol users speak.

Prometheus relabel so one job covers many URLs

The scrape job does not target blackbox_exporter as if it were a node. It targets the URLs you care about, then relabels __address__ to 127.0.0.1:9115 and keeps the original URL as a parameter. That pattern is in the upstream docs for a reason. Copy it once and add names.

scrape_configs:
  - job_name: blackbox-http
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://example.com/health
          - https://example.com/
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

Probe the public hostname, not localhost, if TLS certificates are issued for the public name. A probe to 127.0.0.1 will fail name verification or skip the reverse proxy path that users hit. If nginx terminates TLS and the app listens on 127.0.0.1:3000, you want both: HTTP to the public URL, and TCP to the app port. The first is user-visible. The second tells you whether the proxy is papering over a dead backend.

What to graph and what to page

probe_success is the binary. probe_duration_seconds is the user-ish latency, including DNS if the module does DNS. probe_http_status_code explains a failure. probe_ssl_earliest_cert_expiry is a Unix timestamp; subtract time() and divide by 86400 for days left. Page when probe_success is 0 for 2 minutes. Page when certificate days left drop under 14. Graph duration so you see a slow climb before it becomes a timeout.

Do not page on a single 8 second timeout. Networks jitter. for: 2m at a 30s scrape is four failures, which is a real outage on a site you own. If the probe runs from the same VPS it is probing, you will not see the host going off the network. That is a later post about external checks. Blackbox on-box still catches nginx misconfig, app crashes, and certificate expiry, which are the majority of self-inflicted outages.

On High-Speed SSD and Xeon Platinum, the exporter is idle. The cost is in scrape cardinality if you add 50 URLs. Keep the target list to the URLs you would check by hand: homepage, health, maybe an API ping. Internal admin panels can wait.

Fail closed on TLS

insecure_skip_verify: false is the point. A probe that ignores certificate errors will not warn you about a broken chain after a renewal. fail_if_not_ssl: true stops a silent downgrade to HTTP. If you terminate TLS at a reverse proxy, the public probe still validates the public cert. That is the cert browsers see, and the one Let's Encrypt will let expire if cron failed.

When a probe fails, the first SSH commands are curl -sI https://example.com/health and openssl s_client -connect example.com:443 -servername example.com. Blackbox is those two commands on a timer. If you cannot reproduce a failure with curl from the box, look at DNS, then at whether Prometheus is probing a different URL than you think.

Takeaway

Blackbox exporter turns curl and openssl s_client into time series. Probe the public HTTPS URL, page on probe_success and certificate expiry, and keep the exporter on localhost. You can run it beside Prometheus on a Netbay VPS in Lucknow — netbayhosts.in.