Auto-Renewal Failures: What Breaks When Certbot Stops

Let's Encrypt leaf certificates last ninety days. Certbot tries to renew at thirty days remaining. That gap is the whole operational story: if renewal is broken, you have about a month of green padlocks to notice, and then a hard fail. Browsers do not warn at T-minus-10 days for visitors. They fail on expiry. Auto-renewal is not "install certbot and forget." It is a timer, a still-valid challenge path, a disk that can write /etc/letsencrypt, and a reload that makes nginx open the new files.

This post is the Ubuntu systemd timer, the hooks that reload nginx, and the outages that look like "TLS died overnight" when the timer had been failing for weeks.

The timer is installed; confirm it actually runs

On Ubuntu, apt's certbot package ships certbot.timer. It fires twice a day with a randomized delay so the world does not renew at once. The service runs certbot -q renew. Quiet means you will not see mail unless you configure it. Check the units, the last run, and a dry-run.

systemctl status certbot.timer
systemctl list-timers certbot.timer
journalctl -u certbot.service -n 50 --no-pager
certbot renew --dry-run
ls -l /etc/letsencrypt/live/example.com/

If the timer is disabled, masked, or the VPS was cloned from an image that never enabled it, nothing renews. Snap certbot versus apt certbot also disagree on paths. Pick one installer and do not run both.

Renewal reuses the original challenge

certbot renew reads /etc/letsencrypt/renewal/example.com.conf. That file remembers --webroot, --nginx, or --dns-*. If you issued with HTTP-01 and later firewalled port 80 because "the site is HTTPS only," renewal breaks. Keep a server block on 80 that serves /.well-known/acme-challenge/ even if the rest redirects.

If you issued with a DNS plugin, the credentials file must still exist, still be readable by root, and still be authorized at the DNS provider. Rotating an API token and forgetting the ini file is a classic thirty-days-later outage.

Hooks matter. A successful renew that does not reload nginx leaves the old cert in memory until the next restart. Prefer --deploy-hook over a random cron.

certbot renew --deploy-hook "systemctl reload nginx"
# persist it in the renewal config or a global cli.ini:
# deploy-hook = systemctl reload nginx

Reload, not restart, unless you enjoy dropping connections. nginx reads the new pem files on reload. If ssl_certificate points at archive/example.com/cert1.pem instead of live/example.com/fullchain.pem, reload still serves the old file. Fix the path once.

What actually expires in production

These are the failures that show up after a quiet month:

• Port 80 closed after a hardening pass.
• nginx default site no longer includes the ACME location.
• Disk full on / or /var, so certbot cannot write new files.
• AppArmor or permissions on /etc/letsencrypt/live.
• Clock skew so the box thinks the cert is still young (NTP is a later post; enable systemd-timesyncd now).
• CAA record added at the registrar that excludes Let's Encrypt.
• A record moved to a new VPS; the old timer still runs and fails; the new VPS has no timer.

When it fails, users see NET::ERR_CERT_DATE_INVALID. openssl s_client -connect example.com:443 -servername example.com shows notAfter in the past. The fix is: restore the challenge path, certbot renew, reload nginx. Do not jump to a new certificate name unless the line is actually gone.

Alert before the browser does

Do not wait for customers. Probe notAfter and the timer.

A one-liner in cron or a small script: openssl x509 -enddate -noout -in /etc/letsencrypt/live/example.com/cert.pem. Alert if remaining time is under 20 days. Alert if certbot.timer is inactive. Alert if dry-run fails. That is enough for a single VPS.

Keep /etc/letsencrypt in your backup. Losing privkey.pem means re-issue, not restore. The account key under /etc/letsencrypt/accounts also belongs in backup so you do not create extra ACME accounts by accident.

Auto-renewal is a chain: timer, stored challenge, writable disk, valid DNS/HTTP proof, deploy-hook reload. Break any link and expiry is delayed, not prevented. On a Netbay Ubuntu VPS in Lucknow (DC01) you can enable the timer the same hour you issue, then confirm dry-run before you walk away — netbayhosts.in.