Break-Glass SSH: Second Key, Port, and Console

Every sshd change in this series can lock you out. A Match Address that forgot the bastion, an algorithm drop-in the client cannot speak, IdentitiesOnly pointing at the wrong file, a reverse tunnel that stole the only session. Break-glass access is the practice of assuming you will lock yourself out, and of keeping a second path that does not share failure modes with the first. A second key is not a copy of the first key. A second port is not security by obscurity alone. A serial-console mindset is how you recover when SSH itself is the thing that broke.

On a VPS you do not have a rack PDU. You have whatever out-of-band the provider exposes. Netbay gives you Web VNC into the instance. That is the console. Use it like a serial port: last resort, slow, always there, never the daily driver.

A second key that is not a backup copy

A copied id_ed25519 is one key in two files. If you leaked it, both files are leaked. Break-glass means a second key pair, generated on a different machine or a hardware token, with a long passphrase, stored offline. Its public half is in authorized_keys or issued as a longer-lived cert from the same CA. The daily key can be eight hours. The glass key can be thirty days, because you use it twice a year.

Name it in ~/.ssh/config as Host prod-glass with a different IdentityFile and IdentitiesOnly yes. Do not load it into the daily agent. Do not forward it. Do not put it on the CI runner. Test it on a calendar. An untested glass key is a paperweight.

If you use certificates, issue a glass cert with a principal like deploy-glass mapped through AuthorizedPrincipalsFile, so you can revoke daily certs without killing the emergency one. Put the serial in the runbook.

A second port so a Port directive cannot trap you

sshd can listen on more than one port. Port 22 and Port 2222 in the global config, or a second drop-in. Changing Port to 2222 and reloading while your client still aims at 22 is the classic lockout. Listening on both during a migration, then removing 22 only after the client config has moved, is the adult version.

sudo tee /etc/ssh/sshd_config.d/90-glass-port.conf > /dev/null <<'EOF'
Port 22
Port 2222
Match LocalPort 2222
    AllowUsers deploy
    MaxAuthTries 2
    AllowTcpForwarding no
    AllowAgentForwarding no
EOF
sudo sshd -t && sudo systemctl reload ssh
ss -lnt | grep -E "22|2222"

Match LocalPort 2222 lets the glass listener be stricter: fewer users, no forwarding, lower MaxAuthTries. Scanners will find 2222. L3/L4 filtering still applies. The port is not a secret; it is an independent bind so a botched Port 22 stanza does not close the only door.

Client side, Host prod-glass uses Port 2222 and the glass IdentityFile. Keep that stanza even when you are not migrating. The day you need it, you will not want to remember flags.

A firewall that only allows 22 from the bastion should also allow 2222 from the bastion, or the second port is fiction. Check ufw or nftables in the same change as sshd.

A console mindset, and Web VNC as the serial port

When sshd will not start, no port helps. sshd -t failed after a package upgrade, a disk filled so the daemon could not write, PAM broke, or you set ListenAddress to a NIC that no longer exists. That is console work.

On metal, you walk to the serial concentrator. On a Lucknow VPS, you open Web VNC from the Netbay panel, log in at the getty, and fix the file. Treat Web VNC like a serial console: it bypasses sshd, it is not for file transfer, and it is how you recover PermitOpen mistakes. Practice once. Set a password or a locked root account you actually know, or you will VNC into a box you cannot log in to. Key-only SSH with no local password and no console account is a lockout waiting for a bad drop-in.

sudo sshd -t
sudo journalctl -u ssh -n 80 --no-pager
sudo nano /etc/ssh/sshd_config.d/70-match.conf
sudo systemctl restart ssh
ss -lnt | grep ssh

Do this from VNC when SSH is already dead. Do not wait to learn nano or vim from a console you cannot paste into. Keep a printed copy of the last known-good drop-in, or a git repo you can clone from the console if the network still works.

Never make Web VNC the daily admin path. It is out-of-band. Leave it enabled as rescue, keep the OS account usable, and document which panel user can open it.

Rehearse the lockout

Pick a maintenance window. Open a spare SSH shell. Reload a deliberately slightly wrong config (MaxStartups 1:1:1 is mean; a wrong Match Address is enough). Confirm the spare shell still works. Confirm prod-glass on 2222 still works. Confirm Web VNC still gets a login. Then revert. A break-glass plan you have never failed over is a wish.

A short takeaway: daily key and port for work, glass key and port for when work breaks SSH, Web VNC for when SSH is gone. Independent paths, tested, documented. Spin up an Ubuntu 24.04 instance on Netbay in under 60 seconds, enable the second port, and click through Web VNC once so the rescue path is boring — netbayhosts.in.