mTLS with Client Certificates: What Proxies Need to Know

Client certificates flip the trust model of TLS. The server no longer proves its identity alone; the client must also present a certificate signed by a CA the server trusts. This is mutual TLS, mTLS, and it is the cleanest native way to authenticate machines rather than humans at the protocol level.

How mTLS works

The handshake extends the normal TLS flow. After the server presents its certificate, the server requests a client certificate, the client sends its chain, and the server verifies it against configured trust stores. Only then does the session complete. If the certificate is missing or untrusted, the server closes the handshake with an alert such as certificate_unknown, and no application-level code ever runs. That is the key property: authentication happens before a single request byte is processed.

When client certificates matter

• Machine-to-machine APIs where a bearer token is too easy to leak; the private key stays on the calling host and is never copied into source files or CI variables.
• Zero-trust service edges where every workload carries its own identity and nothing about the network is implicitly trusted.
• Partner and regulatory integrations where access needs a real audit trail of who could connect, not just who logged in with a password.

Client certificates also carry real friction. They expire, backups get lost, and a private key copied around "just for this one server" becomes a standing risk. Adopt them only where the operational cost is justified by the sensitivity of the traffic.

Issuing client certificates

A small internal CA keeps the loop closed without depending on a public issuer, and it is also the fastest way to learn the mechanics:

openssl req -x509 -newkey rsa:2048 -nodes -days 3650 -keyout ca.key -out ca.crt -subj '/CN=netbay-internal-ca'

openssl req -newkey rsa:2048 -nodes -keyout cli.key -out cli.csr -subj '/CN=api-service-1'

openssl x509 -req -in cli.csr -CA ca.crt -CAkey ca.key -CAcreateserial -days 365 -out cli.crt

The CA pair is now your trust root. Distribute only signed client certificates, never the ca.key, and store the CA key outside the proxy that verifies against it so a compromised edge cannot mint new identities.

Enforcing mTLS at the proxy

nginx makes the verification explicit per virtual host, and the failed-handshake behavior is exactly what a security boundary should be:

server {
    listen 443 ssl http2;
    server_name api.example.com;

    ssl_client_certificate /etc/netbay/tls/ca.crt;
    ssl_verify_client on;

    location / {
        proxy_pass http://10.0.0.7:9000;
        proxy_set_header X-Client-CN $ssl_client_s_dn;
    }
}

Failures surface as a handshake alert such as unknown_ca or certificate_expired, which you can match in the TLS handshake log before the request reaches an application.

Trust decision flow

When to skip client certs

If all your callers are browsers loading public pages, mTLS is usually the wrong tool: users do not hold certificates, and prompting them for one in the browser is a bad experience. Reserve client certificates for services, pair them with short-lived certificates and automatic renewal so a rotated key never causes a standing outage, and always keep an out-of-band recovery path for rotation accidents.

Takeaway

mTLS is a protocol-level identity boundary: no certificate, no session. It pays off for machine-to-machine edges where tokens are a liability, and it costs real certificate operations you have to budget for. Start small with one internal CA and one service, then expand as the renewal tooling hardens. You can build this exact setup on a Netbay VPS in Lucknow and probe it with openssl s_client before the domain even gets a CNAME; see netbayhosts.in for the walkthrough.