Designing Webhook Endpoints Clients Can Trust

A webhook is your API calling your client's URL when something happens — a server provisioned, an invoice paid, a certificate renewed. It is also, from the client's perspective, an unauthenticated stranger asking your code to run. Trust is therefore the whole design problem: prove the message is real, deliver it even when receivers are flaky, and make handling it twice harmless. Three mechanisms carry the weight: signatures, retries, and idempotent handlers.

Payloads that age well

Design the event payload like a resource you version. Include an event id, a type, and a timestamp, and keep the business data under a clearly named field. Version events explicitly so future changes do not silently change what existing handlers process: an event type such as vm.provisioned.v1 signals its own contract. Send the minimal payload that makes the receiver useful, and point at a full resource they can fetch if they need more. A receiver should never have to guess whether a field means what it looks like it means.

Signatures prove it came from you

Anyone who can learn your client's URL can forge a POST to it. Signatures close that hole. Compute an HMAC-SHA256 of the raw request body using a secret shared only with the client, and send the digest in a header such as X-Webhook-Signature. The receiver recomputes the digest from the exact bytes it received and compares in constant time. Verify on the raw body before any parsing, and never compare hex strings with a plain equality operator — timing-safe comparison is the whole point.

import hashlib
import hmac

def verify(secret: str, body: bytes, sig_header: str) -> bool:
    expected = hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()
    return hmac.compare_digest("sha256=" + expected, sig_header)

Retries and delivery guarantees

At-least-once is the honest standard for webhooks — no general transport offers exactly-once. So the sender must retry and the receiver must be able to tolerate duplicates. The usual shape is exponential backoff across a day or two: five minutes, then fifteen, then forty-five, giving up at a maximum attempt count after a few days. Each retry window is a chance for a receiver that was down to rejoin. When retries are exhausted, surface the failure out-of-band — a dead-letter list, a dashboard, a support alert — instead of silently dropping events.

Receivers control the retry schedule with their response: a 2xx acknowledges and stops retries; a 4xx says stop, the payload is bad; a 5xx or a timeout means retry. The receiver should acknowledge fast and process asynchronously, rather than holding the webhook response hostage to slow work.

Handlers that survive replay

Because duplicates are guaranteed, the handler must be idempotent. Deduplicate on the event id — keep a set of processed ids with a generous window, because a late retry can arrive hours later — and make the actual write an idempotent upsert keyed on a stable business identifier. That combination converts at-least-once delivery from a liability into a strength.

// Handler: verify, dedupe, ack fast, process async
const processed = new Set();

app.post("/webhooks/vm-events", async (req, res) => {
  if (!verify(process.env.WEBHOOK_SECRET, req.rawBody, req.header("X-Webhook-Signature"))) {
    return res.status(401).end();
  }
  const event = JSON.parse(req.rawBody);
  if (processed.has(event.id)) return res.status(200).end();
  processed.add(event.id);
  await upsertByBusinessId(event.data);   // idempotent write
  res.status(200).end();                  // ack fast, never retried
});

Takeaway

A webhook is a promise you make over an unreliable transport. Sign every payload, retry with expanding backoff, and make handlers idempotent — do all three and receivers can build real automation on your events instead of bolting on guesses.

The same discipline applies in miniature to any event-driven setup, including the lifecycle events you can automate against the Netbay V1 API; start your next webhook experiment with the docs at netbayhosts.in.