#Purpose
Rules for webhooks in both directions. A webhook is an HTTP request to a server you do not control, about an event that already happened. Two facts drive everything:
- Delivery is at-least-once. Duplicates are normal, not a bug.
- The receiver's endpoint is public. Anyone can POST to it, so the payload must be cryptographically attributable.
#Signing (sender)
makefilePOST /hooks/acme HTTP/1.1
Webhook-Id: evt_01J8ZQ3M7K
Webhook-Timestamp: 1756392779
Webhook-Signature: v1,k3Yb2Q…base64…
Content-Type: application/json
Sign id.timestamp.body with HMAC-SHA256 over the raw request bytes:
tsconst signed = `${id}.${timestamp}.${rawBody}`;
const sig = crypto.createHmac("sha256", secret).update(signed).digest("base64");
Requirements:
- Include the timestamp inside the signed payload, so it cannot be altered.
- Include a unique event id, so receivers can deduplicate.
- Support multiple active signatures (
v1,sigA v1,sigB) so a secret can be rotated without a coordinated cutover. - One secret per endpoint, generated with a CSPRNG, shown once.
→
Security/secret-management
Never sign a re-serialised body. JSON.stringify(req.body) reorders keys and
changes whitespace; the receiver's HMAC will not match. Sign and verify the exact
bytes on the wire.
#Verifying (receiver)
ts// Express: the raw body is required, so capture it before JSON parsing
app.post("/hooks/acme", express.raw({ type: "application/json" }), (req, res) => {
const ts = Number(req.get("Webhook-Timestamp"));
if (Math.abs(Date.now() / 1000 - ts) > 300) return res.sendStatus(400); // replay window
const expected = crypto
.createHmac("sha256", process.env.WEBHOOK_SECRET)
.update(`${req.get("Webhook-Id")}.${ts}.${req.body}`)
.digest();
const given = Buffer.from(parseSignature(req.get("Webhook-Signature")), "base64");
if (expected.length !== given.length ||
!crypto.timingSafeEqual(expected, given)) return res.sendStatus(401);
enqueue(JSON.parse(req.body)); // hand off, do not process inline
res.sendStatus(200); // acknowledge fast
});
Four things this gets right, each of which is commonly wrong:
- Raw body. Verification against a parsed-and-restringified body fails intermittently and inexplicably.
- Timestamp window (±5 minutes) — without it a captured request is replayable forever.
timingSafeEqual, with a length check first (it throws on mismatched lengths).===on a signature leaks it byte by byte under timing analysis.- Acknowledge, then process. Do the work in a background job.
→
Backend/queues
Never trust any field in the body — including a user_id or an amount —
before the signature verifies. And never process an unverified payload "just to
log it": that is still parsing attacker-controlled input.
#Consumers must be idempotent
Duplicates arrive because the sender retried after your 200 was lost in transit.
The event id is the deduplication key.
sqlINSERT INTO webhook_events (id, received_at) VALUES ($1, now())
ON CONFLICT (id) DO NOTHING; -- zero rows affected means already processed
Ordering is not guaranteed. A subscription.updated may arrive before
subscription.created. Handle it:
- Include a monotonic
sequenceor the resource'supdated_atin the payload and discard events older than the state you already hold. - Or treat the webhook as a notification only and re-fetch current state from the sender's API. This is the most robust pattern and sidesteps ordering entirely.
#Delivery (sender)
| Concern | Rule |
|---|---|
| Retries | Exponential backoff with jitter: 1m, 5m, 30m, 2h, 6h, 24h |
| Retry on | Timeouts, connection errors, 5xx, 429 |
| Do not retry | 4xx other than 429 and 408 — the request is wrong, not late |
| Timeout | 5–10 seconds. A slow receiver must not hold your worker |
| Disable | After N consecutive days of failure, with notification first |
| Concurrency | Bound per endpoint so one slow receiver cannot starve the fleet |
Provide a dead-letter view and manual replay in the dashboard. Every integration eventually needs to reprocess a window of events, and without a replay button that becomes a support engineering task.
Publish your source IP ranges so receivers can allowlist them, and keep them stable.
Log every attempt with the response status, latency and body prefix, and expose that log to the customer. This is the single highest-value support feature a webhook system has.
#Endpoint design
- Return
200/204quickly — under a second. A202is also fine. - Any non-2xx means "retry"; be sure that is what you intend.
- Receivers should respond
200to an event type they do not recognise, not400— otherwise adding a new event type breaks existing integrations. - Guard against SSRF when a customer supplies the destination URL: reject private address ranges, link-local addresses, and redirects to them, resolving DNS at request time.
#Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| No signature | Anyone can POST forged events | HMAC over raw bytes |
| Signing a re-serialised body | Key order and whitespace differ | Sign the wire bytes |
| No timestamp in the signature | Captured requests replay forever | Signed timestamp + window |
=== on signatures | Timing side channel | timingSafeEqual |
| Processing before verifying | Attacker-controlled input in business logic | Verify first |
| Processing inline | Sender times out and retries; duplicates multiply | Enqueue, then 200 |
| Assuming exactly-once | Duplicates are normal | Deduplicate by event id |
| Assuming ordered delivery | Out-of-order updates corrupt state | Sequence check or re-fetch |
Retrying on 4xx | Hammering a permanently broken endpoint | Retry only 5xx/429/timeouts |
| Fixed-interval retries | Synchronised thundering herd | Exponential backoff with jitter |
| No replay tooling | Every gap becomes a support escalation | Dead-letter view + replay |
400 on unknown event types | New event types break integrations | Ignore and return 200 |
| Unvalidated customer-supplied URL | SSRF into internal networks | Reject private/link-local ranges |
#Checklist
- Verify: Every delivery is HMAC-signed over the raw body, id and timestamp
- Verify: Multiple concurrent signatures are supported for secret rotation
- Verify: Receivers verify against the raw bytes, before parsing
- Verify: A timestamp tolerance window rejects replays
- Verify: Signature comparison is constant-time with a length check
- Verify: Receivers acknowledge fast and process asynchronously
- Verify: Every event carries a unique id, and consumers deduplicate on it
- Verify: Out-of-order delivery is handled by sequence check or state re-fetch
- Verify: Retries use exponential backoff with jitter, only on retryable statuses
- Verify: Per-endpoint concurrency is bounded and a delivery timeout is set
- Verify: Failing endpoints are disabled after notification, not silently
- Verify: Delivery attempts are logged and visible to the customer
- Verify: A dead-letter view with manual replay exists
- Verify: Unknown event types are ignored rather than rejected
- Verify: Customer-supplied destination URLs are validated against SSRF
#Anchors (restated last, read last)
The rules that must hold when you stop, repeated here because the end of the context is what you act on:
-
Never sign a re-serialised body.
JSON.stringify(req.body)reorders keys and changes whitespace; the receiver's HMAC will not match. Sign and verify the exact bytes on the wire. -
Never trust any field in the body — including a
user_idor an amount — before the signature verifies. And never process an unverified payload "just to log it": that is still parsing attacker-controlled input. -
Every delivery is HMAC-signed over the raw body, id and timestamp
-
Multiple concurrent signatures are supported for secret rotation
-
Receivers verify against the raw bytes, before parsing
-
A timestamp tolerance window rejects replays
-
Signature comparison is constant-time with a length check
-
Receivers acknowledge fast and process asynchronously
Before reporting done, prove the module still imports — run the line for this stack and paste its output:
bashpython -c "import <package>" # Python: the package you changed
node -e "require('./<entry>')" # Node CJS, or: node --input-type=module -e "import './<entry>.js'"
go build ./... # Go