Webhook Reliability Patterns for Production Systems
How to build webhook delivery systems that never lose events, from retry strategies to dead letter queues.
Webhooks are deceptively simple in concept — send an HTTP POST when something happens — but building a reliable webhook delivery system is one of the hardest distributed systems problems you will encounter. Here are the patterns that separate production-grade webhook systems from toys.
The Retry Strategy
Never use fixed-interval retries. They create thundering herd problems when a consumer’s server comes back online and every queued webhook fires simultaneously. Instead, use exponential backoff with jitter:
- Attempt 1: Immediate
- Attempt 2: 1 minute + random jitter (0-30s)
- Attempt 3: 5 minutes + random jitter
- Attempt 4: 30 minutes + random jitter
- Attempt 5: 2 hours + random jitter
- Attempt 6: 12 hours + random jitter
The jitter is critical — it prevents retry storms where thousands of failed webhooks all retry at the exact same moment.
Idempotency Keys
Every webhook event should include a unique idempotency key. Consumers will inevitably receive duplicate deliveries due to network issues, timeouts, or retry logic. The idempotency key lets them safely deduplicate on their end.
Include this key in both the payload and a header (X-Webhook-Id), and document clearly that consumers should use it for deduplication.
Dead Letter Queues
After exhausting all retry attempts, failed webhooks should go to a dead letter queue rather than being silently dropped. The DLQ serves two purposes:
- It gives consumers a way to manually inspect and replay failed events
- It gives you visibility into systematic delivery failures that might indicate a consumer-side issue
A good webhook management dashboard — like what APIForge provides — lets consumers browse their DLQ, inspect payloads, and trigger manual redelivery with a single click.
Signature Verification
Always sign your webhook payloads with HMAC-SHA256 using a per-consumer secret. Include the signature in a header (X-Webhook-Signature) so consumers can verify that the payload was not tampered with in transit. This is not optional — without signature verification, webhooks are a security vulnerability.
