Building Webhook Infrastructure That Never Drops a Payment Event

Webhooks are the nervous system of modern payment infrastructure. When a payment completes, dozens of downstream systems need notification: order management, inventory, accounting, fraud review, customer communication. A dropped webhook means broken business processes.

The Durability Requirement

Our webhook system must survive any single point of failure: server crash, network partition, datacenter outage, or deployment rollback. This requires durable storage before any delivery attempt.

// Event generation with WAL guarantee
async function emitPaymentEvent(event: PaymentEvent) {
  // 1. Write to durable log (survives crashes)
  await writeAheadLog.append(event);
  
  // 2. Acknowledge to payment system
  await markEventPersisted(event.id);
  
  // 3. Async delivery (at-least-once semantics)
  deliveryQueue.enqueue(event);
}

Intelligent Retry Logic

Failed webhook deliveries retry with exponential backoff, jitter, and per-endpoint circuit breaking. We've observed that 67% of failures recover within 5 minutes—usually transient network issues or brief endpoint unavailability.

• Retry schedule: 30s, 2m, 10m, 1h, 4h, 12h, 24h
• Jitter: ±20% to prevent thundering herd on recovery
• Circuit breaker: Opens after 5 consecutive failures
• Dead letter queue: Events undelivered after 7 days require manual review