Webhook retry handling is a state-management problem. A sender may redeliver after a timeout even when the receiver completed some work. If every attempt creates a new ticket or repeats a change, the integration magnifies an ordinary network failure into an operational incident.
Distinguish delivery attempts from business effects
Define a delivery key using the provider's documented identity and the subscription or tenant scope. Separately define the business effect, such as updating one existing authentication finding. Multiple valid events may legitimately relate to the same finding, while repeated delivery of one event should not apply its effect twice.
GitHub's webhook guidance says requested redeliveries retain X-GitHub-Delivery. It also advises responding with a 2XX within ten seconds and considering asynchronous processing. Treat those as GitHub-specific documented behaviors. Do not promise that every provider automatically retries or uses the same timeout; confirm its actual delivery policy and missed-delivery recovery route.
Acknowledge at the right boundary
Verify the event first, then durably record accepted work before acknowledging success. Process expensive work from a queue. If durable acceptance fails, do not send a success response merely because parsing succeeded. Conversely, do not hold a connection open until a long downstream investigation finishes when the provider requires a quick response.
Use states such as queued, processing, retry pending, complete and dead-letter. Track attempt count and next retry time. A worker crash may leave a processing claim abandoned, so define a safe lease or recovery rule. Protect uniqueness and state transitions with database constraints or transactions rather than a read-then-write check vulnerable to concurrent deliveries.
Design retryable effects
For external calls, use a supported idempotency key or a stable target identity where available. When a timeout leaves the outcome uncertain, read or reconcile the target before repeating a side effect. If the downstream API cannot offer idempotency, document the residual duplicate risk and add a compensating review path.
Suppose a verified delivery is durably queued, but its success response is lost. A later requested redelivery should find the same delivery state, not create a second ticket merely because another HTTP request arrived. Separately, suppose the worker calls a ticket API and times out after that API may have created the ticket. The queue’s delivery deduplication alone cannot resolve this downstream uncertainty: inspect the target or use its supported idempotency key before repeating creation.
These are hypothetical failure paths, not an executed queue, virtual-clock test or exactly-once guarantee. Test concurrent duplicates, worker interruption, an abandoned processing lease and bounded exhaustion using the actual storage and downstream contract. A dead-letter item should retain safe evidence and require a controlled replay after its cause is corrected, rather than silently resetting the attempt count.
Close the loop on exhausted work
Use bounded retries with backoff and applicable rate-limit signals. GitHub REST best practices explain rate-limit-aware behavior for that API; apply the actual downstream service's contract to production. Move repeatedly failing items to a controlled dead-letter queue and alert the accountable operator with safe evidence and the next step.
Separate retryable transport errors from permanent schema or authorization failures. Replaying an unauthorized request indefinitely will not fix its permission. Retain enough identity and outcome evidence to explain every event's final state, and provide a deliberate operator replay procedure after correction. Monitor oldest queued work, retry exhaustion and unresolved outcomes, not only the count of successful HTTP responses. No live provider retry behavior or EmailAuth capability is demonstrated here.