Webhook Retry Strategy: A Practical Design Guide

The five things that a sound webhook retry strategy will do are: retry on failures where recovery is possible, wait between attempts with exponential backoff with jitter, respect Retry-After, give up after a defined timeframe and move the event to a monitored dead-letter queue, and expect every event to be received multiple times. The remainder of this guide covers how to implement each of those facets, as well as common mistakes made when building them. It also details the retry policy to publish for consumers of your webhooks.

TL;DR

• Retry only transient failures: timeouts, 5xx and 429. Retry loops should treat 400, 401, 403 and 410 as permanent failures and notify a human operator.
• Start with two phases: a couple quick retries for brief network glitches, followed by progressively lengthening waits with full jitter applied to distinguish intentional outages.
• Respect Retry-After as the minimum wait period and limit the overall window to as few days as possible, usually one to three.
• Messages are delivered at-least once. Events must have a unique ID, and the consumer should de-dupe by ID.
• A dead-letter queue that no one monitors is equivalent to data loss. Monitor its length, and create a replay mechanism.

Table of Contents

• Why webhook delivery needs a retry policy at all
• Step 1: decide which failures deserve a retry
• Step 2: space attempts with backoff and jitter
• Step 3: use two phases, not one curve
• Step 4: honour Retry-After
• Step 5: make receivers idempotent
• Step 6: dead-letter queues, replay and circuit breakers
• What to publish as your webhook retry policy
• Common mistakes to avoid
• Our perspective: retries are an operations feature
• Building this into your platform
• Testing your retry logic before production does
• FAQ
• Sources

Why webhook delivery needs a retry policy at all

Think of a webhook as an HTTP request sent from a system you own to a system you don't own. The destination could be mid-deploy, rate limiting you, have an expired TLS certificate, or just be really slow. Packets can be dropped anywhere in transit between you two.

The IETF draft standard on event delivery says it clearly: each attempt at delivery is best-effort and does not guarantee receipt . That is an Internet-Draft, not yet a ratified standard, but it references the semantics of delivery that modern platforms have already implemented. Operators have therefore interpreted webhooks as at-least-once delivery. As the AWS Compute Blog puts it, duplicates are not an exception; they are part of the delivery model, and expecting unique event IDs to compare against a store is how you should process them.

A retry policy is the contract that answers four questions:

1. Which failures are worth another attempt?
2. How long do we wait between attempts?
3. When do we give up?
4. What happens to events we gave up on?

The next sections take them in order.

Step 1: decide which failures deserve a retry

Most calls don't need retrying. The purpose of retries is to recover from temporary errors. Retries do not make up for specifying an incorrect URL, providing invalid credentials, or sending a malformed payload. Those errors will persist until someone fixes the calling site.

Handle permanent groups with extra care. Retrying 401s at full throttle will flood the receiver’s logs and look like an attack. Dead-letter those events, and trigger an alert.

A few judgement calls remain:

• 404: unclear. Is it a mistyped URL, or someone deployed while you were trying to navigate to a route? Having some limited retry budget makes sense.
• Redirects: follow redirects with payloads and signatures handled consistently. Pick a rule or policy and write it down.

Responses that are too slow are failures. Agree on a delivery timeout, often a few seconds, and communicate what it is to receivers.

Step 2: space attempts with backoff and jitter

The natural response to a failed delivery is to retry it as soon as possible. That is correct for the first retry that makes sense to do, since the transient network hiccup may have resolved itself by then. However, if the recipient is truly congested, rapid retries only exacerbate the problem.

Exponential backoff

Retries wait twice as long each attempt: base, 2x base, 4x base, and so on. That's fast growth. This handy calculator for outbound webhook retry practice shows that with a base of one second, ten tries means waiting over 17 minutes, while twenty tries means waiting over 12 days. You'll want to set a limit.

The IETF draft suggests randomised delays between zero and the smaller of the cap and base * 2^n, where n is the attempt number. Common numbers are a base of 1 second with a cap of 60 to 600 seconds. Production systems often use an even higher cap of an hour for the slow phase. The correct value depends on how big your window is, discussed below.

Why jitter is not optional

Picture an endpoint that is unavailable for 10 minutes with 5k events stacked against it. Without jitter, all 5k events retry at precisely the same timestamps. The receiver comes back online and gets smacked with 5k parallel requests. The receiver falls over. Repeat. Thundering herd.

Jitter spaces out the retries. Full jitter, where the delay is anywhere from zero to the exponential maximum, is recommended by default because it smooths traffic best. Equal jitter always uses at least half the cap.

That one line, applied consistently, removes the most common cause of retry storms.

Step 3: use two phases, not one curve

A single exponential curve is either too aggressive for outages or too sluggish for blips. Infrastructure-grade systems stagger the schedule. Hookdeck's two-phase approach favors short intervals initially to bounce back from ephemeral networking issues. It then backs off to longer intervals, measured in hours or days, to survive provider outages and maintenance without exhausting the retry budget.

A worked example you can adapt:

The author also mentions that retry windows are frequently capped by production systems at one to three days, with a maximum of about 50 attempts, depending on their policy. Real world reference: Stripe retries failed events for up to three days with exponential backoff in live mode, according to third-party integration documentation. Always refer to the provider’s own documentation before relying on the exact timing.

These numbers are starting points. Match the window to the business value of timeliness:

• Payments and safety alerts: short timeouts and aggressive early retries, because stale data can be worse than no data.
• Bulk synchronisation: long, slow schedules are fine. An overnight sync of contacts between a CRM and accounting app is a textbook example of a flow that can absorb the slow phase. If you work with teams building these types of flows, you'll see the tell-tale signs in our CRM integration options that sync to finance, e-signature and workflow apps.

The best enterprise systems make this configurable, such as aggressive and normal profiles, rather than one schedule for everyone globally.

Step 4: honour Retry-After

If a receiver responds with a 429 or 503 response it may include a Retry-After header. That header represents the receiver telling you when it will be able to cope. The IETF draft clearly states that Retry-After requires senders to use that as the minimum backoff time, overriding any local algorithm. If you ignore Retry-After you risk being blocked at the IP or network infrastructure layer.

Practical rules:

• Take the larger of the computed backoff and the Retry-After value.
• Parse both forms of the header: seconds and HTTP date.
• Cap it to something sane. An event requesting one week should not silently extend your window. Dead-letter it and raise an alert.

If you operate an API and also send webhooks, it works both ways. Limits that cause those 429 responses are usually designed like this, as discussed in our article on production API rate limiting with a sliding window.

Step 5: make receivers idempotent

Retries belong to the sender. Handling them gracefully is the responsibility of the receiver. If each event does not contain a unique identifier, the receiver cannot distinguish between a real new event and a retry of an event it already received. This leads to duplicate charges, double-counted records, or corrupted state.

Senders should:

• Provide a unique event ID that does not change between retries of the same event. UUID is fine.
• Keep the payload identical between attempts. Only the timestamp and signature should change.
• Document the delivery model as at-least-once and note that ordering is not guaranteed.

Receivers should:

1. Verify the signature first.
2. Query the event ID from some persistent store, like a database table or DynamoDB.
3. If seen, return 200 and do nothing.
4. If new, record the ID and process, ideally in a single transaction.

Return 2xx immediately and do the heavy lifting asynchronously. An inline handler that does slow processing will timeout and the sender will retry an event you already received.

Step 6: dead-letter queues, replay and circuit breakers

Dead-letter queue with an owner

When the retry window closes, the event goes to a DLQ, a dead-letter queue. A silent DLQ is the same as tossing data out. Reliability requires three components:

• Alerting on depth and age, so someone sees the first parked event, not the thousandth.
• A replay mechanism: allows your engineers to bulk re-process events after resolving the root cause, whether that be an expired TLS certificate or an unplanned server outage. Read more about this operational replay as part of achieving true reliability in the Hookdeck webhook retry guide.
• Complete request and response logging for every attempt so you know why it failed.

Circuit breaking on unhealthy endpoints

If an endpoint fails almost every time you send a request to it, retrying simply consumes your resources and may compound the problems on the receiver’s end. Intelligent systems monitor the health of destinations. If failure rates reach extremes, such as 95%, the system will trip a circuit and stop retries, as detailed in this gethook policy design article. Messages are queued or suspended, and a probe request is made to test for recovery before allowing traffic to continue.

Send a notification email to the endpoint owner automatically as well. And have a disable step after failing for some period of time, with an obvious way to re-enable.

What to publish as your webhook retry policy

If you deliver webhooks to customers, your policy is part of your product. Get it in writing. Elements of a strong public policy:

• The success condition, such as any 2xx within N seconds.
• Which responses are retried and which are not.
• The schedule, with the cap and total window.
• How Retry-After is treated.
• The idempotency key field name and its guarantees.
• What happens after exhaustion: DLQ, dashboard visibility, manual replay, or API resend.
• When an endpoint is disabled and how to re-enable it.
• Which IP ranges or signatures identify your sender.

Transparency prevents support tickets. Most “you lost our event” tickets are actually “we didn’t realize you gave up after six hours.”

Common mistakes to avoid

Our perspective: retries are an operations feature

We make software to run where every missed event is expensive: datacenter telemetry, building automation, construction safety. Webhooks in those environments are rarely your entire integration. They sit between sensors, controllers, monitoring systems, and business software, often with older pieces throughout the chain.

That shapes how we think about retry design:

• Tie the policy to the consequence. An alarm that something has changed on the site needs a different schedule than a nightly report. One global curve doesn't do either.
• Don't be cute. If people can't see it, it doesn't matter how smart your system is. A simple two-phase schedule with a dashboard, alerts, and a replay button will beat a clever algorithm that can't be monitored.
• Legacy endpoints require softer handling. Bursting is often poorly absorbed by older equipment. Wrapping these systems with something that queues and delimits at the front of them tends to be safer than relying on retry tuning alone. See our blog post about wrapping a legacy system with no downtime.
• Retries are not reconciliation. If you are storing critical data, schedule a reconciliation job that compares the two systems and fixes drift.

We believe retry logic should be designed upfront, not added as an afterthought after the first dropped event.

— Harry

Building this into your platform

Designing event delivery for an enterprise platform? Need to integrate the event flow between building and datacenter infrastructure applications? Let us help. Our SaaS development and enterprise automation software groups build integrations like these every day. For building and datacenter specific integrations, review our BMS and PMS integration services or our PODVIEW monitoring platform. If you'd like to start a conversation about your project, please begin at podtech.com.

Testing your retry logic before production does

A policy that you haven’t exercised is nothing more than a guess. Here are some things to check pre-launch:

1. Kill the receiver mid-stream. Verify retries begin shortly after transmission starts, fan out, and then level off at the window.
2. Return each class of status code. Make sure 5xx retries and 401 dead-lettering happen as designed.
3. Send the same event twice. Confirm the receiver’s side effects happen once.
4. Return 429 with a long Retry-After. Verify the sender backs off for at least that long.
5. Let the window expire. Verify that the event goes to the DLQ, the alert fires, and replay works.
6. Replay a batch. Verify the receiver can handle the burst, and that replays are rate limited.

Remember the findings. They are the proof behind the policy you report.

FAQ

How do you handle webhook retries on the receiving side?

Respond with 2xx as soon as you've verified the signature and saved the event, then process it later. Log the unique event ID so you can notice and discard retries of successfully processed events. Never assume exactly-once delivery.

How do you handle webhook failures that keep happening?

First determine if the failure is transient or permanent. If there are repeated 5xx responses or timeouts, then the receiver is struggling, so back off and consider opening a circuit breaker. Repeated 4xx responses indicate configuration or payload issues that require human intervention, after which you can replay the dead-lettered events.

What is a sensible retry schedule?

Typical patterns are a few retries over a few seconds, then backoffs that grow to an hour with full jitter. Windows between one and three days are common, but make them smaller for time-sensitive data and larger for batch synchronisation. Publish whatever you choose clearly.

Should every failed webhook be retried?

Retry timeouts, as well as 5xx and 429 responses. For permanent errors such as 400, 401, 403 and 410 that will fail identically each time, publish them to a dead-letter queue and notify the owner.

Sources

• IETF Internet-Draft: event delivery semantics
• Sending and receiving webhooks on AWS (AWS Compute Blog)
• Webhook retry policy design: beyond exponential backoff
• Webhook retry best practices for sending webhooks
• Implementing webhook retries
• Stripe webhooks: events, retries and local testing

Recommended

• Sliding Window Default: Production API Rate Limiting for Architects
• API versioning strategy: a practical guide for developers