Handling out-of-order webhook events
Why webhook events can arrive out of order even when nothing is broken, and how to build a consumer that keys on occurred_at instead of trusting arrival order.
By André LugWebhooks arrive out of order because retries, concurrent delivery attempts, and asynchronous infrastructure on both ends break arrival order — at-least-once delivery trades strict ordering for reliability. Build the consumer for that tradeoff: store the occurred_at of the last event applied per resource and only overwrite state when an incoming event's occurred_at is strictly newer, alongside deduplication by event ID.
It is tempting to assume a webhook consumer will receive events in the same order they happened — request handlers usually run in the order they're called, so why wouldn't deliveries? Because a webhook sender and a webhook receiver are two separate systems connected over an unreliable network, with retries, timeouts, and concurrent delivery attempts sitting in between. Arrival order is not event order, and a consumer that assumes otherwise will eventually overwrite a correct, recent state with a stale one.
PostGate handles OAuth, publishing and delivery evidence for every network through one API. Start free or read the docs.
Why can events arrive out of order at all?
Three ordinary things combine to break arrival order, none of them exotic:
- Retries. PostGate retries a delivery whenever your endpoint times out or returns a non-2xx status — see webhook retry best practices. If your endpoint is briefly slow when event A fires but fast again by the time event B fires seconds later, B can be acknowledged and processed before A's retry ever lands.
- Concurrent delivery attempts. Multiple events for the same resource can be in flight to your endpoint at the same time. HTTP requests racing across the network, load balancers, and your own server's request-handling concurrency do not guarantee that whichever request left the sender first arrives, or finishes processing, first.
- Asynchronous infrastructure on both ends. The sender's event queue and your own request pipeline (proxies, queues, worker pools) each introduce their own scheduling, and none of them promise strict FIFO delivery to your handler.
None of this is a bug in the sender. At-least-once delivery over a network is a well-understood tradeoff: you get reliability (an event that fails to deliver gets retried instead of silently vanishing) in exchange for giving up a strict ordering guarantee. A consumer has to be built for that tradeoff, not against it.
A concrete failure: processing overwriting published
Say your handler naively does UPDATE posts SET status = event.data.status WHERE id = event.data.post_id on every incoming event, with no other check. A post.published event fires, then — because of a retry on an earlier, slower delivery attempt — a post.partial event for the same post that was actually superseded by the published state arrives second and overwrites it. Your stored status now says partial for a post that is genuinely fully published. Nothing in the delivery was wrong; the events themselves were correct and the retry behaved exactly as documented. The bug is entirely in a handler that trusted arrival order as if it were event order.
Key on the event timestamp, not on when you received it
Every PostGate webhook event carries an occurred_at field in its envelope, alongside id, type, and a resource-specific data object — see the webhook payload shapes reference. Use occurred_at, not the order your handler ran in, to decide whether an incoming event should update your stored state. The rule is simple: only write the new state if the incoming event is newer than what you already have for that resource.
async function applyPostStatusEvent(event) {
const { post_id, status } = event.data;
// Compare against the last event actually applied, not against "did we get one already"
const current = await db.postStatus.findUnique({ where: { post_id } });
if (current && current.last_event_occurred_at >= event.occurred_at) {
// A newer (or equal) event already landed — this one is a late arrival, discard it
return;
}
await db.postStatus.upsert({
where: { post_id },
update: { status, last_event_occurred_at: event.occurred_at },
create: { post_id, status, last_event_occurred_at: event.occurred_at },
});
}Isn't idempotency by event ID enough?
Deduplicating by id (also sent as the X-PostGate-Event-Id header) stops you from processing the same event twice when a delivery is retried. It does nothing to stop a different, older event from arriving after a newer one and overwriting it — that's an ordering problem, not a duplication problem, and it needs the occurred_at comparison above in addition to, not instead of, ID-based dedup. See webhook retry best practices for the dedup half of this; this post is specifically about the ordering half.
Don’t assume one event type always precedes another
It's natural to reason that a post.published event logically comes "after" a post.scheduled one, since that's the order the underlying state machine moves through. That's true of the state transitions themselves — it is not a guarantee about delivery order. A handler that assumes it will always see the scheduled event before the published one, and errors out (or drops the event) when that assumption breaks, will fail on exactly the request that most needs to succeed: a genuinely valid, if delayed, notification. Design each event handler to be safe to run without assuming any other specific event already arrived — check the current stored state and the incoming occurred_at, not a remembered sequence of prior events.
A short checklist for an order-safe consumer
- Deduplicate by event
idso a retried delivery is never processed twice. - Store the
occurred_atof the last event you applied per resource, and only overwrite state when a new event'soccurred_atis strictly newer. - Never branch handler logic on an assumption that one event type must have already arrived before another.
- Treat every handler invocation as independent — safe to run in isolation, with the stored state (not memory of prior calls in the process) as the only source of truth about what happened before.
Frequently asked questions
Why do webhooks arrive out of order?
Three ordinary things combine to break arrival order: retries (if your endpoint is briefly slow when event A fires, event B can be processed before A's retry lands), concurrent delivery attempts racing across the network and your own server's request concurrency, and asynchronous infrastructure on both ends — the sender's event queue and your proxies, queues, and worker pools. None of it is a bug in the sender; it is the cost of at-least-once delivery.
Is deduplicating by event ID enough to handle out-of-order webhooks?
No. Deduplicating by event ID stops you from processing the same event twice when a delivery is retried, but it does nothing to stop a different, older event from arriving after a newer one and overwriting it. That is an ordering problem, not a duplication problem — it needs an occurred_at comparison in addition to, not instead of, ID-based dedup.
Can I assume a post.scheduled event always arrives before post.published?
No. The state machine moves through those transitions in order, but that is not a guarantee about delivery order. A handler that errors out or drops an event when the expected predecessor has not arrived fails on exactly the notification that most needs to succeed. Design each handler to be safe to run without assuming any other specific event already arrived — check stored state and the incoming occurred_at.
How do I make a webhook consumer order-safe?
Four rules: deduplicate by event ID so a retried delivery is never processed twice; store the occurred_at of the last event applied per resource and only overwrite state when a new event's occurred_at is strictly newer; never branch handler logic on an assumption that one event type must have arrived before another; and treat every handler invocation as independent, with stored state as the only source of truth about what happened before.
