Receiving webhooks: the five parts that make it reliable
Receiving webhooks looks like one of the easiest tasks there is. You write an endpoint, an external service POSTs to it, you write the data to your database. Five lines of code.
That code works — in testing. In production the questions start: what happens if the same event arrives twice? If an old event, resent, arrives after a newer one? If somebody sends a forged request to your endpoint?
This article walks through the five parts of a webhook receiver. None of them is optional.
1. Verify the signature
A webhook endpoint is an open door on the internet. Anyone who knows the address can send a request to it.
That is why serious services attach a signature to every request. Skip verifying it and somebody can send you a forged "payment succeeded" event.
import crypto from 'node:crypto'
function verify(rawBody: Buffer, signature: string, secret: string) {
const expected = crypto
.createHmac('sha256', secret)
.update(rawBody)
.digest('hex')
return crypto.timingSafeEqual(
Buffer.from(expected),
Buffer.from(signature),
)
}There are two important details here, and both get forgotten often.
The first is that you must use the raw body. If your framework parses the JSON and you then JSON.stringify it back, whitespace and field order change and the signature will not match. So on the webhook route you have to preserve the body in its original form.
The second is constant-time comparison. A plain === takes different amounts of time depending on how many characters matched, which in theory opens a path to guessing the signature.
A third addition: if a timestamp arrives alongside the signature, check that too. It protects against replaying an old request — reject events older than five minutes, for instance.
2. Answer fast
The most common architectural mistake is doing all the work inside the webhook.
app.post('/webhooks/payment', async (c) => {
const event = await c.req.json()
await updateOrder(event) // database
await sendReceiptEmail(event) // external service
await notifyWarehouse(event) // another external service
return c.text('ok')
})The problem is that the sending service usually waits only a few seconds for a response. If the email service slows down, you fail to answer in time, the sender treats that as a failure and resends the event. Now you have two identical events.
The right shape is to separate receiving from processing:
app.post('/webhooks/payment', async (c) => {
const raw = await c.req.arrayBuffer()
if (!verify(Buffer.from(raw), c.req.header('x-signature'), SECRET)) {
return c.text('invalid signature', 401)
}
await db.insert(webhookEvents).values({
provider: 'payment',
eventId: JSON.parse(...).id,
payload: JSON.parse(...),
}).onConflictDoNothing()
return c.text('ok') // immediately
})The receiver does exactly two things: verifies the signature and stores the event. A background worker does the rest.
That separation buys you something else too: if processing fails, the event is not lost. It sits in the database and you can run it again.
3. Idempotency
Every webhook sender will eventually deliver the same event twice. That is not a malfunction — it is their guarantee model: at-least-once delivery.
The protection belongs in the database, not in code:
create table webhook_events (
id uuid primary key default gen_random_uuid(),
provider text not null,
event_id text not null,
payload jsonb not null,
processed_at timestamptz,
received_at timestamptz not null default now(),
unique (provider, event_id)
);unique (provider, event_id) is the entire defence. A repeat event gets dropped silently through onConflictDoNothing.
The provider column matters too: different services may use the same identifier format, and they must not collide.
If a sender gives no event id at all — rare, but it happens — build one yourself from a hash of the body plus a timestamp. Not perfect, but it catches most duplicates.
4. Out-of-order events
This is the most commonly overlooked problem.
Webhooks can arrive out of order. An "order paid" event can turn up before "order created" — if the first one was a retry, for instance.
The simple fix: every event carries a version or timestamp, and you refuse to apply an older one.
await db.update(orders)
.set({ status: event.status, statusUpdatedAt: event.occurredAt })
.where(and(
eq(orders.id, event.orderId),
lt(orders.statusUpdatedAt, event.occurredAt), // only if newer
))If the condition does not hold, no update happens and the stale event is silently ignored. Exactly the behaviour you want.
If ordering is truly critical there is a second path: park the event and refuse to process it until the expected predecessor arrives. But that is more complex and usually overkill — version comparison is normally enough.
5. Observability
A webhook is an invisible flow. Nobody tells you when it breaks; you find out when a customer complains.
I watch three things:
- Unprocessed event count — rows where
processed_at is nulland older than five minutes. Should be zero. - Signature failure count — should be zero. If it is not, either the secret is wrong or somebody is probing you.
- Age of the last event — if you normally get dozens per hour and nothing has arrived for three hours, the problem is probably on the sender's side.
That third one is especially useful, because it catches nothing happening — which goes unnoticed far more often than errors do.
Also, get into the habit of storing the raw body. Disk is cheap, and when investigating an incident, the answer to "what exactly arrived?" is priceless.
Putting it together
A reliable webhook receiver works like this:
- Takes the raw body and verifies the signature in constant time.
- Stores the event behind a
unique (provider, event_id)constraint. - Returns
200immediately. - A background worker processes it and discards stale events by version.
- Three numbers are monitored.
There is no complicated technology in that list — one table and one background worker suffice. But if any one of the five parts is missing, the problem will surface eventually. And the worst thing about webhook problems is that they happen silently: nothing crashes, the data simply goes wrong.