Webhook Idempotency and Deduplication: Stop Processing Events Twice

Why Duplicate Webhooks Are Guaranteed

Webhook providers face a choice when a delivery is ambiguous: deliver at-most-once (and silently drop events when things go wrong) or at-least-once (and occasionally send the same event twice). Every serious provider picks at-least-once, because a dropped payment.succeeded is worse than a duplicated one. That decision pushes the deduplication problem onto you, the receiver.

Retry schedules amplify this. If your endpoint flaps during an incident, a single event can arrive five or six times over hours. For the full picture of how providers back off and when, see our guide on webhook retries best practices.

What Providers Actually Send: Stripe, GitHub, Shopify

The good news: most providers give you a stable identifier for exactly this purpose. The details differ enough that it's worth being precise.

If your provider sends nothing usable — some internal services and smaller SaaS tools don't — you'll need to derive a key yourself. That's next.

Choosing the Right Idempotency Key

An idempotency key is the value you use to answer one question: “have I seen this exact event before?” Getting the key wrong breaks dedup in both directions — too broad and you drop distinct events, too narrow and duplicates slip through.

import { createHash } from "crypto";

// Fallback only: derive a key from the exact raw bytes.
// Re-serializing parsed JSON changes key order and whitespace,
// so two identical events can hash differently.
const idempotencyKey = createHash("sha256")
  .update(rawBody)
  .digest("hex");

Implementation Patterns: Postgres and Redis

Two patterns cover the vast majority of real systems. Both hinge on the same property: the check and the claim must be a single atomic operation.

Pattern 1: Postgres unique constraint

Create a table whose primary key is the idempotency key, and let the database enforce uniqueness:

CREATE TABLE webhook_events (
  event_id    TEXT PRIMARY KEY,   -- "stripe:evt_..." scoped key
  source      TEXT NOT NULL,
  received_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

In the handler, insert and let the conflict tell you whether this is a duplicate:

const inserted = await db.query(
  `INSERT INTO webhook_events (event_id, source)
   VALUES ($1, 'stripe')
   ON CONFLICT (event_id) DO NOTHING
   RETURNING event_id`,
  [`stripe:${event.id}`]
);

if (inserted.rowCount === 0) {
  // Duplicate. Acknowledge with a 200 so the provider stops retrying.
  return res.status(200).send("ok");
}

await processEvent(event);
return res.status(200).send("ok");

Postgres has no native TTL, so prune the table on a schedule — a nightly cron is fine:

DELETE FROM webhook_events
WHERE received_at < now() - INTERVAL '30 days';

Pattern 2: Redis SET NX with TTL

If you're already running Redis, SET ... NX EX gives you an atomic claim with automatic expiry in one command:

const key = `wh:stripe:${event.id}`;

// Atomic "set if not exists" with a 7-day TTL.
// Returns null if the key already existed.
const isFirstDelivery = await redis.set(key, "1", {
  NX: true,
  EX: 60 * 60 * 24 * 7,
});

if (!isFirstDelivery) {
  return res.status(200).send("ok"); // duplicate
}

await processEvent(event);
return res.status(200).send("ok");

Dedup at the Door vs Idempotent Side Effects

Everything above is “dedup at the door”: reject duplicates before processing starts. The complementary strategy is making the processing itself idempotent, so that running it twice produces the same end state. The strongest systems do both — the door check handles the common case cheaply, and idempotent side effects catch whatever slips through (an expired TTL, a lost Redis key, a replay from six months later).

A pattern you'll see in a lot of codebases — and should not copy — is check-then-act:

// DON'T do this: race condition between the check and the insert
const existing = await db.query(
  "SELECT 1 FROM webhook_events WHERE event_id = $1",
  [event.id]
);

if (existing.rowCount > 0) {
  return res.status(200).send("ok");
}

// Two concurrent deliveries of the same event can BOTH reach
// this line, because neither saw the other's uncommitted insert.
await db.query("INSERT INTO webhook_events (event_id) VALUES ($1)", [
  event.id,
]);
await processEvent(event); // ...and both process the event.

This isn't theoretical. Providers retry on a timer, your handler runs on multiple instances behind a load balancer, and a retry can land while the original delivery is still in-flight. Two requests run the SELECT at the same moment, both see zero rows, both proceed. INSERT ... ON CONFLICT DO NOTHING wins because the database serializes the conflict: exactly one insert succeeds, and the loser finds out atomically via rowCount. The same logic applies to Redis: GET then SET is a race, SET NX is not.

Idempotent side effects extend the same thinking into your domain logic:

Testing Duplicate Handling With Hooklistener

Dedup logic is easy to write and easy to get subtly wrong, and the failure only shows up when a duplicate actually arrives. You shouldn't wait for production to find out. The test you want is simple: take one real event and deliver it to your handler several times.

Capture once, replay repeatedly

Point your provider at a Hooklistener endpoint (or run a CLI tunnel in front of your local server) and trigger one real event. Once it's captured, replay it against your handler as many times as you like — same headers, same body, same signature. The sequence to verify:

• • Replay 1: handler processes the event and responds 200
• • Replays 2–5: handler short-circuits as a duplicate, still responds 200, and responds noticeably faster (no processing happened)
• • Side effects:exactly one row, one email, one charge — check your database, not just the status codes

Replay also lets you modify the body and headers before sending. Change the event ID to a fresh value and replay again: your handler should treat it as a brand-new event. That two-sided test — same key dedupes, new key processes — catches the embarrassing failure mode where your dedup accidentally swallows everything.

Read the responses, not just the requests

Hooklistener records the HTTP response for each captured webhook — status code, headers, body (up to 10 KB), and whether it came from the endpoint's default response or a matched mock rule — in a Response tab on each request. The request list has a color-coded status column (green 2xx, blue 3xx, amber 4xx, red 5xx), so a dedup test reads at a glance: five identical requests, five green responses. If replay three is amber or red, your duplicate path is throwing instead of acknowledging — which means the provider would keep retrying forever. On paid plans you can also select two to five captured requests and run an AI comparison that breaks down shared patterns, differences, timing, and anomalies across them — useful when you're staring at near-identical retries trying to spot what changed.

Generate real provider retries

Replays test your logic; real retries test the provider's actual behavior — headers, timing, whether the payload is byte-identical across attempts. Configure a Hooklistener endpoint to return a custom 500 response, point Stripe or Shopify at it, and trigger an event. The provider sees the failure and begins its retry schedule, and every retry lands in your webhook inbox where you can confirm the event ID really is stable across attempts before you bet your dedup strategy on it.

Lightweight dedup in Automations with the Datastore

If you're using Hooklistener Automations to forward or transform webhooks (both paid features), you can dedup inside the chain itself using the Datastore — an organization-wide persistent key-value store with namespaced keys, JSON values up to 64 KB, and optional TTLs with automatic cleanup. Each endpoint's Automate tab runs an ordered chain of actions on every captured webhook, so a dedup chain looks like this:

• • Extract JSONpulls the provider event ID from the payload — e.g. $request.body.id$for Stripe — into a variable
• • Condition checks whether a Datastore key for that event ID already has a value (read via $store.namespace.key$ template variables) and stops the chain if it does
• • Store Variable writes the event ID to the Datastore with a 24-hour TTL, so the key expires on its own after the retry window
• • HTTP Request forwards the now-known-unique event to your real handler

Duplicates hit the Condition, stop, and never reach your downstream system — no Redis, no schema migration. The per-request run trace shows each step's phase and status, so you can watch a duplicate get stopped mid-chain. The Datastore page in the app sidebar lets you browse keys by prefix and namespace, and the same data is reachable via the REST API and MCP tools if you want your AI assistant to inspect dedup state. For more on what automation chains can do, see running custom JavaScript on every webhook and conditional webhook routing. Plan details are on the pricing page.

To be clear about the boundary: Datastore dedup is a great fit for automation pipelines, internal tooling, and keeping noisy duplicates away from downstream systems. For money-touching production handlers, keep the dedup check in your own infrastructure where it can share a transaction with your side effects — and use Hooklistener to prove it works before you ship.

Idempotency Checklist

Related Resources