How to test idempotent webhook handling

The symptoms

• •Replay of the same provider-issued event identifier produces two or more rows in the database table the webhook represents
• •Customer-visible charge, message, or notification fires twice after the producer's automatic retry, despite the consumer logging only one accepted event
• •Logs show the same event identifier processed more than once within the consumer's claimed dedupe window with no intervening dedupe hit
• •Schema has a unique index on the side-effect row's natural key, yet duplicate rows still land because the index is added after the write or is enforced on a different column
• •A second instance or worker handling the same event in parallel produces two side effects because neither instance observed the other's dedupe record yet

Likely causes

• •Consumer stores only a payload hash as the dedupe key instead of the provider's stable event identifier, so any payload regeneration causes the duplicate to look like a fresh event
• •Dedupe check happens after the side effect rather than before, so retries interleave and double-write before the check ever reads state
• •Cache or database write for the dedupe key fails silently and the next delivery proceeds as if the event was unseen
• •The endpoint under test is actually a parallel consumer that bypasses the dedupe store entirely because the replay was pointed at the wrong host
• •The producer adds or refreshes generated fields such as request identifiers or timestamps between retries, defeating naive payload equality checks

First ten minutes

1. 01Identify the provider's stable event identifier from the official webhook documentation and confirm the consumer logs it on every delivery alongside the handler outcome
2. 02Enumerate the side effects produced per event type so the verification later has an explicit assertion target: database writes, outbound API calls, queued messages, sent notifications
3. 03Inspect the dedupe store directly: list the most recent keys, check TTL settings, and confirm whether the write path is atomic with the side effect
4. 04Pull the producer's recent delivery log, noting retry attempts, timestamps, and any manually triggered replays so the test can mirror them
5. 05Confirm signature verification runs before any dedupe read or side effect; the test environment must never exercise a code path that skips signature verification
6. 06Decide whether the test will replay real producer events or synthetic ones signed with a test signing secret; lock that decision before writing any assertion code

Evidence to collect

• •Producer-side event identifiers and delivery timestamps from the provider's dashboard or events endpoint, sufficient to identify which events were retried vs. delivered once
• •Consumer-side log lines pairing each event identifier with the handler exit status, the dedupe-store read/write outcome, and the side effect dispatched
• •Side-effect counts grouped by the provider's event identifier in the downstream store, on the queue, and in the notification subsystem
• •Signature header value and computed message authentication code for each replay, to confirm only verified events reached the dedupe layer
• •Configuration values for retry policy, handler timeout, and concurrent worker count, so the test reflects the production concurrency shape
• •Dedupe-store records before and after each replay, including any TTL expiry observed during the test window

Where to look

• •Consumer entry point: signature verification, early-return branches, and silent catch blocks that could swallow a failed dedupe write
• •Deduce persistence layer: write path order, key namespace, TTL, and read-after-write consistency in cluster or replica setups
• •Transaction boundaries around the side effect: confirm whether the dedupe insert and the side effect commit atomically, or whether a crash between them re-fires the side effect on retry
• •Producer's retry configuration and any manual replay tooling: retry schedule, exponential backoff, and the maximum retry window recorded by the provider
• •Schema for both the dedupe store and the resource being mutated: unique index definitions, constraint names, and any triggers that might bypass application logic

Diagnostic steps

1. 01Inject one known event identifier via the producer's test-event tooling or replay interface; capture the raw request body and headers to an evidence file before any consumer mutation
2. 02Verify the signature as the first operation against the replay; if signature verification fails, stop and report rather than proceeding with assertions
3. 03Snapshot the dedupe store before the test to confirm no prior record exists for this event identifier, then run the first delivery and snapshot again to capture the post-state
4. 04Replay the same event identifier a second time inside the consumer's claimed dedupe window and assert the downstream side-effect count remains at exactly one
5. 05Replay the same event identifier after an artificial crash mid-handler by terminating the process at a known safe breakpoint and confirm recovery does not produce a second side effect
6. 06Replay a payload whose only differences from the first delivery are generated fields such as timestamps or request identifiers, and verify the consumer still dedupes on the stable event identifier
7. 07Issue two concurrent deliveries of the same event identifier from two workers and confirm exactly one side effect emerges, with the second observing the first's dedupe record or losing the dedupe race deterministically

Common mistakes

• •Asserting dedupe from a payload hash instead of the provider's stable event identifier, so any payload regeneration between deliveries passes the duplicate test as a fresh event
• •Reading the dedupe store outside the transaction that performs the side effect, allowing a race window where two simultaneous deliveries both observe no record and both proceed
• •Skipping signature verification in the test harness because the focus is on idempotency, which hides authentication regressions and inflates false-pass results
• •Asserting only against log lines and not against actual downstream row counts, masking duplicate writes when logging is suppressed or off-path on the duplicate branch
• •Replaying through the producer's interface but checking against an environment whose dedupe store differs from production shape, including different namespaces, TTLs, or backing stores

Safe fixes

• •Persist the dedupe key with an atomic insert-or-ignore primitive so concurrent deliveries deterministically elect a single winner before any side effect is dispatched
• •Move the dedupe insert into the same transaction as the side effect, or implement an outbox where the side effect becomes observable only after the dedupe record is durably committed
• •Key dedupe on the provider's event identifier from the official webhook contract rather than on any payload-derived value that may change between retries
• •Bound retries at the handler boundary and surface non-retryable errors to the producer's dead-letter mechanism so silent retries cannot extend the dedupe-required window indefinitely
• •Add a contract harness that replays a fixed set of event identifiers and asserts row and notification counts, failing the build on any drift from exactly one

Prove the fix

1. 01After two sequential replays of the same event identifier inside the dedupe window, the downstream store contains exactly one row attributable to that event identifier and zero additional outbound calls were issued
2. 02An interleaved crash-and-replay scenario produces no additional side effect beyond the original successful outcome, with no orphan dedupe record preventing future unrelated events
3. 03A replay whose payload differs only in generated fields still short-circuits on the stable event identifier and never reaches the side effect branch
4. 04Signature verification is enforced on every replay; no code path under test bypasses signature checking even when the event body is otherwise trusted
5. 05Concurrent delivery of the same event identifier from two workers yields exactly one side effect and exactly one dedupe record, with the loser observing a deterministic conflict and not retrying the side effect

Prevention and next steps

• •Treat the provider's stable event identifier as the single source of truth for dedupe and document this choice in the consumer's published contract alongside the signing secret rotation policy
• •Run the replay-and-assert harness in continuous integration on every change to the handler, with fixtures covering normal delivery, replay within window, replay after crash recovery, and concurrent retry
• •Alert on a non-zero dedupe miss-rate, defined as events arriving more than once without a corresponding dedupe-hit record, as a service-level objective violation rather than a transient warning
• •Retain dedupe records for at least the producer's maximum retry window plus a documented safety margin, and prune them through a separate audited process rather than inline TTL expiry

Safe commands and checks

Replay fixtures/event.replay.json through the repository's test webhook client with a test event identifier and test signing secret; do not use production credentials.
Use the provider's official test-event tooling to submit the same fixture twice, preserving the original event identifier and captured signature headers.
Inspect the webhook consumer process table with the PID captured from the test service log; keep the check read-only.
Compare the captured consumer PID's elapsed time and command line with the test service start time.
Read the dedupe key for the captured event identifier from the test Redis namespace using the repository's configured test client.
Scan the test Redis namespace for recent dedupe keys and compare the count with replayed event identifiers.
Run the repository's read-only duplicate-row query against the test database using the configured test connection.
Read the consumer service log for the captured event identifier, signature verification result, and dedupe-hit marker over the test window.