The pattern
Networks deliver at-least-once: clients retry after timeouts, webhook providers replay events, queues redeliver messages. If the service performs the side effect on every delivery, duplicates are guaranteed eventually. Idempotency means the first durable result wins and every replay returns that same result.
( 01 )Symptoms
How this failure announces itself.
- warningDuplicate orders, invoices, or emails that cluster around timeouts and provider incidents.
- warningA client timeout followed by a retry creates two resources with different ids.
- warningWebhook replays carrying the same event id change state twice.
( 02 )First moves
The first ten minutes — establish facts before touching code.
- 1Find two duplicate resources and compare their originating request or event ids - the same id means replay handling failed; different ids mean the client retried without a key.
- 2Trace one duplicate end to end: when was the dedupe check, when was the side effect, and what could interleave between them?
- 3Check whether the dedupe store and the side effect share a transaction; if not, find the crash window.
- 4Reproduce by replaying the same request or event twice in a test.
( 03 )Where to look
The code and config that usually owns this bug.
- searchIdempotency key handling - is the key read, validated, and actually used to deduplicate?
- searchThe processed-event store - does anything durably record which event ids have been handled?
- searchSide-effect ordering - is the dedupe record written before or after the side effect? The gap between them is the duplicate window.
- searchReplay responses - does a duplicate request get the original result, an error, or a second execution?
( 04 )Common fixes
Fix the cause, then make the regression impossible.
- buildPersist the idempotency key or event id durably before acknowledging - ideally in the same transaction as the side effect.
- buildDeduplicate on the provider's event id for webhooks; never assume a provider will not call twice.
- buildOn replay, return the stored first result with its original status - replays should be observationally identical.
- buildAdd duplicate-delivery tests to CI: every consumer must survive the same message twice.
( 05 )Prove the fix
A fix you can't demonstrate is a guess. Close the loop.
- verifiedDeliver the same request or event twice in a test: exactly one side effect, identical responses.
- verifiedCrash the process between side effect and acknowledgement in a test and confirm redelivery does not duplicate.
- verifiedWatch production metrics for duplicate side effects after the deploy, not just test results.