Retrying a non-idempotent side effect: find the duplicate operation

The symptoms

• •Customer support tickets describing a single user action (for example one "Submit" click) producing two confirmations, two receipts, or two visible downstream effects, with timestamps seconds apart and one or more requests returning 502 or a transport-level timeout.
• •Server logs showing the same business key (order ID, payment ID, email recipient) processed twice in close succession, where the second request carries a fresh nonce or new correlation ID rather than the original one.
• •Database or queue audit tables containing two completed rows for one logical event while the application-side "did the first call succeed?" check returns "unknown" or "false."
• •Metrics dashboards showing a retry rate that exceeds the rate of confirmed upstream failures, indicating retries are happening even on operations whose first outcome could not be verified.
• •Reconciliation jobs flagging near-duplicate ledger entries with identical amounts, identical parties, and timestamps inside the retry window.

Likely causes

• •The client treats any non-2xx response, connection reset, or read timeout as a definite failure and retries, even though the upstream may have already committed the side effect before the network failure.
• •The server endpoint is non-idempotent at the protocol or business level: it performs write actions tied to the request body rather than a stable client-supplied key, so each retry creates a new effect.
• •An intermediary (proxy, load balancer, sidecar) returns 502 Bad Gateway during partial upstream failure, per MDN's definition, and the client interprets the 502 as a retry cue rather than an ambiguous outcome. Source: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/502
• •Retry middleware is configured with a static backoff that does not consult any prior-attempt record, so a second duplicate request is dispatched before any idempotency cache or dedupe store can answer.
• •The system lacks, or has lost, a request-deduplication layer keyed on a stable client-supplied identifier, so the second request looks brand new to the service.

First ten minutes

1. 01Stop further retries for the affected operation class: pause the auto-retry job or feature flag so additional duplicates are not generated while you investigate, and capture the current in-flight queue.
2. 02Identify the exact operation type and boundary (for example POST /payments vs POST /transfers) by looking at the user-reported ticket or the most recent error log entry, and confirm whether the endpoint is documented as idempotent.
3. 03Pull the request/response log pair for the suspected first attempt and the suspected duplicate, and record the HTTP status, request ID header, body hash, and timestamp for both; per MDN, a 502 indicates the server got an invalid response from the upstream, which is an ambiguous-not-definite outcome. Source: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/502
4. 04Cross-check the audit or write-ahead store for the same business key to determine whether the first attempt actually committed its side effect before deciding the second attempt was the duplicate.
5. 05Document your tentative classification (duplicate execution vs duplicate delivery only) and the evidence ID for each row before making any code or retry-config change.

Evidence to collect

• •The HTTP status codes of both attempts, with special attention to 502, 504, connection reset, and read timeouts because these are ambiguous completion signals.
• •Request ID, correlation ID, Idempotency-Key header (if present), and a body hash for each attempt, taken from the access log or the client trace.
• •Server-side audit, ledger, or outbox rows for the business key, including commit timestamp and the writer identity (process, pod, or worker ID).
• •Retry middleware configuration: max attempts, backoff schedule, and whether the retry decision consults a prior-attempt store.
• •Downstream side effects observable outside the request path, such as notifications dispatched, funds moved, or messages enqueued, with their timestamps.

Where to look

• •The client retry boundary: the SDK, middleware, or wrapper that decides "did this fail, should I retry" without consulting prior-attempt state.
• •The server handler boundary: the route or RPC method that performs the side effect, where idempotency-key handling should occur before any write.
• •The audit and outbox boundary: the durable record that proves whether a write actually committed, independent of the HTTP response the client saw.
• •The proxy and gateway boundary: the layer that can return 502 to the client when the upstream is unreachable or returns an invalid response, per MDN. Source: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/502
• •The reconciliation boundary: the batch job or stream consumer that compares expected effects to observed effects and flags duplicates after the fact.

Diagnostic steps

1. 01Compare the two attempts' request bodies, headers, and timestamps: if the body is identical but the request IDs differ, classify the second as a retry-without-key; if both IDs and bodies match, look for a double-dispatch at the client.
2. 02Query the durable write store for the business key with a time window spanning both attempts; a single committed row indicates the first attempt succeeded despite an ambiguous response, while two committed rows indicate duplicate execution.
3. 03Inspect the server handler for idempotency-key handling: trace whether the key is read, whether a dedupe record is created before the side effect, and what happens when a key is reused with a different body.
4. 04Reproduce the failure boundary under controlled conditions by simulating a network-level timeout (for example a process that drops the response after the server commits) and observing whether the client retries.
5. 05Review proxy or load balancer configuration to determine whether a 502 originated from an upstream timeout versus an upstream connection error, since these have different implications for whether the side effect ran. Source: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/502
6. 06If a reconciliation report flags near-duplicates, correlate the duplicates back to the originating request pair using the request IDs from the access log to confirm the retry path.

Common mistakes

• •Assuming a non-2xx response means the operation did not run; many side effects commit before the response is sent, so a 502 or timeout can still leave a completed write behind.
• •Adding retries without adding an idempotency key, which guarantees that any ambiguous completion will now produce a duplicate effect instead of a single confirmed one.
• •Comparing requests by correlation ID alone; a fresh retry often has a brand-new correlation ID, so the duplicate is invisible unless you also compare a stable business key.
• •Treating the duplicate as a benign double-delivery; for payments, transfers, or notifications, a duplicate is a correctness defect, not a performance defect, and must be reconciled at the business layer.
• •Implementing idempotency on the response cache only; if the dedupe record is keyed on the response rather than the request, a cached failure response can suppress a legitimate retry that would have succeeded.

Safe fixes

• •Add a client-supplied Idempotency-Key header to the request, generate it once per logical operation, and have the server persist the key together with the result before the side effect completes; a second request with the same key returns the stored result rather than re-executing.
• •Introduce a "status only" read endpoint the client can call when it sees an ambiguous response; only retry when the read confirms the prior attempt did not commit, and never retry on ambiguous outcome alone.
• •Make the side effect itself conditional on a server-side dedupe record keyed on the business identifier; treat writes as upserts keyed on the idempotency key, so a repeated request collapses to the original outcome.
• •Configure retry middleware to consult a short-lived prior-attempt cache keyed on the idempotency key, so a second dispatch within the retry window is suppressed rather than re-issued.
• •When a duplicate has already executed, do not attempt to "undo" by guessing; instead run a reconciliation pass that compares the two committed rows, determines which is the canonical one, and emits a corrective record the business layer can act on.

Prove the fix

1. 01Run a fault-injection test that drops the response after the server commits the side effect; verify that the client surfaces an ambiguous outcome, performs the read-before-retry check, and does not produce a second committed row.
2. 02Confirm via the audit or outbox table that for every ambiguous-response event in the test window, exactly one committed row exists per business key, not two.
3. 03Inspect the retry middleware counters: the retry rate should now be lower than the ambiguous-response rate, because ambiguous responses are routed through the read-before-retry path rather than the automatic retry path.
4. 04Replay the reconciliation job against the test window and confirm it reports zero duplicates for the fixed operation type, where it previously reported one duplicate per ambiguous response.
5. 05Verify that a replayed request with the same idempotency key returns the original response body and status, and that a replayed request with a different body but the same key is rejected as a key-mismatch, per standard idempotency semantics.

Prevention and next steps

• •Treat idempotency as a first-class requirement at every endpoint that performs an externally visible side effect; require an Idempotency-Key header and reject requests that omit it on write paths.
• •Separate the "did the call reach the server" signal from the "did the side effect commit" signal in client SDKs, and route ambiguous outcomes through a read-before-retry path rather than an automatic retry path.
• •Run periodic chaos drills that simulate response loss after commit, and assert that no duplicate side effects are produced; this catches regressions in the dedupe layer before customers do.
• •Maintain a reconciliation job that runs on a short cadence and compares expected vs observed side effects per business key; treat any duplicate finding as a paging alert, not a logging event.
• •Document, per endpoint, the exact contract for what constitutes an ambiguous completion and which error codes qualify, so client teams do not have to infer retry safety from HTTP semantics alone.

Safe commands and checks

grep -n "Idempotency-Key" <access_log_path> | head -n 50
grep -n " 502 " <access_log_path> | tail -n 100
awk '{print $1}' <access_log_path> | sort | uniq -c | sort -rn | head
grep -E "POST /payments" <access_log_path> | tail -n 200
find <log_dir> -name "retry*" -mtime -1 -ls