How to test an API idempotency key under duplicate delivery

The symptoms

• •Identical POST or PUT requests that carry the same idempotency key header return different response bodies, different resource identifiers, or different timestamps on each attempt.
• •A client retry storm produces duplicate downstream records, duplicate charges, or duplicate notifications even though the client sets an Idempotency-Key on every retry.
• •The server returns 502 Bad Gateway intermittently on the second or third delivery, masking whether the original request was actually processed before the upstream gateway failed.
• •Concurrent requests with the same key race and both succeed, indicating the server did not serialize the critical section against the key.
• •Conflicting bodies submitted under one key are both accepted instead of being rejected with a 422 or 409 conflict response.
• •The stored response expires or is evicted before retries stop arriving, causing late retries to be treated as new requests.

Likely causes

• •The server stores the idempotency key only in process memory or in an instance-local cache, so a retry that lands on a different node misses the prior result and creates a duplicate.
• •The idempotency record is written only after the side effect commits, so a crash or a 502 from the upstream gateway between the commit and the record write leaves no dedupe evidence and the next delivery is treated as new.
• •The idempotency layer hashes or normalizes the request body inconsistently across nodes, so two requests with one key but slightly different serializations are not recognized as duplicates.
• •The response cache TTL is shorter than the retry window of the caller, so legitimate retries arrive after the dedupe record has expired.
• •The endpoint allows the same key with a different body, which violates the idempotency contract and can mask or enable double-processing depending on the implementation.
• •Upstream network instability returns 502 on the response leg while the side effect has already persisted, and the client cannot tell whether the original delivery succeeded.

First ten minutes

1. 01Capture the exact request: HTTP method, path, headers including Idempotency-Key or equivalent, and a redacted body digest. Confirm the key is present, well-formed, and stable across retries.
2. 02Send two identical requests from a controlled client using the same key and body, in sequence, and record both responses verbatim, including status, headers, and body.
3. 03Send the same key with a deliberately different body and observe whether the server rejects the conflict with a 409 or 422 rather than silently accepting both.
4. 04Inspect the server-side idempotency store directly if permitted, or via an admin endpoint, to confirm whether the key exists, what body digest is bound to it, and what stored response is associated with it.
5. 05Check whether any intermediate proxy or gateway is rewriting or stripping the Idempotency-Key header before forwarding to the origin server, which would invalidate the contract.
6. 06Reproduce a 502 by introducing a transient upstream failure and observe whether the server still records the idempotency result, so the next retry returns the original response rather than re-executing.
7. 07Decide whether the failure is in the contract (key handling, storage, body binding, TTL) or in the transport (gateway stripping headers, upstream 502 masking commit state) before changing code.

Evidence to collect

• •The exact Idempotency-Key value sent on each attempt, with confirmation that no proxy mutated it between client and origin.
• •Full HTTP response pair for two sequential deliveries with the same key and body, showing status, response headers, and body equality.
• •A row in the idempotency store with the key, request body hash, response status, response body hash, and creation timestamp, plus any expiry timestamp.
• •Server logs for each attempt, correlated by request ID or by idempotency key, showing whether the side effect was executed once or twice.
• •A 502 response on one attempt and the corresponding upstream or proxy log line indicating whether the origin had already committed the side effect before the gateway tore down the response.
• •The configured TTL of the idempotency record and the maximum retry interval the client is configured to use.

Where to look

• •At the API gateway or reverse proxy boundary to confirm the Idempotency-Key header is forwarded verbatim and is not stripped, renamed, or duplicated.
• •At the idempotency store table or key-value namespace where keys are persisted, including any separate cache tier that fronts durable storage.
• •At the request handler entry point where the key is read and the dedupe decision is made, including any middleware that runs before the route handler.
• •At the side effect boundary, for example the database transaction, the message queue publish, or the payment processor call, to confirm dedupe precedes commit.
• •At the upstream service that returns 502, distinguishing a true upstream failure from a gateway-induced timeout where the origin was still processing.
• •At the response cache or replay layer that returns stored bodies for duplicate keys, to confirm TTL, eviction policy, and body hash comparison.

Diagnostic steps

1. 01Replay the exact request twice with one key and compare the response bodies byte-for-byte; equal bodies and one durable side effect confirm a working contract, divergence indicates a contract failure.
2. 02Replay the same key with a different body and verify the server returns 409 or 422 with a conflict marker, rather than accepting both and creating two effects.
3. 03Read the idempotency store row for the key after the first commit and confirm it records both the body hash of the accepted request and the response that should be replayed.
4. 04Force a 502 after the side effect has committed and confirm the stored response is still returned to the next retry, demonstrating that the dedupe record was written before the response was sent.
5. 05Force a 502 before the side effect has committed and confirm the next retry re-executes the effect exactly once, demonstrating that no partial record blocks a legitimate retry.
6. 06Send concurrent requests with one key from two clients and confirm only one reaches the side effect path while the other waits, replays the stored response, or returns 409.
7. 07Wait past the configured TTL and retry with the same key; confirm the server treats this as a new request unless the caller explicitly opts into longer retention.
8. 08Compare the body hash the server stored with the hash of the body the client actually sent, to detect normalization differences in encoding, whitespace, or field ordering that break dedupe.

Common mistakes

• •Treating a 502 from the gateway as proof that the side effect did not happen, when the origin may have committed before the response was lost; dedupe must be verified against the store, not inferred from HTTP status.
• •Storing the idempotency record only in a per-instance cache, so retries on a different node create duplicates despite the contract being nominally implemented.
• •Hashing the request body with a normalization that differs between client and server, so semantically identical requests do not match and dedupe silently fails.
• •Allowing the same key to be reused with a different body, which removes the safety property and enables duplicate or divergent effects under retry.
• •Setting the idempotency TTL shorter than the client's retry budget, so retries after the window are processed as new requests and create duplicates.
• •Writing the idempotency record after the side effect commits, which leaves a window where a crash or 502 between commit and record write causes a duplicate on the next retry.

Safe fixes

• •If the 502 is masking a successful commit, persist the idempotency record in the same transaction as the side effect, or write it before the response is sent, and configure the response to be replayed from that record on retry.
• •If keys are missing in the store, move idempotency state from per-instance caches to a shared store accessible from every node that can serve the endpoint, and verify the store is reachable on the failure path.
• •If duplicates succeed despite one key, add server-side body hashing bound to the key and reject any retry whose body hash does not match the stored hash, returning 409 or 422 with a clear error code.
• •If retries after the TTL create duplicates, extend the TTL to exceed the maximum client retry interval and document the retention window, or require callers to generate a fresh key for genuinely new requests.
• •If the proxy strips the header, configure the gateway to forward Idempotency-Key verbatim, validate that the header survives compression and HTTP/2 header coalescing, and add an integration test that asserts the header reaches the origin.
• •If concurrent requests race, serialize the critical section per key using a lock, a single-flight pattern, or a conditional write on the idempotency record so only one delivery reaches the side effect path.

Prove the fix

1. 01Run the sequential same-key replay test and observe byte-equal response bodies across at least three deliveries, with exactly one durable side effect recorded in the downstream system.
2. 02Run the same-key different-body test and observe a 409 or 422 conflict response, with no additional side effect recorded.
3. 03Run the concurrent same-key test from multiple clients and observe exactly one side effect, with all other deliveries returning the stored response or a conflict.
4. 04Run the post-TTL retry test and observe behavior consistent with the documented retention policy, either a replay if within window or a fresh execution if past window, with no silent duplicates.
5. 05Introduce a forced upstream 502 after commit and confirm the next retry returns the originally stored response, demonstrating that the dedupe record was written before the response was lost.
6. 06Confirm in the idempotency store that every observed key has exactly one stored response bound to one body hash, and that no key is associated with multiple distinct body hashes.

Prevention and next steps

• •Define a contract for the idempotency key header in API documentation, including required presence, format, length, uniqueness scope, TTL, and behavior on body mismatch.
• •Bind the idempotency record to the side effect in a single transactional step, or use an outbox pattern, so a crash or a 502 cannot leave a committed effect without a dedupe record.
• •Place idempotency storage in a shared, durable store with a TTL that exceeds the longest expected retry window for the endpoint.
• •Add integration tests in CI that replay the same key, replay the same key with a different body, and replay the same key concurrently, asserting on both response equality and downstream side effect count.
• •Instrument the handler to log the idempotency decision path, including key, body hash, store hit or miss, and whether the response was replayed, so future regressions are visible in production logs.

Safe commands and checks

curl -sS -i -H 'Content-Type: application/json' -H 'Idempotency-Key: <key-value>' -d '<body>' '<endpoint-url>'
curl -sS -i -H 'Content-Type: application/json' -H 'Idempotency-Key: <key-value>' -d '<body>' '<endpoint-url>'
curl -sS -i -H 'Content-Type: application/json' -H 'Idempotency-Key: <key-value>' -d '<conflicting-body>' '<endpoint-url>'
openssl dgst -sha256 <body-file>
awk 'NR==FNR{a[$0]=1;next}!a[$0]' first-response.txt second-response.txt
grep -nE 'Idempotency-Key|request-id|side-effect' <service-log-file>