Idempotency Keys for Payment Safety: A Practical Guide

A few years ago, a colleague of mine was woken up at 3 AM by an alert: 'Duplicate charges detected — over $50k in double payments.' A payment gateway timeout had caused the client to retry, and the API processed both requests. The bank didn't care about 'network issues.' The chargebacks cost the company $10k in fees and a dent in merchant reputation.

That story is why idempotency keys exist. They're the safety net that prevents a single network hiccup from becoming a financial catastrophe.

What Is an Idempotency Key?

An idempotency key is a unique identifier that a client sends with a request to indicate that the request can be safely retried. The server uses this key to detect duplicates and return the same result for the same key, even if the request is received multiple times.

In payment processing, the key is usually a UUID generated by the client. The server stores the key along with the response (or the fact that the request is being processed). On subsequent requests with the same key, the server returns the stored response without re-executing the operation.

Example HTTP request with an idempotency key header.

POST /v1/charges
Idempotency-Key: 7a3b9c2d-1e4f-5a6b-7c8d-9e0f1a2b3c4d
Content-Type: application/json

{
  "amount": 4995,
  "currency": "usd",
  "source": "tok_visa"
}

Where Most Implementations Fail

The naive approach is to store the key in Redis with an expiration of 24 hours. That's what my colleague's team did. Here's why it failed:

When the payment gateway timed out, the client retried immediately. The first request was still being processed — it had charged the card but not yet stored the idempotency key. The second request arrived, found no key in Redis, and proceeded to charge again. Race condition. Two charges, one order.

warning

Idempotency keys must be stored atomically with the first processing attempt. Using a read-check-write pattern without a lock is a race condition waiting to happen.

The Correct Approach: Lock, Store, Process

To avoid the race, you need a distributed lock on the idempotency key before processing. Here's the workflow:

1. Client sends request with idempotency key. 2. Server acquires a lock on the key (e.g., using Redis SETNX or a database row-level lock). 3. If lock acquired, check if key already exists in store. If yes, return stored response. If no, proceed to process the request. 4. After processing, store the key and response atomically (same transaction as the operation). 5. Release the lock. 6. If lock is not acquired (another thread is processing the same key), wait or return 409 Conflict.

Python Flask-like handler using Redis for locking and idempotency storage.

import uuid
from redis import Redis
import time

r = Redis()

def process_charge_with_idempotency(charge_data, idempotency_key):
    lock_key = f"lock:{idempotency_key}"
    data_key = f"idem:{idempotency_key}"

    # Acquire lock with 5 second timeout
    if not r.set(lock_key, "locked", nx=True, ex=5):
        raise ConflictError("Request already in progress")

    try:
        # Check if we already have a response
        existing = r.get(data_key)
        if existing:
            return existing

        # Process the charge (e.g., call Stripe)
        result = payment_gateway.charge(charge_data)

        # Store the result atomically with the processing
        r.set(data_key, result, ex=86400)  # 24h TTL
        return result
    finally:
        r.delete(lock_key)

What About Failures?

What if the charge succeeds but storing the key fails? Or the charge fails with a transient error? The key must be stored regardless of the outcome. The client should see the same error response on retry, not a different one. This is called 'idempotency of failure'.

Stripe's API is a good example: if a charge fails due to insufficient funds, retrying with the same key returns the same failure response — it doesn't retry the charge. Only if the original request never reached Stripe (e.g., network timeout before Stripe responds) will a new attempt be made.

If you only store successful responses, a failed request can be retried and succeed — but the client still sees the failure. That's a data integrity headache.

Key Expiration: Not as Simple as 24 Hours

How long should you keep idempotency keys? The answer depends on the operation's settlement time. For credit card payments, the authorization can be released after 7 days, but the charge can be disputed up to 120 days. In practice, keep keys for at least the maximum retry window plus a buffer.

For Stripe, they keep keys for 24 hours. That works because their API guarantees that after 24 hours, a new request with the same key is treated as a fresh request. But if your payment flow includes delayed settlement (e.g., ACH takes 3-5 days), you need longer TTLs.

Don't let the key store grow unbounded. Use a background job to clean up expired keys, or rely on Redis's TTL. For high-throughput systems, consider using a separate database table with an index on the key and a cleanup cron.

The 47-Day Idempotency Key Bug

T-0Client sends payment request with idempotency key K. Server processes and stores response. Key TTL set to 24 hours.
T+24hKey K expires and is deleted from Redis.
T+48hClient retries with same key K (network issue caused retry to be delayed). Server sees no key, processes again. Duplicate charge.

Lesson

Never expire an idempotency key before the client's maximum retry window. If clients can retry after days (e.g., offline mobile devices), use a persistent store with longer TTLs, or implement a key rotation protocol.

Testing Your Idempotency Guarantees

Your test suite should include:

- Retry exactly the same request and verify the same response. - Retry while the first request is still in-flight (race condition). - Retry after the key has expired (should process as new). - Send a different payload with the same key (should reject). - Simulate a partial failure: charge succeeds but storing the key fails.

Stripe's idempotency key documentation is a gold standard. They advise generating keys as UUIDs and never reusing them across different operations. They also recommend using a monotonic timestamp in the key to order requests, though that's more relevant for non-idempotent operations.

Shell script to verify idempotency — the two responses must be identical.

# Test: retry with same idempotency key
IDEM_KEY=$(uuidgen)
curl -s -X POST https://api.example.com/v1/charges \
  -H "Idempotency-Key: $IDEM_KEY" \
  -d '{"amount":1000}' > /tmp/resp1.json
curl -s -X POST https://api.example.com/v1/charges \
  -H "Idempotency-Key: $IDEM_KEY" \
  -d '{"amount":1000}' > /tmp/resp2.json
diff /tmp/resp1.json /tmp/resp2.json && echo "PASS: same response" || echo "FAIL: responses differ"

When Idempotency Keys Aren't Enough

Idempotency keys protect against duplicate requests from the same client. They do not protect against duplicate processing due to database replication lag, queue redelivery, or bugs in your logic. For that, you need idempotent operations at the storage level — e.g., using unique constraints or event sourcing.

If your payment flow involves multiple steps (authorize, capture, refund), each step needs its own idempotency key. A refund with the same key should not refund twice. This is where many systems break: they reuse the charge's key for the refund, or they don't enforce idempotency on refunds at all.

Percentage of payment failures caused by duplicate charges (Source: Stripe 2023)

Implementing Idempotency with a Database

Redis is great for low-latency, but if you can't tolerate data loss, use a SQL database. Here's a PostgreSQL example:

Use a table `idempotency_keys` with columns `key` (UUID, unique), `response` (JSONB), `created_at`, and `expires_at`. Insert the key with `ON CONFLICT DO NOTHING`. If the insert succeeds, process the request and update the response. If it fails (duplicate key), return the existing response.

The key insight: the unique constraint on the key ensures that only one request can insert. This is a simple, reliable lock without distributed locking. But it requires that the insert and the processing happen in the same transaction, or you risk inserting but then crashing before processing.

PostgreSQL schema and example queries for idempotency storage with unique constraint.

CREATE TABLE idempotency_keys (
    key UUID PRIMARY KEY,
    response JSONB NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    expires_at TIMESTAMPTZ NOT NULL
);

-- Create index for TTL cleanup
CREATE INDEX idx_expires_at ON idempotency_keys (expires_at);

-- Insert on first request
INSERT INTO idempotency_keys (key, response, expires_at)
VALUES ('7a3b9c2d-1e4f-5a6b-7c8d-9e0f1a2b3c4d', '{}'::jsonb, NOW() + INTERVAL '24 hours')
ON CONFLICT (key) DO NOTHING;

-- If inserted, process payment and update response
UPDATE idempotency_keys SET response = '{"status":"succeeded"}'::jsonb WHERE key = '7a3b9c2d-1e4f-5a6b-7c8d-9e0f1a2b3c4d';

lightbulb

Make sure your database connection uses a timeout and retry logic. A connection pool exhaustion can cause the same race condition you're trying to prevent.

Final Thoughts

Idempotency keys are a simple concept with deceptively tricky implementation details. The race condition at startup, the TTL trade-off, the failure response handling — each is a potential landmine. But getting it right means your payment system can survive network partitions, client retries, and even some bugs without losing money.

Start with a database-backed implementation for critical paths. Use Redis only if you're comfortable with the risk of key loss (e.g., for non-critical operations). And always test the race condition — it's the one that will bite you at 3 AM.

Frequently asked questions

What happens if an idempotency key is reused with a different request payload?

The API should reject the request with a 409 Conflict or return the original response. The key binds to the first request's payload and result — any mismatch is an error in the client.

How long should idempotency keys be stored?

At least until the payment settles (24-48 hours for card payments). For recurring subscriptions, keep keys for the billing cycle. Expire keys aggressively to avoid unbounded storage, but not before the client's retry window closes.

Can idempotency keys be used for non-payment operations?

Yes — any mutating operation that could be retried benefits from idempotency: user creation, file uploads, order placement. The same principles apply.

What's the difference between idempotency and deduplication?

Idempotency guarantees that the same request (by key) yields the same result, even if applied multiple times. Deduplication is a specific implementation that prevents duplicate processing — idempotency is the property, dedup is one mechanism.

Idempotency Keys: The Safety Net Your Payment API Needs