The payment that charged twice after a timeoutPayments & ReliabilityHardPreparing

00:00

Incident Dossier

The payment that charged twice after a timeout

Support sees duplicate charges when the payment provider times out after accepting a request. Trace the controller, retry wrapper, provider client, and idempotency boundary before changing the smallest safe layer.

Initial failing signalOne checkout creates two provider chargesExpected behavior: A retry reuses the original provider idempotency key. · actual behavior: The retry changes the key, so a timeout after acceptance becomes a second charge.

Codebase incident

Evidence desk

Codebase Incidents · 2

At 09:42 UTC, checkout support reports two settled charges for a single order. The payment provider shows the first request accepted before its response timed out. The application then retried the operation and recorded one successful payment result. Trace the retry boundary and the idempotency key rather than assuming the provider duplicated work on its own.

Customer symptom

A customer is charged twice after checkout briefly times out.

Expected

Every attempt for one order carries the same provider idempotency key.

Actual

The client derives the key from the retry attempt, so the provider sees distinct operations.

Architecture

Checkout controllerPayment serviceRetry wrapperExternal payment clientIdempotency storePayment config

Checkout controller → Payment service · order payment request
Payment service → Retry wrapper · bounded retry
Retry wrapper → External payment client · attempt
Payment service → Idempotency store · payment result
Payment service → Payment config · retry policy

Logs2

Retry timeline

09:42:11 payment attempt=1 key=ord-timeout provider=accepted response=timeout 09:42:12 payment attempt=2 key=ord-timeout:retry:2 provider=accepted response=ok

payment.retry

Provider ledger

order=ord-timeout charges=2 keys=[ord-timeout,ord-timeout:retry:2]

provider.audit

Metrics1

Payment retry outcomes

timeout-after-accept: 17; duplicate-charge alerts: 17; checkout request count: unchanged

payments.duplicate_charge

Recent changes

The provider timeout budget was lowered during a latency cleanup.
The retry wrapper was reused for payment operations.
The provider accepts a request before returning its timeout response.

Plausible hypotheses

Provider duplicated an accepted request — The first response timed out after the provider accepted the payment.
Retry changes operation identity — The duplicate appears only when the retry path runs, so request identity must cross that boundary.
Controller submitted checkout twice — A double-click or client replay can create duplicate requests at the API boundary.

Progress

1Reproduce the bugRun checks

2Edit the fixOpen

3Pass checksOpen

4Submit fixOpen

Reproduction Steps

Run the supplied incident test.
Follow the order id through the retry wrapper and provider client.
Compare the provider keys recorded for both attempts.
Apply the smallest stable-idempotency fix and rerun visible and hidden checks.

Focus Mode

Trace the order identity from the controller through retry.js into the provider client's key construction.

Start

README.md
src/http/controller.js
src/payments/service.js

Likely area

src/payments/retry.js
src/payments/client.js
src/idempotency/store.js

Investigation path

1Confirm the timeout-after-acceptance symptom.
2Trace retry attempt identity into the external client.
3Separate provider acceptance from application response success.
4Keep one stable idempotency key across all attempts and prove duplicate suppression.

Repo scale

15 meaningful files
HTTP, payment service, retry, provider, idempotency, config, and telemetry layers
One stable-key fix

Challenge Coach

Hints reveal broad direction first, then narrow toward the suspect path without showing hidden tests or exact solution code.

TopicPayments & Reliability

DifficultyHard

Est. time35 min

Workspacepreparing

Explorer

Loading files

Evidence Map

Failing signal loaded from challenge API
Workspace files loaded from workspace API
Results rendered only after API run/submit

Autosaved

workspace

Outputread-onlyready$ prepare workspace

Preparing workspace