The pattern
Money-moving workflows combine retries, third-party side effects, and concurrent state changes. Without a durable boundary - idempotency keys, balance guards, recorded workflow state - a timeout becomes a double charge and two concurrent refunds exceed the capture. Treat every step as: may run twice, may run in parallel, may crash midway.
( 01 )Symptoms
How this failure announces itself.
- warningA gateway timeout plus a retry produces two charges for one checkout.
- warningTwo concurrent refunds together exceed the captured amount.
- warningA failed multi-step workflow leaves inventory reserved or credits issued with no order behind them.
( 02 )First moves
The first ten minutes — establish facts before touching code.
- 1Pull the provider's records for one incident and line them up against your own state timeline.
- 2Identify the exact failure: the same idempotency key charged twice (provider issue), different keys (your retry bug), or interleaved balance reads (your race).
- 3Check what the code does after a gateway timeout - treat it as failed, retry blindly, or query the provider for the truth?
- 4Reproduce with two concurrent requests in a test against a mock gateway.
( 03 )Where to look
The code and config that usually owns this bug.
- searchGateway idempotency keys - generated per logical payment, or per attempt (which defeats them)?
- searchRefund balance checks - read-check-write without a lock or compare-and-swap races under concurrency.
- searchWorkflow state - is each completed step durably recorded before the next starts?
- searchReconciliation - what compares your records against the provider's, and how often does it run?
( 04 )Common fixes
Fix the cause, then make the regression impossible.
- buildDerive idempotency keys from the logical operation (order id plus purpose), store them, and reuse them on retry.
- buildGuard refund and capture balances with row locks or compare-and-swap, never read-then-write.
- buildRecord durable workflow state per step; on failure, compensate completed steps explicitly.
- buildAfter a timeout, query the provider for the operation's true status before retrying.
- buildReconcile against provider records on a schedule and alert on drift.
( 05 )Prove the fix
A fix you can't demonstrate is a guess. Close the loop.
- verifiedA concurrent duplicate-checkout test produces one charge at the provider mock and one order locally.
- verifiedA concurrent refund test never lets total refunds exceed the captured amount.
- verifiedKilling the workflow between steps leaves a state that compensation restores to consistency.