The pattern
A retry wrapper re-executes work that already had a side effect, or keeps retrying failures that can never succeed. Retries convert one failure into several, and a timeout into a duplicate. The core questions are always: is this operation idempotent, is this error retryable, and what is the total budget?
( 01 )Symptoms
How this failure announces itself.
- warningDuplicate writes - two orders, two emails, two charges - clustered around timeouts.
- warningLong-tail latency that is a near-exact multiple of the per-attempt timeout.
- warningAn operation reports failure to the user but actually succeeded on a later, silent retry.
- warningError rates that climb under load as retries multiply traffic against a struggling dependency.
( 02 )First moves
The first ten minutes — establish facts before touching code.
- 1Find one duplicated side effect and pull the logs for its request id - count the attempts and note what triggered each.
- 2Classify the triggering error: transient (network blip, 503) or permanent (validation, auth)?
- 3Locate where the side effect happens relative to the failure - did the first attempt complete after the caller gave up waiting?
- 4Map every retry layer in the path (client, gateway, SDK, queue) - stacked retries multiply each other.
( 03 )Where to look
The code and config that usually owns this bug.
- searchRetry conditions - which error types re-enter the loop, and are permanent failures (validation, auth) among them?
- searchBackoff policy - fixed, exponential, jittered? Zero backoff under load is an outage amplifier.
- searchIdempotency guards - what stops attempt two from repeating attempt one's side effect?
- searchThe total timeout budget - per-attempt timeout times attempts, versus what the caller can actually tolerate.
( 04 )Common fixes
Fix the cause, then make the regression impossible.
- buildMake the operation idempotent (keys, upserts, dedup tables) before tuning retry counts.
- buildStop retrying non-retryable failures; classify errors explicitly instead of catching everything.
- buildEnforce one total deadline across all attempts, not just per-attempt timeouts.
- buildUse jittered exponential backoff so synchronized clients do not hammer a recovering dependency.
( 05 )Prove the fix
A fix you can't demonstrate is a guess. Close the loop.
- verifiedInject the original failure in a test and confirm the side effect happens exactly once across all retries.
- verifiedConfirm permanent failures now fail fast, with no retry attempts in the logs.
- verifiedCheck that worst-case end-to-end latency stays inside the caller's deadline.