Race-condition checklist

The symptoms

• •Identical operation issued twice from different replicas succeeds on both, leaving the persisted state with two accepted records instead of one.
• •A read immediately after a write returns a value that contradicts a write observed on a different node seconds earlier, with no intervening error.
• •Intermittent integrity violations on a column with a uniqueness constraint, with the application log showing a successful insert rather than a constraint error.
• •Idempotency key or request token recorded as present, yet a duplicate downstream effect was observed, indicating the check and the act were not atomic.
• •Outcome of a workflow changes depending on the interleaving observed in trace data, while every individual component reports success.

Likely causes

• •Read-modify-write performed without row-level or document-level locking, where two transactions read the prior value, both compute an increment, and both write back, losing one update.
• •Check-then-act pattern across a network boundary, such as SELECT followed by INSERT or UPDATE, where the gap between the two statements allows a concurrent writer to slip in.
• •Application-side deduplication relying on a non-atomic exists-then-insert, instead of a database-enforced unique constraint or an atomic upsert.
• •Cache invalidation race where two readers miss the cache, both compute the value, and both write the cache, so the slower writer overwrites a fresher result.
• •Retry of a non-idempotent request after a timeout where the original completed but the response was lost, producing a duplicate side effect the system does not detect.
• •Logical clock or vector clock drift between nodes such that the conflict resolution chooses a causally older value as the winner.
• •Lock or lease that is process-local rather than cluster-wide, so two processes each believe they hold the same critical-section guard.

First ten minutes

1. 01Freeze further writes to the affected resource or feature flag if the duplicate side effect is still ongoing, and record the exact request identifiers and timestamps from the first three observed duplicates.
2. 02Pull the transaction log or access log for the affected table or document and filter by the suspected time window, listing every operation whose commit ordering disagrees with arrival ordering.
3. 03Open the distributed trace for two concurrent requests and verify whether the critical section is enclosed by a single span that holds a distributed lock, or whether it spans multiple spans across nodes.
4. 04Compare the application-generated identifier or idempotency key against the database's unique-index contents to determine whether the duplicate key was ever present at the moment of insert.
5. 05Capture the current isolation level, lock waits, and in-progress transactions using the standard statistics views before any restart, so the post-fix comparison is meaningful.

Evidence to collect

• •Concurrency: the number of concurrent writers, clients, or worker processes observed at the time of the incident, with timestamps aligned to a single time source.
• •Ordering: the commit timestamps and the client-observed arrival timestamps for the conflicting operations, exported from the database statistics views and the access log.
• •Atomicity: the exact statements, including the isolation level, that bracket the critical section, taken from the trace or the prepared-statement cache.
• •Identity: the idempotency keys, request tokens, or business keys that should have been unique, and whether the database index reported any violation or silently allowed the row.
• •Lock state: held locks, lock waits, and lock types at the moment of conflict, sourced from the database monitoring statistics views rather than application-internal counters.

Where to look

• •The database monitoring statistics views, which expose per-session lock waits, transaction start and stop times, and wait events that indicate serialization delays.
• •The application access log or query log, where the textual order of statements reveals the check-then-act gap between SELECT and INSERT or UPDATE.
• •The distributed tracing backend, focusing on the critical section span and whether it crosses a network or queue boundary where a lock cannot be held end-to-end.
• •The cache layer's key and version metadata, where a missing version increment or absent compare-and-set is the concrete marker of an unsynchronized write.
• •The message broker or queue's delivery and acknowledgement records, which expose duplicate delivery of a non-idempotent message that triggered the race.
• •The application's in-process lock registry, which is the boundary to inspect when the team believed a cluster-wide lock was in force but each process held its own.

Diagnostic steps

1. 01Reproduce with concurrency: run two near-simultaneous requests with distinct identifiers through a staging replica and observe whether the database statistics view reports a unique-constraint violation or two successful inserts.
2. 02Confirm the boundary: identify whether the race is intra-row, cross-row, cross-node, or cross-service by mapping every shared mutable state to a single owning component, and decide whether the fix belongs at the database, cache, queue, or application layer.
3. 03Inspect the critical section: read the exact SQL or driver call sequence, not the high-level ORM description, and confirm whether the isolation level is read-committed, repeatable-read, or serializable, since each level defines different anomaly guarantees.
4. 04Quantify the window: from the trace and log, measure the time between the read and the write inside the critical section, and compare it against the observed inter-arrival time of competing requests. If the window is wider than the inter-arrival time, a race is structurally possible.
5. 05Distinguish lost update from phantom: if both writers saw the same prior value, classify as lost update; if one writer saw a row the other did not, classify as phantom, and select an isolation-level or constraint-based remedy accordingly.
6. 06Verify lock scope: confirm that any lock taken is held by the same session or transaction for the full duration of the critical section, and that the lock name is cluster-scoped rather than process-local.
7. 07Test idempotency: replay the same request with the same idempotency key and confirm that exactly one downstream effect is observed; failure here is direct evidence the system is not deduplicating atomically.
8. 08Decide remediation layer: based on the boundary, choose between a database-enforced unique constraint, an explicit row lock, an optimistic version column with compare-and-swap, or a distributed lock service, and document why the chosen layer matches the boundary.

Common mistakes

• •Adding an application-level exists check before insert instead of a database unique constraint, which moves but does not remove the race window.
• •Increasing the isolation level without first confirming that the application can tolerate the resulting serialization failures, which can convert a race into a deadlock.
• •Trusting ORM-generated statements without reading the actual SQL, which can hide that a SELECT and an UPDATE were issued as separate auto-committed statements.
• •Using GET-then-PUT on a cache without a version token, which recreates exactly the read-modify-write pattern the cache was meant to avoid.
• •Assuming a single-node test is representative, since a race condition can require true cross-process or cross-node concurrency to manifest.
• •Retrying on timeout without an idempotency key, which is functionally equivalent to issuing the request twice and expecting the system to detect the duplicate.

Safe fixes

• •If the race is a lost update on a single row, add a database-enforced unique constraint on the business key, or add a version column and require the UPDATE to match the previously read version, returning a conflict signal on miss.
• •If the race is a check-then-act across a network, enclose the read and the write in a single explicit transaction with row-level locking, and verify in the database statistics view that the second writer waits rather than proceeds.
• •If the race is in a cache write, replace the unconditional PUT with a compare-and-set keyed on the version the reader observed, and have the writer re-read on conflict rather than overwriting.
• •If the race is duplicate delivery from a queue, make the handler operation idempotent by checking the idempotency key inside the same transaction that performs the side effect, and have the broker track the key.
• •If the race is between processes that each believed they held a cluster-wide lock, replace the in-process guard with a distributed lock that includes a fencing token, and require downstream resources to validate the token.
• •Each fix is conditional: apply only after the diagnostic step has classified the race as lost update, phantom, duplicate delivery, or lock-scope mismatch, and only at the boundary where that classification was located.

Prove the fix

1. 01Replay the original two concurrent requests against the fixed system and observe in the database statistics view that exactly one insert succeeded, the other received a unique-constraint violation or a serialization failure, and the affected row count is one.
2. 02Run a sustained load test that issues N concurrent requests with distinct keys and confirm that the database reports exactly N rows, with no duplicates, no retries creating extra effects, and no increasing lock-wait counter over time.
3. 03Replay the same request with the same idempotency key M times and confirm that the trace shows a single downstream effect and M-1 short-circuit responses, with the idempotency record present in the same transaction as the side effect.
4. 04Capture the trace of the critical section and confirm that the lock acquire and the side-effect write are inside the same span, that the lock is released only after commit, and that the trace records no span that holds the resource without a lock.
5. 05Re-run the post-fix statistics snapshot and compare the lock-wait distribution and the transaction duration distribution against the pre-fix snapshot, requiring the regression criteria to be met before the change is considered verified.

Prevention and next steps

• •Treat every cross-process or cross-node read-modify-write as a candidate race, and require an explicit atomicity mechanism, either a database constraint, a versioned update, or a distributed lock with a fencing token.
• •Make every externally triggered action idempotent by default, with the idempotency key stored in the same transaction as the side effect, so retries and duplicate deliveries are safe by construction.
• •Add a chaos test in the continuous integration pipeline that issues concurrent requests for the same key and asserts on exactly-once downstream effects, treating any duplicate as a release-blocking failure.
• •Document the isolation level, the lock scope, and the conflict-resolution rule for each shared resource, and review the document whenever the owning component changes.

Safe commands and checks

psql -h <host> -p <port> -U <user> -d <database> -c "SELECT pid, datname, usename, state, wait_event_type, wait_event, xact_start, query_start FROM pg_stat_activity WHERE state <> 'idle' ORDER BY xact_start;"
psql -h <host> -p <port> -U <user> -d <database> -c "SELECT locktype, relation::regclass, mode, granted, pid FROM pg_locks WHERE NOT granted;"
psql -h <host> -p <port> -U <user> -d <database> -c "SELECT relname, seq_scan, idx_scan, n_tup_ins, n_tup_upd, n_tup_hot_upd, n_live_tup, n_dead_tup FROM pg_stat_user_tables ORDER BY n_tup_ins DESC LIMIT 20;"
psql -h <host> -p <port> -U <user> -d <database> -c "SELECT datname, conflicts, deadlocks FROM pg_stat_database WHERE datname = current_database();"
psql -h <host> -p <port> -U <user> -d <database> -c "SHOW transaction_isolation; SHOW default_transaction_isolation;"
grep -n -E 'INSERT INTO|UPDATE .* SET|SELECT .* FOR UPDATE' <access-log-path> | head -n 200
awk '{print $1, $4}' <access-log-path> | sort | uniq -c | sort -nr | head -n 50
grep -n -E 'duplicate key|serialization failure|deadlock detected|unique constraint' <application-log-path> | head -n 200