All playbooks

Playbook 15 / 17

Debugging Database Consistency Bugs

How to trace query consistency issues across repositories, replicas, and API response contracts.

The pattern

The data is fine; the read is wrong. A repository filter forgets deleted_at, a cursor resumes from an ambiguous position, a read-after-write hits a lagging replica, an empty result is treated as an error. These bugs masquerade as data corruption, which is why the first instinct - repairing data - never fixes them.

( 01 )Symptoms

How this failure announces itself.

  • warningSoft-deleted rows reappear in lists or exports.
  • warningRecords repeat or vanish across pages.
  • warningA record returns 404 immediately after it was created, then appears seconds later.
  • warningEmpty collections surface as errors instead of empty states.
( 02 )First moves

The first ten minutes — establish facts before touching code.

  • 1Take one wrong read and run the underlying query by hand - if the raw query is right, the bug is in code; if wrong, in query construction.
  • 2List the invariants the read should apply (deleted, tenant, status) and check them against the actual WHERE clause.
  • 3For read-after-write 404s, log which node served the read; lag means routing, not data.
  • 4For paging issues, check sort uniqueness before anything else.
( 03 )Where to look

The code and config that usually owns this bug.

  • searchRepository filters - which query paths apply the soft-delete, tenant, and visibility invariants, and which forgot?
  • searchCursor construction - does the resume key uniquely identify a position under the current sort?
  • searchReplica routing - which reads are pinned to the primary after a write, and which are allowed to lag?
  • searchResponse helpers - does no-rows map to empty, null, or throw, and is that consistent across endpoints?
( 04 )Common fixes

Fix the cause, then make the regression impossible.

  • buildCentralize query invariants in scopes or base queries so deleted and tenant filters cannot be forgotten per call site.
  • buildUse stable, unique cursor ordering - sort key plus id.
  • buildPin read-after-write flows to the primary, or read your own writes via session consistency.
  • buildMake empty a first-class result everywhere; reserve errors for actual failures.
( 05 )Prove the fix

A fix you can't demonstrate is a guess. Close the loop.

  • verifiedRegression tests cover deleted, empty, and cross-page states on the failing endpoints.
  • verifiedRead-after-write returns the new record under induced replica lag in a test.
  • verifiedAn audit of sibling endpoints shows the same invariant applied - this class of bug repeats.