Debugging Stale Locks

How this failure announces itself.

• warningA pipeline or job stays in a running state long after its process died.
• warningWorkers sit idle while the queue grows - they are all waiting on the same lock.
• warningManual retries fail immediately because another run is supposedly in progress.

The first ten minutes — establish facts before touching code.

• 1Find the stuck lock's owner: which process acquired it, when, and is that process still alive?
• 2Read the owner's last logs - did it crash, hang, or exit cleanly without releasing?
• 3Map every code path from acquire to release and list the ones that can exit without releasing.
• 4Compare the TTL against the real duration of the work it protects.

The code and config that usually owns this bug.

• searchLock TTLs - infinite or hours-long TTLs turn a crash into an outage.
• searchError cleanup - is the release in a finally block, or only on the happy path?
• searchAtomicity - check-then-set acquire sequences race; releases without owner checks can free someone else's lock.
• searchOwner tokens - can a slow process release a lock that already expired and was re-acquired by another worker?

Fix the cause, then make the regression impossible.

• buildSet bounded TTLs sized to the work plus margin, so crashed owners self-heal.
• buildRelease locks in finally blocks so every exit path cleans up.
• buildStore an owner token at acquire time and require it at release, so only the owner can release.
• buildUse atomic primitives for acquire and release (SET NX PX, compare-and-delete) - never check-then-set.

A fix you can't demonstrate is a guess. Close the loop.

• verifiedKill the owner mid-work in a test and confirm the lock frees itself within the TTL.
• verifiedThrow inside the critical section and confirm the lock is released on the error path.
• verifiedRun two competing workers and confirm exactly one proceeds while the other waits or fails cleanly.