The pattern
Exclusive work spans multiple machines, and the lock's guarantees rest on expiry, cleanup, and clocks. Any mismatch - a TTL shorter than the work, a release without ownership proof, client clocks deciding expiry - and two workers both believe they own the same job.
( 01 )Symptoms
How this failure announces itself.
- warningThe same scheduled job runs twice on different hosts.
- warningAfter a failure, retries stay blocked because a dead worker still appears to hold the lock.
- warningWorkers disagree about whether a lock is expired - usually a clock-source mismatch.
( 02 )First moves
The first ten minutes — establish facts before touching code.
- 1Collect both workers' logs for one double-run and build a single timeline of acquire, heartbeat, expiry, and release events.
- 2Determine why the second worker got the lock: did the first one's TTL lapse mid-work, did someone release a lock they no longer owned, or did clocks disagree?
- 3Compare the TTL with the actual p99 duration of the protected work.
- 4Inspect the release code for a missing owner-token check.
( 03 )Where to look
The code and config that usually owns this bug.
- searchAcquire and release paths - are they atomic primitives, or racy check-then-act sequences?
- searchOwner tokens - does release verify the releaser still owns the lock?
- searchTTL versus heartbeat - long work must extend the lock before expiry, not assume it survives.
- searchClock source - expiry must be decided by the lock server's clock, never by each client's.
( 04 )Common fixes
Fix the cause, then make the regression impossible.
- buildRelease locks in finally paths, with an owner-token compare so you cannot free a successor's lock.
- buildHeartbeat to extend the TTL during long work, and treat heartbeat failure as a stop-work signal.
- buildAlign queue visibility timeouts with lock TTLs so the two systems agree about ownership.
- buildLet the lock service's clock decide expiry (server-side TTL), never client timestamps.
( 05 )Prove the fix
A fix you can't demonstrate is a guess. Close the loop.
- verifiedTwo workers race for the lock in a test: exactly one runs, the other backs off cleanly.
- verifiedKill the owner mid-work: the lock self-heals after the TTL and a successor completes the job exactly once overall.
- verifiedStall the owner past its TTL in a test and confirm it detects lost ownership and aborts instead of finishing.