Stale distributed locks: diagnose ownership that outlives work

The symptoms

• •Critical sections execute concurrently with seemingly single-owner logic, producing duplicate side effects or interleaved writes.
• •Workers or cron jobs block indefinitely on a resource guarded by a lock while no observed owner process exists.
• •Lock records persist in the store long after the owning process heartbeat stopped, lease field is past expiry, yet no LUA/refresh path is firing.
• •Following a network partition, the previously-acquired lock is honoured by one partition and rejected by another, producing split-brain execution.
• •Recovery automation spins up a replacement worker, but it cannot acquire the lock because the stale entry still satisfies the uniqueness check.

Likely causes

• •Owner process crashed (SIGKILL, OOM, panic) without executing any release or extended-lease cancel path.
• •Lock acquired without a fencing token, so storage layers cannot reject stale writers that reappear after a pause.
• •Lease/TTL shorter than the longest possible GC pause, stop-the-world stall, or disk-suspend event, so the lock expired but the work did not.
• •Clock skew between lock service and owner caused early or late expiry; monotonic clocks not used for lease arithmetic.
• •Network partition longer than the configured TTL caused the store to drop the entry, but the owner continues to act as still-holding after reconnect.
• •Watcher/keyspace notification was relied on for release but the store lost the expiration event or the consumer dropped the event under backpressure.

First ten minutes

1. 01Confirm the symptom shape: enumerate processes that recently entered the guarded section, list the lock-store key, and compare against the expected single owner.
2. 02Decide whether the lock store is still authoritative or has been partitioned; inspect replication lag and quorum health before trusting any release.
3. 03Pull the lock entry metadata: owner identifier, acquisition timestamp, TTL/lease, and any fencing token. Compare lease against current monotonic time.
4. 04Look for the owning process by identifier in process listings, orchestrator state, and recent heartbeat logs; record last-seen time.
5. 05Check for any scheduled refresh/extend path and confirm whether it last ran before or after the suspected pause or crash.
6. 06Decide on a fenced break-glass release only after the ownership-evidence rules below are satisfied; do not delete keys speculatively.

Evidence to collect

• •Lock store entry: key, value (owner ID + token), PEXPIRE/PTTL, and persist vs volatile classification.
• •Owner process evidence: PID, container ID, orchestrator task ID, last heartbeat timestamp, last log line, and exit status if terminated.
• •Network and partition evidence: lock-store replication state, quorum size, partition start/end timestamps, and any client-side reconnect events.
• •Time evidence: monotonic clock delta between owner and lock store, NTP offset, and any recorded pause or stop-the-world duration.
• •Refresh path evidence: last successful lease extension timestamp, extension interval vs TTL ratio, and any extension failures.

Where to look

• •Lock-store boundary: the Redis/KV/database instance holding the lock key, including replicas and proxy hops.
• •Owner process boundary: the worker, scheduler, or service that acquired the lock, including its supervisor and orchestrator.
• •Time boundary: NTP/PTP daemon on owner and lock-store nodes, and any virtual-clock sources that influence lease arithmetic.
• •Network boundary: switches, service mesh sidecars, and DNS resolvers between owner and lock store, including partition detection outputs.
• •Coordination boundary: scheduled-job runners, queue consumers, and downstream resources that act on the assumption of single-owner execution.

Diagnostic steps

1. 01Capture the lock entry with TTL and value, then compute remaining lease: if remaining is positive and no owner is alive, the lock is stale only if the owner cannot resume within the lease window.
2. 02Compare owner last-seen timestamp against current time and lease remaining; if owner last-seen older than lease, the lock has logically expired even if the key still exists.
3. 03Verify replication/quorum state on the lock store; if a minority partition is being read, the entry there may be stale relative to the majority that already expired it.
4. 04Inspect the extension path: if extensions stopped earlier than lease remaining implies, the owner paused or the network dropped; correlate with GC/scheduler pauses.
5. 05Check for fencing-token propagation: protected resources must reject writes whose token is lower than the highest token observed; absence of this check is the root cause of stale-writer damage.
6. 06Decide release eligibility: only release the lock if owner identity is unverifiable AND the protected resource enforces fencing tokens AND the lease has elapsed past a safety margin.
7. 07After release, observe whether a new owner acquires within one extension interval and whether the protected resource rejects any in-flight write from the prior owner.

Common mistakes

• •Deleting the lock key on the suspicion of staleness without verifying owner identity, breaking the very safety property the lock was supposed to provide.
• •Using a wall-clock TTL when the owner or store may jump forward, causing premature expiry and double execution.
• •Setting the lease shorter than the worst-case GC pause, so a paused owner returns to find its lock already taken while its work continues.
• •Trusting the lock store without fencing tokens at the protected resource, so a stale owner can still overwrite newer state.
• •Auto-releasing any lock whose owner has not been seen for a fixed interval, ignoring partition scenarios where the owner is alive but unreachable.
• •Equating a zero TTL on listing with "no lock exists"; volatile keys may be evicted under memory pressure rather than by expiry.

Safe fixes

• •Introduce fencing tokens by monotonically incrementing a per-resource counter on each acquire and storing the token inside the lock value; reject writes whose token is not the highest observed.
• •Set TTL as a multiple of the safe refresh interval (e.g., 3x to 5x) and refresh using a Lua or watch-and-transaction script that checks owner identity before extending.
• •Use monotonic clocks for lease arithmetic on the owner side; rely on server-side expiry for authoritative release, never on owner-controlled wall-clock.
• •Make protected resources idempotent or version-checked so that a stale owner, even if it does write, cannot corrupt newer state.
• •On suspected owner death, require two independent signals (process gone AND lease elapsed past safety margin) before any operator-initiated release.
• •Persist only the latest known fencing token at the protected resource; reject any write carrying a token lower than the recorded maximum.

Prove the fix

1. 01Kill the owning process with SIGKILL during a guarded section; observe that the protected resource accepts no further writes from the dead owner after its lease elapses.
2. 02Inject a stop-the-world pause longer than the lease on the owner; after resume, confirm the owner fails to extend and that downstream resources reject its writes via fencing token.
3. 03Partition the owner from the lock store for longer than the lease; on heal, confirm the owner does not silently re-extend a lock the store already expired.
4. 04Force an NTP step forward on the owner beyond the lease; verify the owner treats the lock as gone and does not perform guarded work.
5. 05Run a chaos drill that releases a lock manually while the owner is alive; the owner must observe the loss and abort the guarded section within one refresh interval.
6. 06Confirm a new owner can acquire the lock within one lease interval after the prior owner is verifiably gone, and that the protected resource records the newer fencing token.

Prevention and next steps

• •Adopt a single, explicit contract for lock ownership: owner ID + fencing token + lease, persisted together and validated on every operation.
• •Bound lease duration against the longest realistic GC pause, scheduler stall, and network partition observed in the environment, with a safety multiplier.
• •Instrument lock-store operations with metrics: acquire, release, extend, expire, and force-release, each tagged with owner and token.
• •Alert when lease-to-refresh ratio exceeds a threshold or when an extension has not succeeded within one refresh interval.
• •Periodically rehearse the stale-owner drill in staging to confirm protected resources reject stale writes and recovery acquires succeed.

Safe commands and checks

redis-cli -u <lock-store-uri> GET <lock-key>
redis-cli -u <lock-store-uri> PTTL <lock-key>
redis-cli -u <lock-store-uri> CLIENT LIST | grep -E 'addr=|age=|idle='
redis-cli -u <lock-store-uri> INFO replication | grep -E 'role:|connected_slaves:|master_link_status:'
redis-cli -u <lock-store-uri> ROLE
ps -o pid,etime,stat,cmd -p <pid>
journalctl _PID=<pid> --since '<utc-iso8601>' --no-pager | tail -n 200
chronyc tracking | grep -E 'Last offset|Stratum|System time'