Distributed lock never releases: compare lease ownership and expiration

The symptoms

• •Workers block on lock acquisition long after the supposed owner process is no longer present in process listings, service registries, or load-balancer member lists.
• •Lock keys remain present in the coordination store (Redis, ZooKeeper, etcd) with TTL fields that never reach zero, while no active holder can be correlated.
• •Acquisition failures log monotonically growing wait counts but the lock's expiry timestamp keeps extending in lockstep with the waits, indicating background renewal rather than abandonment.
• •Critical-section completion events stop firing, yet the lock's value keeps being rewritten by a process that appears in metrics as idle or paused.
• •Retry storms after a partial outage leave every lock slot occupied, and manual deletion of one key is immediately re-created by another component.

Likely causes

• •The lock store's TTL is longer than the holder's crash detection latency, so an orphaned lock cannot expire before other workers conclude the holder is gone.
• •A renewal goroutine or scheduler runs even after the protected work has succeeded or aborted, repeatedly extending the lease while the application believes it already released.
• •Release logic writes to a different key namespace than acquisition (prefix mismatch, hash tag difference in Redis Cluster), so the unlock targets a non-existent key and the original lease persists.
• •Clock skew between the holder and the store causes the holder to compute remaining lease time incorrectly, leading to premature or perpetual renewals.
• •Network partitions cause the holder to assume it still owns the lock while the store has already reassigned or expired it, producing two competing renewers.

First ten minutes

1. 01Confirm the symptom scope: list the lock keys in the coordination store and record each key's value, owner identifier field, and expiry timestamp before changing anything.
2. 02Cross-reference the recorded owner identifiers against the current process inventory and service registry to separate truly orphaned locks from those with a live holder.
3. 03Sample the TTL/expiry field of each suspect lock at short intervals to distinguish a static lock (no changes, expiration approaching) from a renewing lock (expiry keeps moving outward).
4. 04Search application logs for the last successful critical-section completion event correlated with each lock key, to determine whether the holder believes it has finished.
5. 05Inspect the renewer subsystem: scheduler queues, lease-refresh timers, or heartbeat emitters should be checked for each owner identifier to see whether a renewer is still active for a supposedly done task.
6. 06Establish a baseline of expected lock TTL and renewal cadence from configuration so subsequent comparisons have a reference rather than guesswork.

Evidence to collect

• •List of lock keys with their stored value, owner identifier, and expiration/TTL field at a known timestamp.
• •Process inventory, pod list, or service registry entries matching each owner identifier, including last-seen or heartbeat time.
• •Application log entries showing acquire, release, renewal, and critical-section completion events keyed by lock name and owner identifier.
• •Configuration values for lock TTL, renewal interval, and fencing token generation, with the file or config key each was read from.
• •Time synchronization evidence (NTP offset, clock skew indicator) for the holder host and the lock store host at the moment the anomaly started.

Where to look

• •The boundary between the lock client library and the coordination store: inspect the acquire and release call sites and the key construction, since prefix or hash-tag mismatches isolate acquire from release.
• •The renewer/heartbeat subsystem inside the application: scheduled tasks, background workers, and timer queues that issue extend-lease operations even after the protected task has returned.
• •The configuration layer that defines lock TTL and renewal interval: environment variables, config files, and feature flags that may have been changed at deploy time.
• •The clock domain boundary: the system clock of the application host versus the server clock used by the coordination store to evaluate expiration, since divergent clocks invalidate lease math.
• •The crash-detection boundary: how quickly the rest of the system observes that a holder is gone, compared with the configured lock TTL, to determine whether expiry can ever win over abandonment.

Diagnostic steps

1. 01Snapshot all lock keys and their expiry values, then sample again after a short interval; a static expiry countdown indicates an orphaned lock, while a moving expiry indicates a live renewer.
2. 02For each lock whose expiry moves outward, find the renewer instance by correlating the owner identifier with scheduler threads, timer wheel entries, or heartbeat emitters in the application runtime.
3. 03Compare the application's recorded "work complete" timestamp with the renewer's last refresh timestamp; if the renewer continued after completion, the renewer is the fault.
4. 04Diff the acquire path's key construction against the release path's key construction, including any key prefixes, hash tags, or serialization steps, to detect a namespace mismatch.
5. 05Measure the offset between the application host's clock and the coordination store's reported time at the anomaly window; offsets exceeding a small fraction of the renewal interval invalidate the lease math.
6. 06Determine whether the lock TTL is longer than the system's crash detection time; if so, even a healthy failure mode produces orphaned locks that no one can reclaim before expiry.
7. 07Classify each suspect lock as orphaned-with-static-expiry, renewed-after-completion, or release-routed-wrong-key, and proceed only along the branch matching the evidence.

Common mistakes

• •Deleting a lock key without first confirming the renewer is dead, which causes a live holder to lose its lease mid-work and produces split-brain effects.
• •Assuming "TTL is set" means "TTL will fire," without verifying that the renewer stops extending the lease when the critical section ends.
• •Comparing lock keys by display name only, ignoring namespace prefixes or hash tags that cause acquire and release to address different keys.
• •Reading the application host clock to interpret expiry values, instead of using the store's reported server time for the same field.
• •Increasing the lock TTL to "buy time," which amplifies the window during which an orphaned lock blocks new acquisitions and makes the fault harder to expire out.

Safe fixes

• •If evidence shows a renewing lock with no active critical section, stop the renewer first (disable the scheduler task or feature flag for that owner) and only then allow the existing TTL to expire naturally.
• •If evidence shows an orphaned lock with a static expiry countdown, reduce risk by lowering the lock TTL to the minimum that still exceeds the longest legitimate critical section, so future abandonments clear faster.
• •If evidence shows a key-namespace mismatch between acquire and release, fix the key construction so release targets the same key acquire created, and add an assertion that release fails loudly when the key does not exist.
• •If evidence shows significant clock skew between holder and store, switch to the store's reported time for expiry calculations and add a skew check that warns before renewing.
• •If crash detection latency exceeds the lock TTL, extend crash detection (faster heartbeats, registry pruning) rather than lengthening the TTL, so abandonments are observed before expiry.

Prove the fix

1. 01After a controlled workload, observe lock keys reaching zero TTL without manual deletion, proving expiry is no longer being blocked by a stray renewer.
2. 02Re-run the renewal-after-completion scenario and confirm that the lease stops being extended within one renewal interval after the critical section ends.
3. 03Re-run the release-path scenario and confirm that release and acquire use byte-identical keys, verified by a logging assertion at both call sites.
4. 04Re-run the clock-skew scenario under a simulated offset and confirm that the renewer refuses to extend when the store-reported time disagrees beyond a defined threshold.
5. 05Measure end-to-end acquisition latency after abandonment and confirm it falls within the new lock TTL window rather than exceeding it, proving future orphans will clear.

Prevention and next steps

• •Make lease expiry authoritative by tying renewal to a single in-flight task token and stopping the renewer when the task completes, aborts, or times out.
• •Keep lock TTL shorter than crash-detection latency so that an observed abandonment is always cleared before another worker concludes the holder is gone.
• •Centralize lock key construction in one helper used by both acquire and release, and add an existence assertion on release to surface namespace drift early.
• •Use the coordination store's server time for all expiry and renewal math, and expose clock-skew as a first-class signal rather than an implicit assumption.

Safe commands and checks

List lock keys matching a known prefix and capture their current value and TTL field; record the output for before/after comparison (read-only, uses a generic list-keys pattern aligned with cache-aside inspection guidance).
Sample a single lock key's TTL at two timestamps a renewal interval apart; if the second TTL is greater than the first, a renewer is still extending it.
Grep application logs for the acquire and release call sites of a given lock name to verify that the constructed key string is identical at both sites.
Inspect the configuration source for lock TTL and renewal interval values and compare them against the values observed in the live lock record.
Check scheduler or timer diagnostics for the owner identifier of a renewing lock to determine whether the renewer thread is still scheduled.