How to prove a lock releases after a worker crash

The symptoms

• •Original lock-owning process becomes unreachable (process gone, network partition, or host lost) while the lock is still held in the coordination service.
• •Lock entry in the coordination service persists past the expected lease window with no renewal activity.
• •A second contender is observed acquiring the same lock key after the original owner disappeared, without explicit release.
• •Operations issued by the original owner before its disappearance arrive at the protected resource after the new owner has taken over.
• •Watchdog or lease-renewal threads stop emitting heartbeats coincident with the disappearance event.
• •Clock skew or NTP step is reported around the same window as the owner's last renewal attempt.

Likely causes

• •Lock was acquired without a bounded lease, or the TTL is shorter than the worst-case renewal latency, so a slow owner can outlive its lease without losing it.
• •Owner process died abruptly (SIGKILL, OOM kill, power loss, container eviction) and never executed any unlock or cancel paths.
• •Lease renewal thread is hosted in the same process as the lock user, so a process-wide crash stops both together.
• •Network partition isolated the owner from the coordinator while leaving the owner itself healthy, so the coordinator's TTL alone must decide ownership.
• •Coordinator clock advanced (NTP step or virtualization time skew) and expired the lease while the owner still believed it was valid.
• •No fencing token is associated with the lock, so the protected resource cannot tell which writer is still authoritative after recovery.

First ten minutes

1. 01Capture the exact disappearance time from at least two independent sources (application log timestamp, coordinator lease timestamp, and host uptime).
2. 02Inspect the coordinator entry for the lock key and record its remaining TTL and stored value (owner identifier and fencing token if present).
3. 03Look for host-side signals of why the owner disappeared: OOM killer messages, kernel panic, container restart count, or node loss events.
4. 04List the last renewal attempts emitted by the owner and note the gap between the last successful renewal and the disappearance.
5. 05Check whether a second contender has already taken over the lock, and if so, record the fencing token it received.
6. 06Decide whether the scenario is "owner crashed" or "owner partitioned"; these have different proofs and different SLOs for safe transfer.
7. 07Freeze the captured evidence (logs, TTL snapshots, fencing tokens, timestamps) before any retry, so the regression test can replay them deterministically.

Evidence to collect

• •Adopt short-lived TTLs with monotonic fencing tokens on every lock acquire, and reject acquires that do not return a token consumable by the protected resource.
• •Keep the lease-renewal path in a separate process or supervised sidecar from the user code, so a crash in user code still keeps renewal observable until the lease truly expires.
• •Monitor renewal health and alert when the gap between successful renewals exceeds a fraction of the lease TTL, before expiry actually happens.
• •Synchronize clocks between owner hosts and the coordinator host, and alarm on NTP steps large enough to invalidate the lease contract.
• •Run chaos drills that exercise crash, partition, and clock-skew disappearance modes on a fixed cadence, with the same assertions as the regression test.

Where to look

• •Coordinator service logs and keyspace dump for the lock key, including its TTL history if available.
• •Owner process logs, especially lease-renewal and unlock paths, plus any client wrapper that emits acquire/renew/release events.
• •Host kernel logs and container runtime events (OOM, SIGKILL, cgroup memory pressure, restart count, liveness probe failures).
• •Network plane logs between owner and coordinator: route flaps, NAT timeouts, firewall drops, or proxy idle disconnects that could mimic a partition.
• •Timekeeping layer: chrony or ntpd logs on both owner and coordinator hosts around the disappearance window.
• •Protected-resource access log to see whether stale-owner writes leaked through after the new owner took over.

Diagnostic steps

1. 01Classify the disappearance as crash, partition, or clock anomaly; each demands a different proof shape and a different injection in the regression test.
2. 02Establish the lease contract: confirm the lock has an explicit TTL and that every acquire is paired with a fencing token consumed by the protected resource.
3. 03Measure the time-to-safe-release: the gap from disappearance to the moment the coordinator considers the lock free, and assert it is bounded by the lease TTL plus a small clock-skew margin.
4. 04Measure the time-to-safe-transfer: the gap from disappearance to a new contender holding the lock with a strictly greater fencing token, and assert no contender acquired it before lease expiry.
5. 05Replay stale-owner writes against the protected resource after the transfer and verify they are rejected because their token is stale.
6. 06Inject failures at the renewal boundary, not only at idle: kill the owner mid-renewal and verify the test still observes correct release behaviour.
7. 07Repeat the scenario under simulated clock skew between owner and coordinator to confirm the lease contract degrades safely rather than silently.
8. 08Compare evidence against the regression assertion set; any missing assertion means the test does not actually prove recovery, only that the lock expired.

Common mistakes

• •Asserting that the lock "released" only because the key disappeared, without asserting that a new owner safely took over with a higher fencing token.
• •Testing only graceful shutdown or SIGTERM, which still allows the application to run unlock code; the regression must use abrupt termination that bypasses cleanup.
• •Confusing a long GC pause or stop-the-world event with a real crash; the lock TTL must be sized against both.
• •Hosting the lease-renewal thread inside the very process whose crash you are trying to survive, so renewal stops at the same instant as the user code.
• •Trusting wall-clock time on a single host when the coordinator and owner run on different machines; time proofs require both clocks.
• •Ignoring the protected resource: a lock that "releases" but still lets the stale owner write is not a recovered lock, it is a leaked lock.

Safe fixes

• •In the regression test, terminate the owner with SIGKILL (or an equivalent abrupt signal) so no application-side unlock code can run.
• •Add an explicit lease TTL to every acquire path and reject, in test, any acquire that returns a lock without a fencing token.
• •Size the lease TTL to exceed the worst observed renewal round-trip by a documented safety margin, and assert that margin in CI.
• •Run the protected resource behind a token check that monotonically compares the fencing token on every write, so stale owners fail closed.
• •Run the regression under simulated clock skew and partial network failure, not only under a clean kill, so partition-class disappearances are also proven.
• •Capture and compare owner-clock and coordinator-clock timestamps for every renewal and release, so clock-driven false releases are visible.

Prove the fix

1. 01Coordinator reports the lock key as absent, or with a strictly lower fencing token, after the lease TTL plus a bounded skew margin has elapsed since the disappearance.
2. 02A second contender acquires the same lock key and receives a fencing token that is strictly greater than the token held by the disappeared owner.
3. 03Operations issued by the original owner after its disappearance are rejected at the protected resource because their token is not the current highest.
4. 04No two distinct fencing tokens are observed writing to the protected resource concurrently during or after the recovery window.
5. 05Time-to-safe-release and time-to-safe-transfer both stay within the documented SLO across repeated runs of the regression test.
6. 06The regression assertion set covers crash, partition, and clock-skew disappearance modes, and each mode passes independently in CI.

Prevention and next steps

• •Keep the protected resource's write path responsible for rejecting stale fencing tokens, so lock recovery is enforced at the boundary that matters.
• •Expose lease-renewal lag, fencing-token rejections, and takeover latency as separate signals instead of treating key absence as proof of safety.
• •Make the lock TTL, maximum tolerated clock skew, and takeover window explicit configuration that the regression test reads rather than silently duplicating.
• •Run crash, partition, and clock-skew disappearance drills on a fixed cadence using the same assertions as the regression test.
• •Keep lease renewal outside the user-code process or supervise it independently, so a worker crash cannot make the renewal signal look healthy.

Safe commands and checks

ps -o pid,stat,etime,cmd -p <pid> — read-only process state inspection; obtain <pid> from the lock owner's application log or service registry.
redis-cli -h <redis_host> -p <redis_port> PTTL <lock_key> — read the remaining TTL on the lock key without mutating it; <redis_host>, <redis_port>, and <lock_key> come from the lock client configuration.
redis-cli -h <redis_host> -p <redis_port> GET <lock_key> — read the stored lock value (owner identifier and fencing token if encoded) without mutating state.
redis-cli -h <redis_host> -p <redis_port> TIME — read the coordinator's current time to compare against the owner host's clock when investigating clock-skew expiry.
redis-cli -h <redis_host> -p <redis_port> CLIENT LIST — list connected clients read-only to confirm the disappeared owner no longer holds a connection; restrict to a debug session, not steady-state production.
date -u +%FT%TZ — print the owner host's UTC time for pairing with the coordinator's TIME output above.
dmesg | grep -Ei 'oom|killed process' — read-only scan of kernel logs for evidence that the owner was OOM-killed rather than gracefully exiting.
chronyc tracking — read-only NTP status on the owner or coordinator host to assess clock offset and last sync, useful when lease expiry is suspected to be clock-driven.