Distributed-lock checklist

The symptoms

• •Two or more workers are observed executing the protected critical section concurrently, indicating the lock is being held by more than one party at the same time.
• •A worker that has crashed, been OOM-killed, or had its clock jumped is still listed as the lock holder past its TTL, blocking new acquisitions.
• •Lock auto-renewal continues to extend the lease even though the original acquirer's liveness signal (heartbeat, session, or process) has already failed.
• •Lock acquisition latency spikes correlate with GC pauses, network blips, or scheduler stalls rather than steady-state contention.
• •Side-effect operations guarded by the lock (duplicate writes, double-sends, idempotency violations) appear in application logs more than once per logical request.

Likely causes

• •Lock TTL is configured longer than the holder's worst-case stall (GC, I/O, scheduler starvation), so a paused holder can survive past its lease and race a successor.
• •Clock skew between lock server and clients causes early or late lease expiry, so renewal windows misalign with the server's monotonic clock.
• •Renewal runs on a separate thread or sidecar that outlives the protected worker's liveness, so the lease is extended after the owner is already dead.
• •The fencing token or unique-owner identifier is not checked by the protected resource, allowing a stale holder to write even after the lock has been reassigned.
• •Network partition isolates the holder from the lock server; on partition heal, the holder reasserts ownership while another acquirer has already taken over.

First ten minutes

1. 01Confirm the failure mode class: is this a stale-holder issue (old owner still acting), a contested issue (two owners at once), or a renewal-after-lost-liveness issue (lease extended on a dead process)?
2. 02Capture the lock server's current view of holder identity, lease TTL, and last renewal timestamp for the contested key; record these before any restart.
3. 03Identify every process, sidecar, or cron job that can call the lock acquire or renew API for the affected key, and note their liveness signals.
4. 04Cross-check the holder's last application-level heartbeat against the lock server's last renewal time to see whether they diverge.
5. 05Decide whether to keep the system running with degraded safety, or to fail closed; do not assume a restart will resolve a fencing-token gap.

Evidence to collect

• •Lock-key name, lock server identifier, and the exact client library version observed performing the acquire and renew calls.
• •Holder identity (UUID, owner token, or fencing token) as recorded by the lock server at the time of the incident, plus the lease TTL and last renewal timestamp.
• •Application-level liveness signals: process uptime, last heartbeat, GC log timestamps, and scheduler queue depth for the holder.
• •Lock server clock monotonicity and skew versus the holder's clock at the moment of suspected expiry, taken from server-side metrics, not client claims.
• •Correlation between duplicate side-effects in downstream stores and the timestamps of contested lock acquisitions.

Where to look

• •At the lock-store boundary: the lock server's keyspace, its TTL/lease metadata, and its audit or slowlog for the contested key.
• •At the client boundary: the lock client's internal acquisition state machine, including its renewal goroutine/thread and its owner-token generation logic.
• •At the process boundary: GC logs, container OOM events, cgroup throttling, and scheduler wakeup traces for any process that holds or renews the lock.
• •At the network boundary: packet captures or flow records between holder and lock server spanning the suspected partition window, looking for reconnect storms.
• •At the protected-resource boundary: downstream writes or RPCs that should be guarded by the fencing token, and whether they accept stale owners.

Diagnostic steps

1. 01From the lock server, read the current value and PTTL for the lock key; if PTTL is greater than zero but the recorded owner differs from the live worker list, the holder is stale.
2. 02Compare the lock server's last renewal timestamp with the holder process's last emitted heartbeat; a renewal timestamp that advances while heartbeats are frozen proves renewal-after-lost-liveness.
3. 03Replay the acquisition timeline by correlating lock-server audit entries with client logs; two acquire successes for the same key inside one lease window prove a contested lock.
4. 04Inspect the renewal code path to determine whether renewal runs in the same process as the protected work or in a separate supervisor; separation is the precondition for the renewal-after-liveness bug.
5. 05Check whether the protected resource rejects requests without a monotonically increasing fencing token; absence of this check is what turns a stale holder into a correctness incident.
6. 06Measure clock skew between holder and lock server using the lock server's own clock metrics; do not trust client-reported wall-clock times for lease reasoning.

Common mistakes

• •Restarting the holder process without checking whether a fencing token is enforced at the resource; the stale holder may already have written and a restart only masks the symptom.
• •Increasing the lock TTL "to be safe," which widens the window in which a paused holder can race a successor and corrupts the lease invariant.
• •Treating lock acquisition latency as a network problem when it is actually a renewal-thread stall caused by GC or scheduler pressure on the holder.
• •Relying on wall-clock comparisons between client and server logs; the lock server's monotonic clock is the only trustworthy reference for lease expiry.
• •Assuming a single lock server is the source of truth without verifying replication, failover, or split-brain behavior that can produce two valid holders simultaneously.

Safe fixes

• •If the protected resource supports fencing tokens, gate every guarded write on a strictly increasing token and reject writes with a token lower than the last accepted value; this is a containment fix, not a prevention.
• •If renewal runs in a separate supervisor, move the renewal goroutine into the same process as the protected work and couple its liveness to a heartbeat that the protected code itself emits; this closes the renewal-after-lost-liveness gap.
• •Shorten the lease TTL to a value bounded by the holder's worst-case observed stall (GC pause, I/O hang), and add jitter so that retries do not synchronize; confirm the bound against GC logs before changing it.
• •If you must release a stuck lock manually, do so only after verifying that no live holder is still emitting heartbeats, and record the previous owner token so the release can be audited; never delete the key on suspicion alone.
• •Where the official Redis cache-aside guidance applies to the same datastore, separate lock semantics from cache semantics so that an evicted or expired cache value cannot be mistaken for a lock release.

Prove the fix

1. 01Inject a controlled pause (sleep or SIGSTOP) into the holder process longer than the lease TTL, then attempt acquisition from a second worker; proof is that the second worker acquires and proceeds only after the lease has expired server-side.
2. 02Kill the holder process and observe the lock server's view of the holder identity; proof is that no renewal timestamp advances after the kill, and that the lease expires within the configured TTL plus bounded clock skew.
3. 03Force a network partition between holder and lock server, then heal it; proof is that the holder does not reassert ownership against an already-reassigned lock, and that the protected resource rejects any guarded write carrying the old fencing token.
4. 04Replay the duplicate-side-effect scenario from the incident timeline; proof is that the downstream store records exactly one effect per logical request after the fix, and that a higher fencing token is required for the second attempt to be accepted.
5. 05Run a chaos drill that pauses the renewal goroutine independently of the protected work; proof is that the lease is not extended and a successor acquires cleanly, demonstrating that renewal and protected work share a liveness signal.

Prevention and next steps

• •Design locks so that the renewal thread and the protected work share a single liveness signal; renewal must fail fast when the protected work has stopped making progress.
• •Bound the lease TTL against the worst-case stall you have actually observed in GC logs, scheduler traces, and I/O timeouts, and revisit the bound whenever those measurements change.
• •Require monotonic fencing tokens at every guarded downstream resource, and treat any code path that bypasses the token check as a correctness regression in code review.
• •Separate lock state from cache state when both are stored in the same datastore, so that cache eviction, TTL expiry, or cache-aside invalidation cannot be confused with lock release.
• •Periodically rehearse stale-holder and partitioned-holder scenarios in a staging environment so that operators have runbooks that are already validated rather than improvised during an incident.

Safe commands and checks

Read the current lock key value and remaining TTL without modifying state (placeholders required): redis-cli -h <lock_host> -p <port> GET <lock_key> ; redis-cli -h <lock_host> -p <port> PTTL <lock_key> — interpret PTTL <= 0 as the lease already expired server-side, and a non-zero PTTL with an unexpected owner as a stale-holder signal.
Inspect the lock server's audit or slowlog for the contested key (placeholder required): redis-cli -h <lock_host> -p <port> SLOWLOG GET 128 — filter entries to the lock key and confirm whether multiple acquire successes occurred inside one lease window.
Capture the holder process liveness signal before changing anything (placeholder required): ps -o pid,etime,stat,comm -p <pid> — pair this with the lock server's last renewal timestamp to detect renewal-after-lost-liveness.
Measure holder-side GC pause exposure (placeholder required): jstat -gc <pid> 1s 5 — record the maximum GC pause and compare it against the configured lease TTL to see whether TTL is longer than worst-case stall.
Check container-level throttling that can stall renewal independently of the application (placeholder required): cat /sys/fs/cgroup/system.slice/<service>.slice/memory.pressure 2>/dev/null | head — elevated pressure indicates the holder is being throttled even though its heartbeat still fires.
List active holders across replicas by reading each replica's lock view (placeholder required): redis-cli -h <replica_host> -p <port> GET <lock_key> — divergence between replicas indicates a replication or failover window where two holders can exist.
Verify clock monotonicity at the lock server (placeholder required): chronyc tracking -h <lock_host> 2>/dev/null — record offset and jitter; only the server-side clock is authoritative for lease reasoning.