How to verify lock lease renewal across a pause

The symptoms

• •Two workers are observed holding the same lease key at the same time in the lock store after one of them experienced a long pause (GC, stop-the-world, cgroup throttling, container freeze).
• •Monotonic fencing tokens stamped onto protected writes stop increasing linearly, and a downstream consumer rejects an older token with a "stale fence" error even though the lease TTL has not visually elapsed.
• •Cache repopulation under a cache-aside pattern writes a stale value concurrently with the original owner, because the original owner was paused long enough for the lease to expire but short enough that application logs still show it as "I am the lock holder."
• •Metrics show a healthy heartbeat gap-free interval on the owner, but the lock backend's TTL counters or PEXPIRE counters decrement and reach zero during the pause, after which a second owner successfully acquires the key.

Likely causes

• •Heartbeat cadence is configured shorter than the worst-case GC pause, cgroup freeze, or stop-the-world event, so the pause itself crosses the lease TTL boundary.
• •The "renewal" path is implemented as a script that catches all exceptions and retries silently, so a failed PEXPIRE during the pause leaves the lease to expire without surfacing to the owner.
• •The lock store and the owner use unsynchronized clocks, so a NTP step or container clock skew between PEXPIRE and the eventual expiry check causes the lease to be considered expired while the owner still believes it holds the lock.
• •Fencing tokens are not used, so two owners can both write through the same logical lock ID and the system only detects the conflict at the data layer rather than at the lock layer.
• •The lock client uses a long-lived TCP connection that was silently reset by an intermediate proxy during the pause, so the renewal write never reaches the lock store even though the owner logs "renewal OK."

First ten minutes

1. 01Confirm the failing system's name, the lock store type (Redis, etcd, ZooKeeper, Consul), and the lease key namespace so every later command targets the right instance and is scoped to read-only traffic.
2. 02Open the runbook entry for the lock client and record three numbers: lease TTL, heartbeat interval, and the documented "max tolerable pause" or "max clock drift" — these three numbers define the safety boundary.
3. 03Pull the lock client's recent logs scoped to the suspected owning instance and grep for the explicit renewal event name (often "renewed", "keepalive", "PEXPIRE", or "Heartbeat OK") and its failure counterpart.
4. 04Capture the lock store's view of the key: remaining time-to-live, owner value, and any associated fencing token, so you have a baseline before inducing any test pause.
5. 05Decide which side of the boundary you are debugging — the owner (cannot send heartbeats) or the store (cannot receive heartbeats) — by comparing owner-side renewal logs to store-side write timestamps for the same key.

Evidence to collect

• •Lock store TTL observation for the disputed key, captured at two timestamps that bracket the suspected pause, to prove whether the key actually expired while the owner was paused.
• •Owner-side timeline of renewal events for the same key, including any caught-and-retried errors, with monotonic timestamps and the originating thread or goroutine ID.
• •Process-level evidence of the pause itself: GC log duration, JVM safepoint pause, cgroup freezer entry/exit, container OOM kill, or stop-the-world indicator from the runtime.
• •Clock skew evidence: NTP offset between the owner host and the lock store host at the time of the disputed renewal, taken from the system journal or telemetry.
• •Protected-resource evidence: writes that succeeded under the disputed lock, with their fencing tokens, so you can show whether two distinct tokens were accepted or only one.

Where to look

• •At the lock client boundary: the scheduler or timer that emits heartbeats, and the try/catch or error-handling wrapper around the renewal call, because silent swallow is the most common place a missed renewal is hidden.
• •At the runtime boundary: GC logs, safepoint logs, goroutine scheduler dumps, and cgroup freezer or container runtime state, because the pause manifests at this layer even when the lock client looks healthy.
• •At the lock store boundary: the SET/PEXPIRE command log, the key's TTL, and any monotonic-clock or fencing-token field attached to the value, because the store is the source of truth for "who owns this right now."
• •At the network boundary: TCP reset counters, proxy idle timeouts, and connection-pool eviction events between the owner and the lock store, because a reset connection can drop a renewal without an application-level error.
• •At the protected resource boundary: the downstream consumer that enforces fencing tokens, because it is the only place a "two owners wrote" condition can be detected after the fact.

Diagnostic steps

1. 01Compute the safety margin: leaseTTL minus heartbeatInterval. If the observed pause duration is greater than this margin, the failure is expected by the configured parameters and the diagnosis is a configuration problem, not a code bug.
2. 02Compare owner-side renewal timestamps to lock-store-side PEXPIRE timestamps for the same key over the same window. If the owner logged a renewal but the store did not record a corresponding TTL bump, the boundary failure is between the two.
3. 03Reproduce the boundary under controlled conditions by pausing the owner process for a duration equal to the lease TTL, then attempt a protected write. If the write succeeds, liveness is not actually being proved; if it fails, the boundary is sound and the original incident had a different cause.
4. 04Inspect the renewal error handler by reading the code path or by raising the log level: a handler that logs "renewal failed" but does not invalidate the in-memory lock state is a candidate root cause for the symptom.
5. 05Check the fencing-token contract: if the protected resource rejects stale tokens, the lock layer is doing its job and the failure is at the application layer; if the protected resource accepts any token, the lock layer alone cannot prevent the duplicate-write symptom.
6. 06Verify clock monotonicity on both sides by inspecting the source of the timestamp used to evaluate expiry. If either side uses a wall-clock source that can step backwards, the renewal contract is unsound and the fix lives at the lock client, not the lock store.

Common mistakes

• •Concluding that the lock is "fine" because the owner logged a successful heartbeat, without verifying that the lock store actually applied a PEXPIRE that extended the TTL.
• •Adding a longer lease TTL as the first response, which only widens the window during which two owners can hold the lock and makes the incident harder to reproduce, not safer.
• •Ignoring fencing tokens because "we don't need them, the lock is enough," and then being unable to detect the duplicate-write condition at the data layer after it occurs.
• •Restarting the owner process to "clear the state" without first capturing the lock store's view of the key, which destroys the only source of truth about who currently holds the lease.
• •Treating the symptom as a network problem and chasing packet captures, when the actual failure is that the renewal thread was paused by the runtime and never attempted a network write at all.

Safe fixes

• •If the pause duration is bounded by the runtime (GC, safepoint) and you cannot reduce it, shorten the heartbeat interval so that leaseTTL minus heartbeatInterval strictly exceeds the worst observed pause, and add an explicit assertion at startup that loads the pause budget from configuration.
• •If the renewal error handler is silently swallowing failures, change it to (a) invalidate the in-memory lock state on the first failed renewal, (b) emit a structured log event with a distinct name, and (c) treat the lock as held only while the last renewal succeeded within leaseTTL.
• •If clocks are unsynchronized, switch the lock store's expiry to a monotonic source if the backend supports it, or stamp the renewal with a server-side timestamp returned by the lock store rather than a client-side timestamp.
• •If the connection is being silently reset, set an explicit socket-level keepalive and a connection-pool max-idle shorter than the heartbeat interval, and verify renewal events are observed end-to-end rather than only client-side.
• •If fencing tokens are not being enforced, add a monotonic fencing token to every protected write and verify the downstream consumer rejects tokens older than the highest it has seen, before relying on the lock layer for safety.

Prove the fix

1. 01Replay the original pause scenario against a canary instance, scoped to a non-production lock key, and observe in the lock store that the owner's TTL drops to zero and the key is reclaimed by a second owner before the original owner resumes any protected write.
2. 02Run a chaos test that pauses the owner for leaseTTL plus a small buffer, then assert that the protected write is rejected with a "lock not held" error and that the fencing token returned by the next acquisition is strictly greater than the one the paused owner last used.
3. 03Compare the lock store's TTL observation against the owner-side renewal log during the chaos run: for an honest fix, the lock store's TTL must reach zero during the pause even though the owner logs no successful renewal, and the owner must re-acquire (not silently retain) the lock after resume.
4. 04Add a synthetic monitoring check that asserts the divergence between "owner believes it holds the lock" and "lock store reports the key is held" stays below leaseTTL, and alerts when it crosses, so a regression is caught before it produces a duplicate write.

Prevention and next steps

• •Document the safety budget — leaseTTL, heartbeatInterval, worst-case pause, max clock drift — as a single configuration artifact, and reject startup configurations where leaseTTL minus heartbeatInterval is not strictly greater than the documented worst-case pause.
• •Require fencing tokens on every protected write that flows through a lock, and require the downstream consumer to reject stale tokens, so that a lock-layer failure is detected at the data layer rather than producing silent corruption.
• •Periodically run a pause-and-verify chaos test in a non-production environment that observes the lock store directly, and track the gap between owner-side "I hold the lock" and store-side "key is held" as a first-class metric.
• •Use a monotonic time source for any lease-expiry calculation on the lock client, and treat any wall-clock-based expiry as a configuration error at code review.

Safe commands and checks

redis-cli -h <host> -p <port> PTTL <lock-key> # read-only: returns remaining TTL in milliseconds; a negative value indicates the key does not exist or has no expiry.
redis-cli -h <host> -p <port> GET <lock-key> # read-only: returns the owner value and (by convention) the fencing token encoded in the value; do not parse until you have confirmed the encoding.
redis-cli -h <host> -p <port> CLIENT LIST # read-only: lists connected clients; useful for confirming whether the owner still has a live connection to the lock store during a suspect pause.
redis-cli -h <host> -p <port> SLOWLOG GET 10 # read-only: returns recent slow commands; a missing PEXPIRE for the lock key during the pause window is direct evidence of a dropped renewal.
redis-cli -h <host> -p <port> MONITOR # read-only but high-volume: streams commands in real time; only run against a non-production instance and only for a bounded window to capture a renewal attempt.