Background worker stall debugging checklist

The symptoms

• •Worker logs show the job starting or its handler being invoked, but no completion, failure, or retry event is ever emitted for that job id.
• •Queue metrics or dashboards show an "active" or "locked" job count that does not decrease over multiple lock-duration windows.
• •Newly enqueued jobs of the same name queue up behind a stalled job, indicating the worker pool is blocked on a single long-held lock.
• •Repeated "stalled job" recovery events for the same job id, with the job re-locked shortly after each recovery without producing a result.
• •Job is reported as stalled or moved to a "delayed" state by the lock monitor, yet the original worker process or container appears still running.

Likely causes

• •Worker process is unresponsive (event loop blocked, GC pause, or deadlock) and is not calling the lock extension mechanism before the lock expires.
• •Network partition or DNS resolution failure between the worker and the Redis broker prevents lock-renewal heartbeat traffic from arriving in time.
• •Job handler is performing a synchronous, blocking call against an external dependency with no timeout, so the lock TTL elapses before the handler returns.
• •Worker was killed, OOM-terminated, or its container restarted mid-handler, leaving the lock orphaned in Redis until the stalled-job watcher moves it.
• •Misconfigured lock duration relative to the job's expected runtime, causing legitimate long jobs to be repeatedly flagged as stalled even though the worker is healthy.

First ten minutes

1. 01Identify the exact job id and queue name that is stuck, and record the timestamp it entered the active state so later lock-window math has an anchor.
2. 02Confirm whether the assigned worker process is still alive (process exists, responsive to SIGUSR1 or equivalent, not in D state), and note the host or pod where it runs.
3. 03Check whether the queue's stalled-job watcher has already emitted a "stalled" event for this job id, which determines whether you are pre-detection or post-recovery.
4. 04Compare the configured lock duration against the job's age in the active state to decide if the TTL itself is the proximate cause versus a worker-side fault.
5. 05Verify connectivity between the worker host and the Redis broker using latency and packet-loss signals before assuming the worker is at fault.

Evidence to collect

• •Job id, queue name, job name, attempts count, and the timestamps for the most recent "active" and any "stalled" events from the queue's event stream.
• •Worker host, process or container id, CPU and memory snapshot, and whether the process is consuming CPU or blocked on I/O at the moment of inspection.
• •Configured lock duration value and the actual elapsed time since the worker last reported progress on the lock.
• •Redis broker SLOWLOG entries and INFO commandstats for the worker's connection around the time the stall was first observed.
• •Application logs from the worker covering a window of at least one lock-duration before the stall, filtered to the specific job id.

Where to look

• •The boundary between the worker process and the Redis broker, where the lock-extension heartbeat must cross before each TTL expires.
• •The job handler's outbound calls to external services, where blocking I/O or unbounded retries can hold a lock past its TTL.
• •The stalled-job watcher or "lock monitor" component, whose events tell you whether the framework has already declared the job stalled.
• •The worker host's process and runtime layer (event loop lag, GC pause log, container cgroup limits), which can starve the heartbeat.
• •The event stream for the queue, which is the authoritative record of state transitions versus the worker process's own logs.

Diagnostic steps

1. 01If a "stalled" event already exists for the job id, the framework has detected the lock was not renewed; the question becomes why the worker missed its renewal deadline.
2. 02If no "stalled" event exists and elapsed time exceeds the lock duration, the watcher is not running or the lock-duration value is misread; verify both before blaming the worker.
3. 03Check the worker's event-loop lag or runtime pause metrics during the suspected stall window; a multi-second pause on a renewal interval shorter than the lock TTL is sufficient to cause a stall.
4. 04Inspect outbound calls inside the handler for missing or oversized timeouts; a single synchronous call without a deadline can outlast multiple lock windows.
5. 05Confirm Redis reachability from the worker host at the suspected stall time using broker-side connection logs and SLOWLOG, not only the worker's view.
6. 06Distinguish an orphaned lock (worker process gone) from a live-but-blocked worker (process exists but is not renewing the lock) by checking process state, not just presence.

Common mistakes

• •Restarting the worker before recording the job id, queue name, and last lock timestamp, which destroys the evidence needed to confirm the root cause.
• •Increasing the lock duration as a first reflex without measuring actual handler runtime, which masks the symptom and lets real stalls compound silently.
• •Assuming a healthy-looking worker process is renewing its lock, when in fact the event loop is starved and no Redis traffic is being sent.
• •Reading only the worker's own application log and missing the queue's event stream, which is the source of truth for state transitions.
• •Conflating a delayed retry after stall with a normal delay, and therefore missing that the same job id is being repeatedly recovered.

Safe fixes

• •If the worker process is unresponsive (event loop blocked, GC stuck, deadlock), capture a runtime diagnostic (process snapshot, heap dump, or stack trace) before restarting, so the stall is not silently repeated.
• •If the handler performs an unbounded external call, add or tighten an explicit timeout shorter than the lock duration, and verify with a test job that the timeout actually fires.
• •If the lock duration is shorter than legitimate handler runtime, raise the lock duration only after measuring a representative p99 handler latency, and re-check stall rate afterward.
• •If connectivity between the worker and Redis is the proximate cause, fix the network path (DNS, firewall, TLS) rather than retrying the worker, and confirm renewals resume before considering the incident closed.
• •After any fix, require an observable regression check: the same job name must complete within one lock window on a fresh worker, and no further "stalled" events should appear for that job id.

Prove the fix

1. 01Re-enqueue a representative job of the same name and observe a "completed" or "failed" event in the queue's event stream within one configured lock-duration window.
2. 02Confirm the stalled-job watcher emits no further "stalled" event for the original job id, and that attempts count is no longer incrementing for that id.
3. 03Verify the worker process records a lock-renewal heartbeat (or its equivalent progress signal) at an interval strictly less than the lock duration for the duration of the job.
4. 04Confirm queue metrics show the active-job count returning to zero within one lock window after the test job finishes.
5. 05Compare handler p99 latency against the configured lock duration on the post-fix run; lock duration should exceed observed runtime with margin.

Prevention and next steps

• •Size the lock duration above observed handler p99 latency with explicit margin, and review the value whenever handler dependencies change.
• •Enforce timeouts on every external call inside handlers, and reject any handler that can block longer than the lock duration without renewal.
• •Alert on repeated "stalled" events for the same job id and on active-job counts that exceed one lock window without transitioning.
• •Monitor worker event-loop lag or runtime pause duration, and alert before the pause can outlast a lock-renewal interval.
• •Track worker-to-broker latency and packet loss, since stalls caused by network partition recur if the path is not fixed at the infrastructure layer.

Safe commands and checks

redis-cli SLOWLOG GET 50 | head -n 50
redis-cli INFO clients | grep -E 'connected_clients|blocked_clients'
redis-cli CLIENT LIST | grep -E 'id=|addr=' | head -n 20
ps -o pid,stat,etime,cmd -p <pid>
cat /proc/<pid>/status | grep -E 'State|Threads|VmRSS'