How to prove workers acknowledge after durable commit

The symptoms

• •A job moves to a "completed" state in dashboards but later reappears after a worker restart, indicating acknowledgement preceded durable commit.
• •Job counters show zero failures yet throughput is lost across restart cycles, consistent with silent re-enqueue rather than double execution.
• •Stalled-job events fire for workers that reported success moments earlier, showing the ack and commit ordering is reversed.
• •Logs show "moved to completed" without a paired "durable fsync" or equivalent commit marker before the network response.
• •Replay runs after a crash produce duplicate side effects, suggesting the worker treated acknowledge as equivalent to commit completion.

Likely causes

• •Worker calls broker acknowledgement before the local job state store returns a durable commit confirmation.
• •Manual acknowledgement mode is used together with auto-commit semantics, collapsing two distinct steps into one.
• •Lock refresh and commit happen on separate timers, allowing a commit window to be closed by an ack-only heartbeat.
• •Idempotency is assumed by the client library but not enforced at the commit boundary, so retries re-run side effects.
• •Misinterpreted stalled-job telemetry is treated as cosmetic instead of a hard signal that the lock was lost before commit.

First ten minutes

1. 01Confirm the exact failure mode stated in the contract: a crash before commit must leave work recoverable, not falsely completed, and capture one job id that demonstrates the boundary.
2. 02Identify which acknowledgement mode the worker is using and whether it is paired with an explicit, observable commit step on the same code path.
3. 03Map the worker handler into three phases: pre-commit work, durable commit call, and broker acknowledgement, and locate where each event is logged or emitted.
4. 04Inspect the stalled-job configuration and treat any stall event as evidence the lock was lost relative to the commit, not as a transient warning.
5. 05Decide whether the question is about the contract (ack only after commit) or about a specific incident; the verification steps differ and must be chosen explicitly.

Evidence to collect

• •Worker-side log lines that mark each of the three phases in order: handler start, commit confirmation, and broker acknowledgement, correlated to a single job id.
• •Broker-side records of the job lifecycle states, including any stalled-job events and their timestamps relative to the worker's commit log.
• •The stall interval and lock duration values configured for the queue, since they bound how long the broker will wait before declaring a stall.
• •Side-effect records downstream of the worker, such as database write confirmations, that can prove whether the durable commit actually completed.
• •A reproducible crash scenario where the worker is terminated between handler start and acknowledgement, used to observe re-enqueue behavior.

Where to look

• •The worker handler code path at the boundary between the local job store commit call and the broker acknowledgement call, since this is the only place the ordering can be enforced.
• •The queue's stalled-job settings and the stalled-job telemetry stream, because they define the maximum window between commit and ack that is considered safe.
• •The broker's job lifecycle log or events for the specific job id under test, where state transitions are recorded independently of worker logs.
• •Any wrapper or framework layer above the queue client that issues acknowledge on behalf of the worker, since it can obscure the true ordering.
• •The downstream system that receives the worker's durable write, because its commit confirmation is the only objective signal that commit succeeded.

Diagnostic steps

1. 01Verify that the worker emits a commit-success event strictly before the broker acknowledgement is sent, by reading the handler source and confirming no early-return paths skip the commit.
2. 02Compare the broker's job lifecycle timestamps to the worker's commit timestamp for the same job id; the commit must precede the completed transition under normal operation.
3. 03Reproduce a crash between commit and ack by terminating the worker after the commit log line but before the ack log line, then observe whether the job is re-enqueued rather than marked completed.
4. 04Inspect stalled-job events for the queue and correlate them with crash windows; stalls during the pre-ack window indicate the ack did not follow a durable commit in time.
5. 05Confirm that the worker's acknowledgement mode and the queue's stalled-job semantics are consistent: manual ack must be paired with a deliberate commit step, and auto-commit must not bypass the durable write.

Common mistakes

• •Treating a stalled-job warning as cosmetic rather than as evidence the lock was lost relative to the commit, which can mask ack-before-commit bugs.
• •Assuming the queue client library enforces durable commit before acknowledgement on the developer's behalf, without reading the boundary code path.
• •Relying solely on "no errors" in the worker log as proof of correctness, instead of correlating commit and ack events for the same job id.
• •Configuring aggressive stall intervals that mask ordering bugs by re-enqueueing quickly enough that duplicates are mistaken for transient retries.
• •Conflating acknowledgement with completion in dashboards, which hides whether the durable commit actually persisted before the state change.

Safe fixes

• •If evidence shows ack is sent before commit, restructure the handler so the broker acknowledgement call appears strictly after the durable commit confirmation returns successfully, and add a paired log line for each phase.
• •If stalled-job events are being ignored, raise them to a hard signal in alerting and require an explicit review of the lock-vs-commit timing for any non-zero stall count during a verification window.
• •If manual acknowledgement is used without an explicit commit step, introduce a commit helper that returns a typed success value and gate the acknowledge call on that value, so the ordering is enforced in code, not in convention.
• •If the worker runs in a framework that auto-acknowledges, disable auto-ack for jobs that perform durable writes and document which job classes require explicit commit-before-ack handling.
• •Only after each fix, re-run the reproducible crash test and confirm the job is re-enqueued rather than marked completed, using the same job id tracing described in proofOfFix.

Prove the fix

1. 01Run a controlled worker crash between the commit log line and the acknowledgement log line for a dedicated test job id, then verify the broker re-enqueues the job instead of marking it completed.
2. 02Capture the per-job timeline showing commit timestamp strictly before completed-state timestamp for a statistically meaningful sample of normal runs, with no stalls during the pre-ack window.
3. 03Confirm that for every stalled-job event in the verification window, the corresponding job id has no downstream side-effect duplication, demonstrating that re-enqueue happened before any durable write was acknowledged.
4. 04Add a regression check that fails the build if a new code path introduces an acknowledge call ahead of the durable commit call, enforced at the handler boundary rather than by convention.
5. 05Record the before-and-after stalled-job counts and the before-and-after re-enqueue-after-crash outcomes for the same job class, so the contract is observably restored and not merely asserted.

Prevention and next steps

• •Encode the commit-before-ack contract as a typed return value or interface boundary in the worker handler, so the ordering cannot be reordered silently during refactors.
• •Treat stalled-job telemetry as a first-class signal in monitoring, with thresholds tied to the commit window rather than to throughput, so ordering regressions surface before data loss.
• •Maintain a documented mapping between job classes and their durable commit mechanism, reviewed whenever the queue client or stalled-job settings change.

Safe commands and checks

grep -nE "ack|acknowledg|commit|moveToCompleted|stalled" <worker_handler_source_file> | sort
awk -F',' '$1 ~ /<job_id>/ {print}' <broker_job_lifecycle_log>
date -u +%Y-%m-%dT%H:%M:%SZ # capture UTC timestamp at crash injection point
kill -TERM <worker_pid> # terminate worker between commit and ack phases; obtain <worker_pid> from the process supervisor
redis-cli -u <redis_url_placeholder> LLEN bull:<queue_name>:wait # inspect wait list length using a placeholder URL, not a loopback address