The job that two workers both completedQueues & ConcurrencyHardPreparing

00:00

Incident Dossier

The job that two workers both completed

A notification job is delivered twice during a visibility timeout window. Trace queue delivery, lock acquisition, idempotency, and persistence to find why two workers enter the side effect together.

Initial failing signalOne queue job emits two notificationsExpected behavior: Only one worker owns a job while its side effect is running. · actual behavior: Concurrent workers both pass the asynchronous lock check.

Codebase incident

Evidence desk

Codebase Incidents · 5

At 13:27 UTC, customers receive duplicate shipment notifications. The queue shows one message id with two delivery attempts, and both worker processes report success. The job lock exists and the persistence layer has an idempotency key, so trace the exact moment ownership is claimed before assuming the broker delivered two different jobs.

Customer symptom

A single queued job creates duplicate downstream notifications during retry or concurrency.

Expected

The lock claim is atomic before any asynchronous yield and the side effect is idempotent.

Actual

The lock checks state, yields, and sets ownership later, allowing a second worker through.

Architecture

Queue deliveryWorker processJob lockNotification serviceIdempotency storeVisibility config

Queue delivery → Worker process · at-least-once delivery
Worker process → Job lock · claim
Worker process → Notification service · side effect
Notification service → Idempotency store · dedupe key
Queue delivery → Visibility config · retry window

Logs2

Worker overlap

13:27:01 worker=blue job=job-7 lock=checking 13:27:01 worker=green job=job-7 lock=checking 13:27:01 worker=blue job=job-7 result=sent 13:27:01 worker=green job=job-7 result=sent

worker.job

Lock trace

job-lock check owner=none; await remote read; set owner=blue job-lock check owner=none; await remote read; set owner=green

job.lock

Metrics1

Delivery and side effects

queue message ids: 1; deliveries: 2; notification writes: 2; producer publishes: 1

queue.metrics

Recent changes

Worker concurrency was increased from one to two.
Queue visibility was shortened for faster retries.
A lock helper was made asynchronous for a remote-store migration.

Plausible hypotheses

Broker redelivered after visibility timeout — The duplicate occurs during retry and the job has at-least-once delivery semantics.
Lock claim is not atomic — Two workers can observe an available lock before either records ownership.
Producer published twice — Duplicate side effects can originate before the worker sees the message.

Progress

1Reproduce the bugRun checks

2Edit the fixOpen

3Pass checksOpen

4Submit fixOpen

Reproduction Steps

Run the concurrent worker test.
Trace the lock check and set operations around the asynchronous boundary.
Compare queue delivery count with producer publish count.
Make claim ownership atomic and retain idempotent side-effect protection.

Focus Mode

The important question is whether the lock is claimed before or after the first await.

Start

README.md
src/worker/consumer.js
src/locks/job-lock.js

Likely area

src/locks/job-lock.js
src/worker/consumer.js
src/idempotency/store.js

Investigation path

1Confirm two workers can enter the same job.
2Trace queue delivery and producer counts.
3Locate the asynchronous gap in lock ownership.
4Repair atomic claim and prove retry remains single-effect.

Repo scale

15 meaningful files
queue, worker, lock, idempotency, notification, persistence, metrics, and deployment context
One atomic-claim fix

Challenge Coach

Hints reveal broad direction first, then narrow toward the suspect path without showing hidden tests or exact solution code.

TopicQueues & Concurrency

DifficultyHard

Est. time35 min

Workspacepreparing

Explorer

Loading files

Evidence Map

Failing signal loaded from challenge API
Workspace files loaded from workspace API
Results rendered only after API run/submit

Autosaved

workspace

Outputread-onlyready$ prepare workspace

Preparing workspace