Duplicate background jobs: trace where idempotency was lost

The symptoms

• •Same business key (order ID, user ID, file hash) appears in two or more worker invocations within a short window, even though the producer only published once.
• •Downstream side effects executed twice: duplicate emails, double-charged invoices, redundant database rows with identical payload, or two outbound API calls to a third party.
• •Worker logs show multiple "started" / "completed" / "ack" entries sharing the same correlation ID or job ID.
• •Queue redelivery counters or "approximate receive count" climb above 1 for messages that the worker reports as successful.
• •Idempotency table or cache shows zero hits despite repeated job executions, indicating the dedup check is being skipped or keyed incorrectly.

Likely causes

• •At-least-once delivery semantics from the queue: the visibility timeout expired before processing finished, so the broker redelivered the same message to another poll.
• •Producer retries without a deduplication key, where a network timeout on the enqueue caused the producer to send the same logical job twice.
• •Worker processed the message but the ack or visibility-delete call failed or was delayed, so the queue redelivered after the timeout.
• •Multiple worker instances or pods polled concurrently and both picked up the same in-flight message before visibility took effect.
• •Application-level retries inside the handler combined with broker-level retries, multiplying executions of the same side effect.

First ten minutes

1. 01Capture one concrete duplicate incident: message ID, correlation ID, business key, worker instance, and timestamps for each execution.
2. 02Search worker logs for that business key and count how many "started", "completed", and "failed" entries appear in the duplicate window.
3. 03Compare publish time, first execution time, and second execution time to determine whether the gap originated at the producer or at the broker.
4. 04Check the queue visibility timeout against the observed handler duration to see if redelivery is plausible.
5. 05Look for the idempotency token the handler is supposed to use and confirm whether it was generated, passed to the side effect, and actually checked.
6. 06Pause new publishes for the affected job type if downstream side effects are irreversible, while letting in-flight workers drain.

Evidence to collect

• •Queue message IDs and receipt handles for the duplicate window, including approximate receive counts.
• •Worker logs containing correlation ID, business key, attempt counter, and instance or pod name.
• •Idempotency store entries (DB row or cache key) with their TTL and write timestamps.
• •Downstream target logs or audit trail showing two writes for the same business key within seconds.
• •Visibility timeout, message retention, and concurrency settings for the affected queue.
• •Producer-side request logs, including outbound HTTP retries, SDK retry events, and timeout thresholds.

Where to look

• •Queue console or metrics: messages received, redelivered, deleted, and dead-letter counts.
• •Worker process logs: handler entry, attempt counter, exception traces, and ack or delete calls.
• •Idempotency layer: unique-constraint violations, "already processed" branches, cache misses, and TTL boundaries.
• •Producer service: outbound HTTP retries, SDK retry policies, timeouts, and any idempotency token attached to the publish call.
• •Downstream store: row counts grouped by business key, audit tables, and write timestamps for the duplicate window.

Diagnostic steps

1. 01Reproduce a single duplicate by walking one message ID through receive, start, first ack attempt, redelivery, and second start.
2. 02Determine the layer: if both executions come from different worker instances, suspect a visibility race; if both come from the same instance, suspect internal retry; if both come from different message IDs, suspect producer double-publish.
3. 03Verify the idempotency key strategy: is it derived from a deterministic business key, or is it random per attempt and therefore useless across redeliveries.
4. 04Inspect the handler boundary: does the handler perform the side effect before or after recording the idempotency claim, and where can a crash leave the system inconsistent.
5. 05Check the visibility timeout relative to the longest plausible handler runtime, including any external calls the handler makes before ack.
6. 06Trace whether dead-letter queues or poison-pill handling are creating new copies after a failure rather than just isolating them.

Common mistakes

• •Conflating "the queue is at-least-once" with "duplicates are the queue's fault only"; duplicates can be introduced at producer, broker, and consumer layers.
• •Using a random UUID as the idempotency key instead of a deterministic business key, so legitimate redeliveries bypass dedup.
• •Recording the idempotency claim after the side effect, so a crash between the two creates a window for re-execution.
• •Setting the visibility timeout shorter than the longest plausible handler runtime, especially when the handler calls slow external services.
• •Treating "no error in logs" as proof of single execution when the side effect actually ran twice in two different processes that both reported success.

Safe fixes

• •Introduce a deterministic idempotency key derived from the business key plus a stable job identifier, and check it inside the same transaction as the side effect.
• •Raise the queue visibility timeout to safely exceed the longest expected handler duration, including retries and any external calls it makes.
• •Reorder the handler so the dedup row is written before any irreversible external action, ideally as part of the same transactional unit.
• •Enable broker-native deduplication features where available, scoped to the producer and the affected job type, so the same logical publish cannot enqueue twice.
• •Add a "claimed" state machine so a second pickup of an in-flight job returns early with a logged skip branch instead of re-running side effects.

Prove the fix

1. 01Replay the original duplicate scenario against a test queue and confirm the handler runs the side effect exactly once, with a logged "duplicate, skipped" branch for the second attempt.
2. 02Show the idempotency store holds exactly one row per business key for the test set, regardless of redelivery count.
3. 03Confirm queue metrics still show redeliveries occurring but no additional downstream writes are produced for the same business key.
4. 04Inject a fault that kills the worker after the side effect but before ack, and verify no duplicate downstream record appears after redelivery.
5. 05Capture before-and-after counts of duplicate side-effect occurrences over a fixed publish volume and verify the ratio drops to the expected baseline.

Prevention and next steps

• •Make every job carry an idempotency key derived from inputs the producer has already committed to, and reject publishes that omit the key.
• •Treat "side effect first, ack last" as a structural rule enforced by code review and shared library helpers, not by individual handler authors.
• •Monitor the ratio of downstream side-effect writes to published messages and alert on drift, not only on absolute error counts.
• •Keep producer retry budgets conservative and bound them with a deduplication window so a single timeout cannot enqueue the same job many times.

Safe commands and checks

aws sqs get-queue-attributes --queue-url <queue-url> --attribute-names All --query "Attributes.{VT:VisibilityTimeout, Redrive:RedrivePolicy, Retention:MessageRetentionPeriod}" --output table
aws sqs receive-message --queue-url <queue-url> --max-number-of-messages 10 --visibility-timeout 30 --query "Messages[*].{Id:MessageId, Body:Body, Recv:Attributes.ApproximateReceiveCount}" --output table
grep -n "<business-key>" /var/log/worker/<worker-log-path> | grep -E "started|completed|skipped|ack|delete"
redis-cli -h <redis-host> -n <db-index> GET "idem:<business-key>"
Run the repository's read-only duplicate-job query against the configured test database connection, grouping recent rows by business key and checking for counts greater than one; do not place a password in the shell command.
kubectl -n <namespace> logs <pod-name> --since=30m | grep -E "job_id=<job-id>"
ps -o pid,etime,cmd -p <pid>
ss -ltn '( sport = :<worker-port> )'