Incident Dossier

The notification backlog that never drains

An application's short-lived notification queue accumulates work during a provider interruption. The provider later recovers, yet notifications beyond their useful window remain active. Follow the queue, worker and fake provider to restore bounded retention without breaking useful retries. This is original synthetic training code, not Microsoft's implementation or an explanation of a real outage.

Initial failing signalStale application notifications remain active beyond their useful windowExpected behavior: Stale local jobs expire without a provider attempt, while a fresh job can use the recovered provider. · actual behavior: Old jobs are still deferred and consume the first recovery slot before fresh work.

Codebase incident

Evidence desk

Outage Resilience Simulations · 2

SIMULATED INCIDENT. A status-notification application retains each queued message only while it is useful. Its downstream fake provider returns temporary failures, then recovers. The replay shows old work persisting beyond the intended retention window and winning the next provider slot. Examine queue state across polling turns and keep permanent rejections, useful retries and acknowledgment semantics intact. All data and timings are invented. This is not a reconstruction of Microsoft's systems or the cause of any real outage.

Customer symptom

Fresh application notifications wait behind stale local work after a simulated provider interruption.

Expected

Pending notifications age out locally at their usefulness deadline. Useful work can retry, and provider acceptance remains separate from delivery.

Actual

After several failed polling turns, stale notifications remain eligible and fresh work waits for a provider slot.

Architecture

Invented notification fixtureApplication queuePolling workerLocal fake providerManual clock

fixture → queue · enqueue logical jobs
worker → queue · read due jobs and record outcomes
worker → provider · bounded attempts
clock → worker · deterministic elapsed time

Logs2

Simulated queue timeline

elapsed=0 old-alert enqueued; usefulness=2500ms elapsed=0,1000,2000 provider temporarily unavailable elapsed=3000 old-alert still deferred elapsed=4000 provider recovers; fresh-alert joins the local queue

fixtures/outage.mjs (invented)

Acknowledgment boundary

The fake provider records accepted requests. No delivery callback exists; deliveredAt remains null. These are local fixture events, not observations from a real mail service.

src/fake-provider.mjs

Recent changes

Retry scheduling was consolidated into queue state transitions.
The worker now has an explicit provider-call budget per polling turn.
The replay fixture includes an interruption longer than an old notification's useful window.

Plausible hypotheses

Queue timing changes across retries — The long interruption behaves differently from the brief one, despite identical notification inputs.
Polling budget is too small — Only a limited number of jobs may contact the recovered provider in one turn.
Provider acknowledgment implies delivery — The fake reports accepted requests, but the application has no delivery callback.

Progress

1Reproduce the bugRun checks

2Edit the fixOpen

3Pass checksOpen

4Submit fixOpen

Reproduction Steps

Run npm test, then npm run replay to inspect the deterministic incident trace.
Follow one logical notification through the queue and worker during a brief and a long provider interruption.
Compare its timing fields and terminal states across polling turns.
Make a narrow production-code change and prove stale work expires while useful retries still succeed.

Focus Mode

Start with the replay trace, then follow queued notification state through one worker polling turn.

Start

README.md
tests/incident.test.mjs
src/replay.mjs

Likely area

src/queue.mjs
src/worker.mjs
src/fake-provider.mjs

Investigation path

1Reproduce the stale-backlog symptom with the manual clock.
2Trace one logical job through deferral, eligibility and expiry.
3Separate local queue state, provider acceptance and actual delivery.
4Preserve useful retries and show that expired work does not block fresh work.

Repo scale

9 public files
Queue, polling worker, fake provider, manual clock, replay fixture and visible tests
One focused queue-state fix

Challenge Coach

Hints reveal broad direction first, then narrow toward the suspect path without showing hidden tests or exact solution code.

TopicQueues & Reliability

DifficultyMedium

Est. time25 min

Workspacepreparing

Explorer

Loading files

Evidence Map

Failing signal loaded from challenge API
Workspace files loaded from workspace API
Results rendered only after API run/submit

Autosaved

workspace

Outputread-onlyready$ prepare workspace

Preparing workspace