The pool that never gives connections backData & ReliabilityHardPreparing

00:00

Incident Dossier

The pool that never gives connections back

The service has no database errors at first, but latency and 5xx responses climb after a burst of ordinary reads. Follow connection ownership through the API, repository, pool, and error path before changing pool size.

Initial failing signalRead latency becomes pool queue timeExpected behavior: Connections return to the pool after every repository operation. · actual behavior: Successful reads retain connections until the pool is exhausted.

Codebase incident

Evidence desk

Codebase Incidents · 3

At 16:18 UTC, the orders API begins returning 503s during a sustained but ordinary traffic window. A single request on a fresh process is fast and the database reports no failed queries. Pool metrics show every slot held after successful reads, so the investigation must follow lifecycle ownership rather than increase capacity first.

Customer symptom

Orders intermittently time out or return 503 while simple local requests still pass.

Expected

Each acquire has a matching release on both success and failure paths.

Actual

The repository returns early after a successful query and leaves the connection checked out.

Architecture

Orders APIOrder serviceOrder repositoryDatabase poolPool config

Orders API → Order service · request
Order service → Order repository · lookup
Order repository → Database pool · acquire/query
Order repository → Database pool · release
Database pool → Pool config · max connections

Logs2

Pool timeout

request=req-19 route=GET /orders/42 wait=1000ms error=pool_timeout request=req-20 route=GET /orders/43 wait=1000ms error=pool_timeout

orders-api

Repository timings

query_ms=3 transform_ms=1 result=ok release_event=missing

orders.repository

Metrics1

Connection lifecycle

max=2 acquired=2 idle=0 pending=14 query_errors=0 after=48 successful reads

db.pool

Recent changes

The pool max was reduced to protect the database.
The repository added a result transformation after the query.
Traffic shifted from batch reads to many small API reads.

Plausible hypotheses

Database is slow — Latency rises under load and database time is a common first suspect.
Repository leaks connections — Acquired slots remain high even though queries report success.
Traffic exceeds configured capacity — The incident starts during a sustained traffic window and could be a sizing problem.

Progress

1Reproduce the bugRun checks

2Edit the fixOpen

3Pass checksOpen

4Submit fixOpen

Reproduction Steps

Run the incident test on the fresh repository.
Compare pool acquired/idle counts before and after several successful reads.
Inspect the repository's success and error paths.
Add one lifecycle boundary and rerun visible and hidden checks.

Focus Mode

The pool reports a successful query; inspect what happens after the repository transforms that result.

Start

README.md
src/http/controller.js
src/orders/repository.js

Likely area

src/db/pool.js
src/orders/repository.js
src/orders/service.js

Investigation path

1Confirm one read is fast on a fresh pool.
2Measure acquired and idle slots after repeated reads.
3Trace release behavior on success and failure.
4Keep pool capacity unchanged and repair the ownership boundary.

Repo scale

15 meaningful files
API, service, repository, pool, query, metrics, config, and error context
One finally-boundary fix

Challenge Coach

Hints reveal broad direction first, then narrow toward the suspect path without showing hidden tests or exact solution code.

TopicData & Reliability

DifficultyHard

Est. time35 min

Workspacepreparing

Explorer

Loading files

Evidence Map

Failing signal loaded from challenge API
Workspace files loaded from workspace API
Results rendered only after API run/submit

Autosaved

workspace

Outputread-onlyready$ prepare workspace

Preparing workspace