The pattern
The code that exists to explain failures becomes the failure: a logger that throws without request context, metric labels with unbounded cardinality, a serializer that leaks secrets into error logs. Telemetry runs on every request, so a bug in it has a one hundred percent trigger rate.
( 01 )Symptoms
How this failure announces itself.
- warningCrashes inside logging or metrics code, often only for background jobs that lack request context.
- warningThe metrics backend degrades as a label (user id, URL, error message) explodes in cardinality.
- warningSecrets or auth headers appear in error logs after a serializer change.
( 02 )First moves
The first ten minutes — establish facts before touching code.
- 1Read the crash stack: if telemetry frames are on top, treat telemetry as the suspect, not the code it was observing.
- 2Reproduce the telemetry call with the empty or background context that triggered it.
- 3For metrics pressure, query series counts per metric and find the label whose value count grows with traffic.
- 4For leaks, search recent logs for known secret shapes and identify the serialization path that emitted them.
( 03 )Where to look
The code and config that usually owns this bug.
- searchLogger context builders - what do they assume exists (request, user, trace) that jobs and startup code do not have?
- searchMetrics label construction - any label fed from user input or unbounded ids is a cardinality bomb.
- searchError serializers - which fields get included when an unexpected object is thrown?
- searchMiddleware ordering - telemetry that runs before context is established sees a different world than the handler.
( 04 )Common fixes
Fix the cause, then make the regression impossible.
- buildMake telemetry non-fatal: wrap emission so its failures are counted, never thrown.
- buildBound metric labels to closed sets (status class, route template) - never raw ids or messages.
- buildRedact at the serialization boundary so every path through it is covered, not just the known ones.
- buildKeep request ids in logs and traces, not in metric labels.
( 05 )Prove the fix
A fix you can't demonstrate is a guess. Close the loop.
- verifiedRun the previously crashing path (background job, startup) and confirm telemetry degrades gracefully.
- verifiedSeries counts stay bounded under a traffic replay.
- verifiedA canary secret placed in a request never appears in any log sink.