Metrics stay flat during an outage: find the instrumented path that disappeared

The symptoms

• •User-reported errors or a status page alert is firing, but latency, error-rate, and request-rate panels remain at the same horizontal line as a healthy day.
• •One specific endpoint, route, service, or queue stops appearing in dashboards while others keep reporting normally; the missing path corresponds to where users feel the impact.
• •Logs and traces from the same path are missing or incomplete, even though the application is still running and CPU, memory, and network utilization on the host look healthy.
• •Alerts based on rate-of-change (for example, error-rate delta over the last 5 minutes) do not fire because the underlying series simply stops emitting points rather than emitting zero values.

Likely causes

• •The failing code path throws before reaching the instrumented wrapper, so the metrics layer never gets a chance to record the request or its outcome.
• •An OpenTelemetry or vendor SDK was disabled, removed, or never added to a refactored handler, and the new code path emits no telemetry at all.
• •An exporter, collector, or agent has been stopped, is out of capacity, or is filtering by a label that the failing requests do not carry, dropping points before they reach storage.
• •Label cardinality controls or sampling rules are silently dropping the specific labels (route, status code, customer tier) that the dashboard relies on, so the aggregated series becomes a single constant line.

First ten minutes

1. 01Confirm the outage is real and where it is felt: correlate user reports, synthetic checks, and dependency probes against the specific route, service, or queue suspected to be failing.
2. 02Open the metric explorer and inspect the raw series for the suspected path: confirm the data point stream stops at a specific timestamp rather than emitting constant or zero values, and note whether all labels or only one label group vanished.
3. 03Check logs and traces for the same path and time window: determine whether the gap is a logging gap, a tracing gap, or a metric gap, because each points to a different boundary.
4. 04Inspect the collector, agent, or sidecar responsible for the affected service for queue depth, dropped-point counters, and recent configuration changes applied to the export pipeline.

Evidence to collect

• •The exact timestamp at which the affected metric series stopped receiving new data points, captured both from the dashboard and from the storage backend's ingestion view.
• •The deployment, configuration, or autoscaling events for the affected service that occurred in the minutes before the series went flat, including SDK init logs and feature-flag changes.
• •Counter values for points received, points dropped, and export errors from the OpenTelemetry Collector or vendor agent serving the affected service.
• •Trace and log evidence for requests to the affected path, distinguishing between "no data emitted" and "data emitted but not searchable by the labels you tried."

Where to look

• •The application runtime boundary: SDK initialization, auto-instrumentation attach state, and any code path added between request entry and the metrics wrapper.
• •The process-to-collector boundary: the OTLP, statsd, or vendor protocol endpoint, the local socket or port used by the agent, and the exporter's retry and queue configuration.
• •The collector's processing boundary: processors, filters, and samplers that can drop data based on attribute key, attribute value, or span outcome before data reaches storage.
• •The storage and query boundary: label cardinality limits, retention rules, and dashboard queries that may aggregate over labels that no longer exist on incoming data.

Diagnostic steps

1. 01For each affected metric, compare the cardinality and label set of the last received data point against a known-good data point from before the incident; if the label set shrank to a single value, a filter or label mapper is likely responsible.
2. 02Reproduce the failing request path against a canary or staging instance with the same SDK configuration and verify whether metrics, logs, and traces are emitted for that instance; absence there confirms an application-side issue.
3. 03Diff the current collector or agent configuration against the last known-good version, focusing on processors, exporters, and feature gates that affect drop, sample, or transform behavior.
4. 04Trace a single test request end-to-end through the pipeline and confirm whether a payload reaches the collector, whether the collector accepts it, and whether storage ingests it; each boundary that the payload does not cross is the boundary that is broken.
5. 05Verify dashboard queries against raw query results; if the raw results contain points the dashboard does not show, the failure is in the visualization or aggregation layer rather than in telemetry collection.

Common mistakes

• •Concluding that "the service is healthy" because CPU and memory are normal, when the actual failure is a missing instrumented wrapper on the code path that is throwing.
• •Adding a new alert for "no data" without first confirming whether the gap is a collection gap or a query gap, which leads to alert storms on routine dashboard misconfiguration.
• •Restarting the collector or agent before capturing the dropped-point counters and recent configuration, which destroys the evidence needed to identify the boundary at fault.
• •Assuming an outage in the metrics backend is responsible without checking that logs and traces for the same path are also missing, which would shift the boundary toward the application.

Safe fixes

• •If the SDK was removed or disabled by a deploy, roll back to the last build with the instrumentation wrapper and re-verify that the metric series resumes within one collection interval.
• •If the collector's processor or filter is dropping on a missing label, add a default-value rule so the exporter always has the required attribute, then confirm the dropped-points counter stops increasing.
• •If the exporter queue is saturated, increase the queue size or batch timeout in a controlled change and verify that the points-dropped counter returns to its baseline while ingest rate recovers.
• •If label cardinality limits are silently aggregating the series, raise the limit only after confirming the new label dimension is expected, and re-verify the dashboard shows the previously hidden breakdown.

Prove the fix

1. 01The previously flat series resumes emitting new data points at the expected interval within one collection cycle after the fix is applied, and the label set matches the pre-incident shape.
2. 02The points-dropped and export-error counters for the affected pipeline return to their pre-incident baseline and remain there under load.
3. 03A deliberately failing synthetic request to the recovered path produces a non-zero error-rate data point and a corresponding log and trace entry, proving that the instrumented wrapper is now reached.
4. 04A rate-of-change alert that did not fire during the outage now fires on a replayed failure, demonstrating that the series is once again a true signal rather than a constant line.

Prevention and next steps

• •Add a deployment check that verifies the metrics SDK is initialized and a known canary metric is emitted before the new build is declared healthy.
• •Track points-dropped and export-error counters as first-class metrics with their own alerts so a silent exporter or filter problem cannot hide behind a flat dashboard.
• •Review collector and agent configuration changes through the same change-management pipeline as application code, so an accidental filter or sampler change cannot ship without review.
• •Maintain a runbook mapping each metric series to its SDK call site and its collector processor chain, so a flat-line alert has an immediate list of boundaries to inspect.

Safe commands and checks

grep -RIn "meter\\.|counter\\.|histogram\\." <service-source-dir> | head -n 50
git -C <repo-path> log --since=<incident-start-iso> --until=<incident-end-iso> -- <telemetry-config-path>
grep -E "points_dropped|export_failed|queue_full" <collector-log-path> | tail -n 200
diff -u <known-good-config-path> <current-config-path>
kubectl -n <namespace> logs deploy/<collector-deploy-name> --since=<incident-start-iso> --tail=<line-count>
journalctl -u <collector-unit-name> --since=<incident-start-iso> --until=<incident-end-iso> --no-pager