Observability-correlation checklist

The symptoms

• •Logs from a single user request do not share a common identifier with the corresponding trace spans, so the request cannot be reconstructed across signals.
• •Metric points emitted during a request carry labels that do not match the trace_id or request_id present in logs, so dashboards and traces cannot be cross-referenced.
• •Distributed trace spans across multiple services are broken at a hop, leaving orphan spans with no parent context and no downstream activity.
• •Log search by request_id returns hits from one service but misses identical requests handled by peer services in the same call graph.
• •Alerting fires on a metric, but the on-call engineer cannot locate the originating log lines or trace for the time window of the alert.

Likely causes

• •Propagation context (W3C Trace Context header or equivalent) is not being injected at the service ingress or is being stripped by an intermediate proxy or library.
• •Application code emits log lines or metric points with locally generated identifiers that are never bound to the request context.
• •Resource attributes on spans, logs, and metrics are set independently per process, so the join key (service.name, service.version, deployment.environment) drifts between signals.
• •Asynchronous work such as queue consumers, scheduled jobs, or background workers do not capture the parent context at enqueue time and therefore cannot resume correlation on consumption.
• •Sampling decisions are applied to traces only, so correlated logs and metrics are emitted for requests that have no retained trace span to anchor them.

First ten minutes

1. 01Confirm the failure mode: pick one recent request and verify independently that its logs, its metric points, and its trace span exist in their respective backends.
2. 02Record the identifiers actually present in each signal for that single request and list which fields overlap and which do not.
3. 03Identify the boundary where correlation breaks by repeating the join for two or three requests and noting whether the breakage is per-request, per-service, or per-signal-type.
4. 04Inspect the request ingress for the presence of a propagation header and note whether downstream services see the same value or a freshly minted one.
5. 05Capture the observed join key fields (trace_id, span_id, request_id, session_id) and the resource attributes attached to each signal before forming any hypothesis.

Evidence to collect

• •One concrete request with its trace_id, the log entries that reference that trace_id, and the metric points that share the trace_id or a derived label.
• •The propagation header value at the ingress edge and the same header value observed at each downstream service boundary.
• •The list of resource attributes (service.name, service.version, deployment.environment, host.name) attached to a sampled span, a sampled log record, and a sampled metric data point.
• •A counter of requests that produced a trace span versus requests that produced log lines during the same time window, to surface sampling-induced gaps.
• •The list of asynchronous boundaries (queues, schedulers, thread pools) touched by a traced request and whether context was captured at each boundary.

Where to look

• •At the service ingress boundary: the HTTP or RPC handler that first receives a request, where a propagation header should be parsed and a context object created.
• •At each downstream service boundary: outgoing client calls and incoming server handlers, where context must be injected and extracted respectively.
• •At asynchronous boundaries: queue producers, queue consumers, scheduled task runners, and thread executor submission sites, where context typically has to be captured manually.
• •At the instrumentation layer: the OpenTelemetry SDK configuration for the log, metric, and trace exporters, where resource attributes and propagators are bound.
• •At the backend join surface: the tracing UI search field, the log search field, and the metric label explorer, to confirm each backend can accept the same identifier.

Diagnostic steps

1. 01Form the join hypothesis: for one known request, list the fields each signal must share. If any field is absent from any signal, that signal is the suspect.
2. 02Test propagation continuity by tracing one request across three hops and recording the propagation header value at each hop; mismatch indicates a strip or rewrite.
3. 03Test resource attribute parity by exporting one span, one log, and one metric from the same process and comparing the attached attributes for drift in service.name or service.version.
4. 04Test asynchronous correlation by enqueuing a job inside a traced request, consuming it, and checking whether the consumer's span is parented to the producer's context.
5. 05Test sampling alignment by checking whether the sampled trace rate approximately matches the rate of requests that emit correlated logs and metrics; large divergence implies sampling is decoupled.
6. 06Test log-to-trace linkage by querying the log backend with a trace_id and confirming the result set includes the expected span, then performing the inverse query from the trace UI.

Common mistakes

• •Assuming correlation works because traces appear healthy; traces can be intact while logs and metrics carry unrelated identifiers.
• •Searching the log backend with a field name (request_id) that the application does not emit, then concluding correlation is broken when the field was never set.
• •Confusing high-cardinality user identifiers (user_id, session_id) with propagation identifiers (trace_id, span_id); they serve different join purposes and must not be conflated.
• •Configuring resource attributes only on the trace exporter and forgetting the metric and log exporters, which causes drift across signals.
• •Reading a sampled trace as evidence that all requests are correlated; sampling hides the gaps where correlation actually fails.

Safe fixes

• •If a propagation header is missing at ingress, configure the inbound instrumentation to parse the W3C Trace Context header and bind it to the request context before any downstream call.
• •If resource attributes drift between signals, centralize the attribute set in the OpenTelemetry SDK so the trace, log, and metric exporters share the same resource definition.
• •If asynchronous work breaks correlation, capture the current context at enqueue time and re-activate it inside the consumer before any log or span is emitted.
• •If logs do not carry trace_id, configure the log appender or bridge to read the active span context and inject trace_id and span_id into each log record.
• •If metric points lack the join key, add exemplars that bind a metric bucket to a recent trace_id, so a dashboard data point can be drilled into its trace.

Prove the fix

1. 01Replay or capture one request end-to-end and confirm that a single trace_id appears in the trace backend, in the log backend search results, and as an exemplar on at least one metric data point.
2. 02Confirm propagation continuity: the propagation header value at the ingress equals the value observed at every downstream hop for the same request.
3. 03Confirm resource parity: the attributes service.name and service.version attached to one span, one log, and one metric from the same process are byte-identical.
4. 04Confirm async correlation: an enqueued job produces a child span whose parent span_id matches the span that enqueued it.
5. 05Confirm regression coverage: the join succeeds for ten consecutive sampled requests, and a log-to-trace pivot from a real log line returns the expected trace within the expected latency.

Prevention and next steps

• •Treat correlation identifiers as part of the service contract: define the set of join fields (trace_id, request_id, service.name, deployment.environment) and enforce them in code review and SDK configuration.
• •Add a synthetic check that emits one correlated log, metric, and trace per build, and fail the build if any signal lacks the agreed join fields.
• •Audit every asynchronous boundary during onboarding of new queues, schedulers, or worker pools to confirm context capture is wired before traffic is enabled.
• •Keep the OpenTelemetry propagator list explicit and reviewed whenever a new client or server library is added, so propagation is not silently disabled by a default.
• •Review resource attributes on every exporter after each release, and verify parity across trace, log, and metric exporters rather than per signal in isolation.

Safe commands and checks

grep -RIn "traceparent" <service_source_path> | head -n 50
grep -RIn "trace_id\|span_id\|request_id" <log_config_path> | head -n 50
grep -RIn "Resource(service\|service.name\|service.version" <otel_config_path> | head -n 50
grep -RIn "set_exporter\|add_exporter\|OTLPLogExporter\|OTLPMetricExporter\|OTLPSpanExporter" <otel_config_path> | head -n 50
grep -RIn "exemplar\|Exemplar" <metric_emission_path> | head -n 50
awk '/trace_id/{print $1}' <sampled_log_file> | sort -u | head -n 20
awk '/span_id/{print $1}' <sampled_log_file> | sort -u | head -n 20