How to verify OpenTelemetry context crosses async work

The symptoms

• •A child span executed inside a queue worker, event-loop callback, or thread-pool task shows a different trace_id than the parent request that enqueued or scheduled it.
• •A span recorded after an async hop appears as a root span (no parent span_id) even though it was clearly triggered by an earlier request.
• •Traceparent or context headers are present on the inbound request but absent, replaced, or empty on the outbound call made after the async boundary.
• •Traces in the frontend show a single trace; the same logical flow in the backend shows two disconnected traces stitched together by a guessed causal edge.
• •Sampling decisions made for the originating request are not applied to the asynchronous child, so the child is dropped or recorded with a different sampling flag.
• •Context-suppression logs or SDK warnings mention missing context, cleared context, or an HTTP textmap propagation failure only on the async path.

Likely causes

• •The context is not captured into the closure, job payload, or envelope before the work is scheduled, so the producer has no context to restore.
• •The context is captured but the consumer ignores it and starts a new context, span, or trace instead of attaching the captured one as parent.
• •The boundary is a thread-pool executor, scheduled task, or worker queue that uses an explicit Context.execute-style wrapper that the code does not use, so propagation is silently dropped.
• •Serialised job payloads (message headers, HTTP headers, gRPC metadata, event properties) strip traceparent fields before re-injection, or rebuild the envelope without the captured context.
• •The instrumentation auto-instrumentation hook is registered on the producing thread but not on the consumer thread, so the consumer creates a new context.
• •Sampling is applied at the producer and the consumer does not receive or honour the sampling flag carried with the context, so the child span is treated as unsampled.
• •Instrumentation is loaded but the propagator is replaced or returns an empty header set, so cross-boundary calls carry no headers at all.

First ten minutes

1. 01Confirm the failing scenario: identify the awaited boundary (queue, thread pool, setTimeout, Promise continuation, event handler, callback) and the two threads or processes on each side.
2. 02Capture the inbound traceparent header on the producing side and write it to a structured log field along with the existing trace_id and span_id.
3. 03Capture the same header field on the consumer side right before the child span is started and log it together with the trace_id the consumer actually used.
4. 04Inspect the job payload, message envelope, or HTTP headers for the presence of context fields and whether they equal the producer values.
5. 05Check the SDK initialisation order: propagation, sampler, and exporter must be configured before any instrumented code runs; record the order observed.
6. 06Compare sampling flags in the producer and consumer logs; mismatched flags indicate the consumer is not honouring the producer decision.
7. 07Hold the captured producer and consumer snapshots side by side and decide whether the child is connected, disconnected, or partially connected before changing any code.

Evidence to collect

• •Producer-side trace_id, span_id, and traceparent header value at the moment work is scheduled or enqueued.
• •Consumer-side trace_id, span_id, and traceparent header (or lack thereof) at the moment the child span starts.
• •Raw job payload, message envelope, HTTP headers, or gRPC metadata fields configured to carry context across the boundary.
• •Sampling flag and any tracestate fields on both sides of the boundary.
• •SDK initialisation order: configuration of propagator, sampler, tracer provider, and exporter; if injection order changed, record the new order.
• •Process or thread identifier observed on each side of the boundary to confirm the hop actually crossed a thread or process.
• •Exporter or collector view that shows the parent-child link or the missing parent for the same logical operation.

Where to look

• •The boundary between the request-handling thread and the worker thread, queue consumer, or event-loop continuation that performs the async work.
• •The serialisation step where the message job, HTTP request, or event payload is constructed and any header-like fields are added or stripped.
• •The propagator configuration on each side: which propagator is registered, which carrier is used, and whether the producer and consumer agree on the same set of fields.
• •The sampler and its parent-based decision logic, because a mismatched decision causes the child to be dropped even when context is present.
• •The auto-instrumentation registration list, particularly for the executor, scheduler, or messaging library in use, to confirm the consumer side is actually instrumented.
• •The exporter view for the trace, looking for two records with the same operation that share a trace_id (good) or have distinct trace_ids (bad).

Diagnostic steps

1. 01Capture context on both sides: log traceparent, trace_id, span_id, and sampling flag right before scheduling and right before starting the child span.
2. 02Compare the producer trace_id to the consumer trace_id; identical values indicate the child is still connected, different values indicate a fresh context was created.
3. 03Inspect the carrier fields used at the boundary and verify the traceparent field is present and non-empty after serialisation and deserialisation.
4. 04Verify propagation symmetry: the same propagator must be registered on both sides, with the same carrier type, otherwise one side will see no headers.
5. 05Check the sampler on the consumer: a non-parent-based sampler will treat the child as a new trace and may apply a different sampling decision.
6. 06Trace the SDK lifecycle: confirm propagator, sampler, tracer provider, and exporter are configured once and before instrumentation runs; reconfiguration after startup is a common silent reset.
7. 07Reproduce the hop in a minimal harness that logs trace_id and span_id on both sides, and decide whether the failure is in capture, transmission, restoration, or sampling.

Common mistakes

• •Assuming propagation is automatic across a thread pool or queue; many async hops require an explicit capture-and-attach step.
• •Serialising the job payload with a generic mapper that strips unknown fields, which removes the traceparent field before the consumer ever sees it.
• •Starting a new span on the consumer without attaching the captured context as parent, causing the child to become a root span.
• •Using different propagators on the producer and consumer, so the headers are written in one format and read in another and neither side finds them.
• •Replacing the global propagator after the first request has been handled, which silently breaks propagation for the next hops.
• •Confusing context attachment with traceparent logging: a logged traceparent is not evidence of attachment unless the child span actually references that parent.
• •Trusting the exporter view alone without checking the carrier on the wire; a connected-looking trace can be the result of back-end stitching rather than real propagation.

Safe fixes

• •If the carrier fields are missing, capture the current context into the job payload at the producing call site and re-inject it on the consumer before starting the child span; only apply after confirming the capture is absent.
• •If the child starts as a root span, attach the captured context as the explicit parent at the child span creation site; only apply after confirming the child span_id has no recorded parent.
• •If the propagators differ, configure the same propagator on both processes before any scheduled work is dispatched; only apply after verifying the producer and consumer propagate sets differ.
• •If the sampler disagrees, switch the consumer sampler to a parent-based strategy and re-run the hop; only apply after confirming sampling flags actually differ.
• •If the SDK lifecycle is wrong, move all configuration into a single initialisation step that runs before any instrumented code, and reload the process to confirm the new order.
• •If capture or attach is missing in framework code, add an explicit wrapper around the boundary that records the trace_id and span_id it captured and applied.

Prove the fix

1. 01Producer log line and consumer log line for the same logical operation show identical trace_id values and the consumer span_id is a child of the producer span_id.
2. 02A trace search in the exporter for the producer trace_id returns the child span with a parent_id pointing to the producer span, not a separate root span.
3. 03The carrier field on the consumer side contains a traceparent value equal to the producer value, and a tracestate value when the producer had one.
4. 04Sampling flags on producer and consumer are consistent for the same trace_id across the hop, and the child span is recorded when the producer was sampled.
5. 05A targeted regression check that exercises the exact async boundary and asserts trace_id equality still passes after the change, and fails on the prior build.

Prevention and next steps

• •Define a single context-capture helper used at every async boundary, and require code review to approve any new boundary that does not call it.
• •Add an integration test that runs a known async hop, asserts trace_id equality on both sides, and fails the build if the assertion breaks.
• •Keep propagator and sampler configuration in one initialisation module and forbid late reconfiguration that would reset the global propagator.
• •Document the carrier field name and format used to carry context across each queue, message bus, or HTTP client in the system.
• •Sample audit traces in pre-production for the hop pattern: root span with no parent on a code path that is downstream of a request handler.

Safe commands and checks

From the exporter or observability backend, list traces that contain more than one root span on the same code path and filter by the name of the async hop.
For a known trace_id, fetch the span tree and confirm the child span of the awaited work has a parent_id that is the producer span_id.
Capture a one-line structured log on the producer side at the schedule point with fields: trace_id, span_id, traceparent, sampling_flag, and a thread or worker identifier.
Capture a matching one-line structured log on the consumer side at the start of the child span with the same fields and a different thread or worker identifier.
Diff the two log lines for the same logical operation: identical trace_id, consumer span_id is a child of producer span_id, sampling flags agree, and traceparent carrier field is non-empty on the consumer.
From the exporter, query the same trace_id and confirm the span tree shows the expected parent-child relationship rather than two separate roots.