OpenTelemetry trace context loss: find the async boundary without propagation

The symptoms

• •Downstream spans appear under a new, unrelated trace_id in your tracing backend, while the upstream span is on a different trace entirely.
• •A span created immediately before an awaited callback, queue dispatch, thread-pool handoff, or RPC fan-out begins a fresh trace rather than linking to the parent operation.
• •Trace context is present in the calling thread but missing in a worker thread, executor task, message handler, or child process where the next span is recorded.
• •Sampling decisions appear inconsistent: a root span is dropped while child work that should inherit its sampling decision is also dropped or vice versa.

Likely causes

• •An asynchronous boundary such as a callback, Promise continuation, executor submit, or message-queue dispatch was not instrumented, so the OTel Context did not flow across it.
• •Custom thread pools, schedulers, or worker threads were started without the OTel Context API, so the worker thread sees a fresh, empty Context.
• •Trace context was extracted from one carrier (e.g., HTTP headers) but the outbound call uses a different protocol or message format, and no propagator is registered for that format.
• •An instrumented library created a span via the tracer provider, but the surrounding application code captured and re-attached Context using a stale snapshot taken before the span activated.
• •Manual context propagation was attempted with the wrong abstraction (for example, copying only the trace_id string instead of the full OTel Context), so flags and trace state did not survive the hop.

First ten minutes

1. 01Confirm the symptom is propagation loss and not a broken exporter: verify the upstream service emits at least one span with a valid trace_id on every request you can observe.
2. 02Identify the single async hop where trace_id changes by inspecting the trace timeline; record the operation name just before the hop and the operation name just after.
3. 03Determine whether the hop crosses a thread, a queue, a process, or a network boundary, and which library or framework owns that boundary in your stack.
4. 04Check which OTel instrumentations are registered for that boundary and whether any of them appear as missing or disabled for the relevant runtime or library.
5. 05Capture the traceparent value at the producer side of the hop and search for it at the consumer side; if it is absent, the boundary is not propagating context.

Evidence to collect

• •The trace_id of the upstream span at the point where work is enqueued or scheduled.
• •The trace_id of the first span created on the receiving side of the hop; compare with the upstream trace_id byte-for-byte.
• •The full traceparent header or equivalent carrier value present at the producer side and whether the same value is observable at the consumer side.
• •Runtime evidence of the hop: thread name change, executor name, queue or topic name, message ID, or child process PID.
• •Registered instrumentations and propagators visible in the OTel SDK configuration or diagnostic output for the affected service.

Where to look

• •The async boundary itself: the submit call into a thread pool, the dispatch into a queue, the scheduling of a callback, or the handoff to a worker.
• •The receiving side of that boundary, which may run on a different thread, in a different process, or in a different service reachable over the network.
• •The instrumentation layer: which OTel instrumentation libraries are attached to the executor, scheduler, message client, or RPC framework involved in the hop.
• •The propagator configuration: where propagators are registered and which carriers (HTTP headers, message metadata, gRPC metadata) they actually cover.

Diagnostic steps

1. 01Reproduce the hop deterministically by triggering the same code path twice and recording both the producer-side and consumer-side trace_id values from your tracing backend.
2. 02Compare the two trace_id values; identical values indicate context flowed correctly, differing values indicate the boundary is the fault line.
3. 03Inspect the SDK diagnostics or configuration dump to list active instrumentations and propagators for the affected service, and confirm whether an instrumentation for the boundary library is present.
4. 04Enable any built-in self-tracing or debug output the SDK exposes for the propagator, and observe whether the carrier is read or written at the suspected boundary.
5. 05Capture the producer-side traceparent at the moment of dispatch using the OTel Context API and compare it to the traceparent the consumer-side instrumentation would have used to extract context.
6. 06If the hop crosses a network or process boundary, record the carrier on the wire and verify the receiver's propagator is configured to read that carrier format.

Common mistakes

• •Assuming the OTel SDK propagates context across every async boundary automatically, when in fact only instrumented boundaries propagate context.
• •Conflating trace_id propagation with span linking; manual links are not a substitute for carrying context across the live boundary.
• •Reading the traceparent from logs rather than from the active OTel Context, which can show a stale or empty value depending on log capture timing.
• •Adding a custom propagator without ensuring both producer and consumer register it for the same carrier format and field names.
• •Capturing Context before the parent span is activated, which snapshots an empty or unrelated Context that gets re-attached downstream.

Safe fixes

• •If the boundary is an unsupported async primitive, wrap the dispatch so that the current OTel Context is captured before scheduling and explicitly made current inside the worker or callback before any span is started there.
• •If a custom executor, scheduler, or worker pool is in use, register or implement an OTel instrumentation for it so context flows from the submitter thread to the worker thread.
• •If the hop crosses a network boundary, ensure the propagators configured on the producer side write the carrier into the outbound headers or metadata, and the consumer side extracts from the same carrier format.
• •If multiple propagators are needed, register them in the same order on both sides and confirm the carrier fields they emit match what the receiver expects.
• •After any fix, re-run the reproduction and require that the producer-side trace_id and the consumer-side first-span trace_id are byte-identical for the same logical operation.

Prove the fix

1. 01For a fixed percentage of repeated reproductions, the trace_id of the span created immediately after the hop equals the trace_id of the span created immediately before the hop, with no manual trace linking required.
2. 02A traceparent captured at the producer side of the hop is observed at the consumer side within the same logical operation, either in-process via the OTel Context or on the wire via the configured carrier.
3. 03Sampling decisions are consistent across the hop: a sampled upstream trace results in a sampled downstream span and vice versa, since sampling propagates with context.
4. 04Re-running the same code path after disabling the suspected fix reintroduces the divergence, confirming the fix is what restored propagation rather than a coincidental change.

Prevention and next steps

• •Maintain an inventory of async primitives, executors, and message clients in use, and map each to the OTel instrumentation that is expected to propagate context across it.
• •In CI, run a propagation smoke test that exercises each known boundary and asserts that the producer-side and consumer-side trace_id match for a known logical operation.
• •Keep propagator configuration identical across services that exchange requests, and review any new carrier format against the registered propagator list before deploying.
• •Avoid hand-rolling context propagation; prefer the OTel Context API and registered instrumentations over copying trace_id strings or span references manually.

Safe commands and checks

ps -o pid,comm -p <pid>
grep -RIn --binary-files=without-match -E 'traceparent|tracestate' <config_or_log_path>
grep -RIn --binary-files=without-match -E 'propagator|instrumentation' <config_or_log_path>