Traces stop at the API boundary: identify missing propagation

The symptoms

• •In the trace backend, the boundary service (e.g., API gateway, ingress, BFF) shows a server span, but downstream calls appear as separate root spans with no parent reference
• •Trace ID continuity breaks at the same edge on every request, producing "fan-out" trees of disconnected traces rather than one correlated trace tree
• •Downstream service traces show a fresh traceparent header (or no header at all) on the receiving side, even when the caller is instrumented
• •W3C traceparent or W3C tracestate headers are absent from outbound requests when inspected at the wire or in client logs, even though the inbound span context is present
• •Manual spans created inside the downstream service appear unparented, while auto-instrumented spans from the same library may or may not attach depending on which client is used

Likely causes

• •An HTTP/gRPC client made inside the boundary service is not wrapped by OpenTelemetry instrumentation, so it emits spans without injecting the W3C traceparent header
• •Manual headers set on an outbound request (e.g., explicitly setting Authorization or X-Request-Id) overwrite or strip the propagation carrier instead of extending it
• •A proxy, sidecar, service mesh, or load balancer in front of the downstream service injects or rewrites headers, breaking the traceparent chain before the receiver sees it
• •Propagators are configured inconsistently between the calling service and the called service, so the receiver expects a format the sender does not emit (e.g., B3 only on one side, W3C only on the other)
• •Sampling decisions are recorded correctly but the propagator is unset, defaults to Noop, or is overridden by a null Context Propagators list, producing spans with no linkable context
• •Custom messaging, queue, or async-job boundary moves context across a non-HTTP carrier and the OTel context is not reattached to the consumer side, creating a new root span per consumer execution

First ten minutes

1. 01Open the trace backend and pick one example trace that ends at the boundary service; record the service name of the last span and the trace ID of the disconnected root in the downstream service
2. 02Confirm the inbound request to the boundary service actually carried a traceparent header by inspecting edge access logs or the first span's attributes for the incoming context
3. 03Capture the full outbound HTTP headers for the same failing request using a single debug-level log or a temporary header-dump middleware that only writes the existing header keys (no header values)
4. 04Compare the W3C traceparent value on the inbound side to the W3C traceparent on the outbound side; identical version-traceid-parentid indicates propagation worked, missing values indicate the propagator did not run
5. 05Disable sampling temporarily on the boundary service (or pin a head-based decision) so the propagation check is not confounded by dropped spans, then re-run a single representative request
6. 06Reject "fix" attempts that change span names, add attributes, or add manual cross-trace links, since these mask the propagation defect rather than resolving it

Evidence to collect

• •The exact set of header keys present on the inbound request to the boundary service and on the outbound request to the downstream service (presence, not values)
• •The Propagators configuration as initialized in both the calling and the called service, including any custom or composite propagator chain
• •The SDK and instrumentation library versions for the outbound client, since several clients defaulted to non-instrumented behavior in earlier releases
• •Whether any intermediate proxy, service mesh sidecar, or API gateway sits between the two services, and whether it is configured to forward or strip traceparent
• •A single trace ID and its expected parent-child relationship from the backend, used as the reference case for the fix verification
• •The sampling decision recorded on the boundary span, so propagation is not confused with a sampling drop

Where to look

• •At the HTTP client boundary inside the boundary service: the outbound request-building code, any wrapper that constructs headers, and the registered client instrumentation module
• •At the propagation configuration layer: the TracerProvider builder, the OpenTelemetry SDK setup file, and any environment variable that selects propagators (OTEL_PROPAGATORS)
• •At the network edge between the two services: reverse proxy, ingress controller, service mesh sidecar, and any header-filtering middleware that could rewrite or drop traceparent
• •At the receiving service: the inbound server instrumentation, the configured Propagators, and the first span's parent_span_id value, which should equal the caller's span_id if propagation worked
• •At any custom context carrier used for async or message-based handoff: producer code that publishes the message, broker configuration, and consumer code that resumes work

Diagnostic steps

1. 01Choose the boundary service span as the reference, and verify whether the downstream service's trace shares the same trace_id; if not, propagation is broken at or before the outbound call
2. 02Inspect the outbound request headers to confirm traceparent is present and well-formed; missing or malformed values indicate the propagator was not invoked on the client side
3. 03Verify the receiver's configured Propagators include W3C TraceContext (and any other carrier the sender uses); mismatched propagator sets prevent extraction even when the header is sent
4. 04Determine whether the outbound client is instrumented by checking the SDK's loaded instrumentations list; an uninstrumented client will not inject headers regardless of propagator configuration
5. 05Check for middleware that builds headers from scratch; manual header construction that does not include the active context will overwrite the propagator's output
6. 06For async or message-based handoff, verify that the producer explicitly passes the current Context into the messaging instrumentation, and that the consumer extracts context from the message headers before resuming work
7. 07Differentiate propagation failure from sampling failure by comparing the inbound sampling decision to the presence of the downstream span; a present but disconnected root span is propagation, not sampling

Common mistakes

• •Assuming "tracing is on" is sufficient, without verifying that the specific outbound client is instrumented by the OpenTelemetry SDK
• •Editing propagation logic in the receiver only, when the sending side is silently stripping or never injecting the header
• •Setting OTEL_PROPAGATORS to a custom value that excludes W3C TraceContext, which breaks correlation with any service that defaults to W3C
• •Confusing a sampled-out downstream span with a missing parent; a sampled-out span produces no trace at all, while a propagation failure produces a disconnected root trace
• •Adding manual traceparent headers in application code, which typically lack the span_id of the active span and therefore break the parent-child link
• •Trusting service mesh defaults; many meshes strip or normalize traceparent unless a specific "trace propagation" mode is enabled on the route

Safe fixes

• •If the outbound client is uninstrumented, register the matching OpenTelemetry instrumentation module for that client library and confirm it appears in the SDK's loaded instrumentations list before redeploying
• •If the propagator is unset or set to Noop, configure TextMapPropagator with W3C TraceContext (and B3 only if required by an existing service) at the TracerProvider builder, and verify the change in a single canary service first
• •If middleware builds headers manually, change that code to extend the existing headers rather than replace them, so the active context's propagation fields are preserved
• •If a proxy or mesh is stripping traceparent, set the route's header policy to forward it (do not rewrite) and verify the value is unchanged byte-for-byte at the receiver
• •If message-based handoff loses context, pass the producer's Context into the messaging instrumentation explicitly and extract it on the consumer side before starting the consumer's span
• •After any change, re-run the failing request and require the downstream trace to share the same trace_id and parent_span_id as the boundary span before declaring the fix verified

Prove the fix

1. 01In the trace backend, the reference trace now contains a single tree where the boundary service's server span is the parent of the downstream service's spans, with shared trace_id and matching parent_span_id
2. 02Wire-level inspection of a sampled request shows the W3C traceparent header on the outbound request, with the trace_id matching the inbound request and the parent_id matching the boundary service's active span_id
3. 03Across at least 20 consecutive requests between the two services, zero requests produce a disconnected downstream root span, demonstrating the fix is not a one-off coincidence
4. 04The receiving service's first span attribute "parent_span_id" equals the boundary service's outgoing span_id, confirming extraction rather than coincidental trace_id reuse
5. 05No new error spans or dropped traces appear in the trace backend after the change, indicating the propagation fix did not regress sampling or instrumentation health

Prevention and next steps

• •Adopt a standard propagator set (W3C TraceContext as the default, with B3 only where required) and document it in the service's tracing configuration so each new service starts consistent
• •Require that every outbound HTTP, gRPC, and messaging client in the service inventory has a registered OpenTelemetry instrumentation module, and review the list on each dependency upgrade
• •Keep the outbound header construction in a single middleware or helper so it cannot accidentally overwrite the propagator's output, and review that code path during pull-request review
• •Configure any service mesh, ingress, or API gateway in the path to forward traceparent without rewriting, and verify this in the platform team's standard checks
• •Add a synthetic check that calls a known downstream service and asserts shared trace_id in the backend, so future regressions in propagation are detected before they reach production dashboards

Safe commands and checks

grep -RIn "Propagators" <service_source_dir> --include="*.go" --include="*.java" --include="*.py" --include="*.js" --include="*.ts"
grep -RIn "traceparent" <service_source_dir>
grep -RIn "OTEL_PROPAGATORS" <service_config_dir>
env | grep -E "^OTEL_(PROPAGATORS|TRACES_EXPORTER|SERVICE_NAME)" | sort
grep -RIn "WithTextMap\|SetTextMapPropagator\|setPropagators\|GlobalPropagators" <sdk_init_dir>
grep -RIn "Instrumentation\\|UseOtel\\|WithTracerProvider\|WithInstrumentation" <service_source_dir>