How to verify log sampling preserves rare failure evidence

The symptoms

• •A rare failure (an error that occurs in fewer than one in a thousand requests) is reported by users, but the log search returns no matching entries for the affected time window.
• •Metrics show the error rate spike, but the corresponding trace IDs cannot be found in the log backend, or are present only as isolated lines without surrounding context.
• •An investigation completes but the root cause cannot be proven because the kept log lines lack the request-scoped fields (user, tenant, build version, feature flag) needed to reproduce the sequence of events.
• •Sampling configuration changes (rate, rules, tail sampling policy) are deployed, but no test exists to confirm that rare failure traces are still being kept with their correlated log evidence intact.
• •Audit logs or compliance evidence for a sensitive operation appear to be missing or truncated, even though the operation was recorded in the metrics layer.

Likely causes

• •Head-based probability sampling drops the entire trace decision early, before logs are evaluated, so logs from rare error spans never reach the backend.
• •Tail-based sampling rules evaluate only on span attributes or status codes and exclude the log records emitted before or after the span lifecycle, breaking correlation.
• •Sampling decisions are made independently per log stream or per service, so the trace survives sampling but the related application log lines are dropped by a separate, uncoordinated sampler.
• •Log enrichment (injecting trace_id and span_id into log records) is missing or incorrect, so even kept logs cannot be joined back to their trace.
• •Volume-based log sampling (drop one in N lines above a rate threshold) silently discards the rare error line that happened to fall in the dropped bucket.
• •Retention and indexing tiers differ between metrics, traces, and logs, so the trace is retained but the log index has aged out by the time the rare failure is investigated.

First ten minutes

1. 01Confirm the reported failure window (UTC start and end) and the trace identifier, request ID, or user identifier that anchors the incident, so the verification target is unambiguous.
2. 02Open the metrics view for the failing service and confirm the error spike is visible in the same window; this proves the failure is real and not a reporting artifact.
3. 03Query the log backend for the known trace_id or request_id from the failure window and record how many log records, if any, are returned for that identifier.
4. 04Inspect the sampling configuration files or runtime settings for the collector, agent, or SDK and identify whether the rule is head-based, tail-based, volume-based, or a combination.
5. 05Read the application logger's enrichment output for one kept record and confirm that trace_id, span_id, and any request-scoped context fields are present and well-formed.
6. 06Compare the kept log record count against the expected count (one per significant event in the trace) and decide whether the gap is consistent with the configured sampling rate or indicates a coverage defect.

Evidence to collect

• •The exact sampling rule definition in effect during the failure window, including rate values, match conditions, and any conditional logic such as error-only or latency-based keep rules.
• •The list of log streams or services that participate in the sampling decision, to identify uncoordinated samplers.
• •A sample of log records emitted by the failing service during the failure window, including their trace_id, span_id, severity, message, and request-scoped attributes.
• •The corresponding trace records, showing which spans were kept, which were dropped, and whether the root decision span is present.
• •Retention and indexing tier settings for the log backend, including the time-to-live for the index that holds the failure window.
• •A baseline measurement of keep ratio for known-good requests in the same window, as a control against the rare-failure keep rate.

Where to look

• •At the OpenTelemetry collector or vendor agent configuration boundary, where sampling policies are translated into operational rules.
• •At the application logging boundary, where log records are emitted and where trace context is supposed to be injected.
• •At the log ingestion boundary, where volume-based or rate-based log droppers may apply before indexing.
• •At the trace storage boundary, where tail sampling decisions are finalized and dropped spans are removed.
• •At the log storage boundary, where retention tiers and indexing policies differ from trace storage.
• •At the correlation join boundary, where trace_id and span_id are used to link logs to spans, and where a missing field breaks the join.

Diagnostic steps

1. 01Reproduce the sampling configuration in a staging environment, generate a controlled trace that includes a deliberate rare error event, and observe whether the error span and its surrounding logs are kept together.
2. 02Inspect the trace record for the failing request and verify that the root span and at least one child span containing the error are both retained; a kept trace with only the parent and no error context is a sampling coverage failure.
3. 03For each kept span, search the log backend by trace_id and span_id and confirm that log records emitted during that span's lifetime are present; missing logs despite a kept span indicates a log-side sampling gap.
4. 04Count the number of log records per kept trace for the failing request and compare to a non-failing baseline trace of equivalent length; a count an order of magnitude lower suggests head-based or volume-based log dropping.
5. 05Verify that log enrichment is consistent by sampling ten kept records and confirming every one contains a non-empty trace_id and span_id; any record missing the field is uncorrelatable.
6. 06Check the log index age relative to the failure window; if the trace is retained but the log index has aged out, the boundary of the gap is retention, not sampling.
7. 07Cross-reference any conditional keep rules (for example, keep all traces with status=error) against the failure trace and confirm the rule condition evaluated true; otherwise the rule definition itself is the defect.

Common mistakes

• •Treating a low kept-log count as proof that sampling is broken, without comparing to the baseline keep ratio for healthy traffic in the same window.
• •Assuming all log sampling is coordinated; many deployments apply independent samplers in the application, the agent, and the collector, each capable of dropping evidence on its own.
• •Reading the collector's sampling documentation rather than the configuration that is actually loaded, which leads to verifying an inactive rule.
• •Ignoring retention tier differences and concluding sampling is at fault when the log index simply expired before the investigation.
• •Verifying only the root span and not the child spans that contain the actual error payload, missing the case where the parent is kept but the error context is dropped.
• •Trusting dashboards that report a "sampling rate" without showing the policy that produced it, leading to verification against a derived number rather than the rule.

Safe fixes

• •If head-based sampling is dropping the trace before logs are evaluated, add or tighten a tail-based keep rule that retains any trace containing an error status, conditional on evidence that the rule is actually loaded by the collector.
• •If log enrichment is missing trace_id or span_id, update the logger configuration to inject context from the active span and verify by inspecting ten kept records, conditional on evidence that enrichment is absent in the sampled output.
• •If independent log samplers are uncoordinated with trace sampling, route logs through the same collector pipeline as traces and disable application-level log sampling, conditional on evidence that the application sampler is active.
• •If volume-based log dropping discards the rare error line, configure the dropper to exempt lines whose severity meets a defined error threshold, conditional on evidence that volume rules are in effect.
• •If the log index ages out before the trace, align retention tiers so log indexes persist at least as long as trace indexes, conditional on evidence that the log TTL is shorter than the trace TTL.

Prove the fix

1. 01Run a controlled experiment that emits a synthetic rare-failure trace with a known trace_id and a known number of error log records; the trace and at least the configured minimum number of correlated error logs must be retrievable from the backend after the experiment.
2. 02Query the log backend by trace_id for the synthetic experiment and confirm the returned record count meets or exceeds the documented minimum correlated records per kept trace.
3. 03Repeat the experiment after any configuration change and confirm the keep behavior is unchanged for healthy traffic while improving for the rare-failure case, proving the fix did not regress baseline sampling.
4. 04Inspect a random sample of ten kept log records from a production window after the fix and verify each contains a non-empty trace_id and span_id, proving enrichment remains intact under the new policy.
5. 05Re-run the original failed investigation using only the trace_id from the reported incident window and confirm the correlated log evidence is now sufficient to reconstruct the event sequence.

Prevention and next steps

• •Treat sampling configuration as code: version it, review it, and require a verification test that asserts rare-error traces and their correlated logs are kept together before any change merges.
• •Maintain a documented alignment between trace sampling policy, log sampling policy, and retention tier durations, reviewed on a fixed cadence against the deployed configuration.
• •Periodically replay a synthetic rare-failure trace through the production pipeline and alert if the correlated log count falls below a defined minimum, so coverage regressions are detected before an incident.
• •Require log enrichment to be verified at startup of every service that emits logs, so missing trace context fails fast rather than at investigation time.

Safe commands and checks

grep -nE 'sampler|tail_sampling|probabilistic' <collector-config.yaml>
grep -nE 'trace_id|span_id|inject' <logger-config.yaml>
awk -F'\"trace_id\"' '/trace_id/{print NR, $2}' <sample-log-output.jsonl> | head -n 20
grep -c '"severity":"ERROR"' <sample-log-output.jsonl>
diff <baseline-keep-ratio.txt> <failure-window-keep-ratio.txt>