Log sampling hides failure: preserve evidence without flooding output

The symptoms

• •A production incident shows a clear error spike in metrics or traces, but the corresponding log search returns empty for the affected request IDs, time window, or correlation IDs.
• •Log volume dashboards look healthy and within budget, yet incident timelines have unexplained gaps where error-level or high-severity events should appear.
• •Sampling is configured at the agent, collector, or ingest endpoint, and toggling the sampling rate briefly to 100 percent makes the missing failures appear, confirming the sampler is the boundary that removed them.
• •Different services or replicas show inconsistent error visibility even though they emit the same log schema, suggesting per-source sampling decisions rather than uniform drops.

Likely causes

• •Head-based probabilistic sampling at the agent or SDK level evaluates each log record independently, so rare events (errors, exceptions, audit entries) are dropped by chance even at moderate sampling rates.
• •Tail-based sampling rules in a collector only retain traces or log batches whose computed priority crosses a threshold; records tagged INFO or without severity metadata fail the priority predicate and are discarded along with useful context.
• •Rate-limiting or volume-based sampling at the ingest boundary caps records per second per service or per stream, and over-quota error records are dropped before persistence.
• •Default sampler configuration is left in place after a debug or load-testing session that set a low sampling ratio, and the value is now silently applying to error traffic.

First ten minutes

1. 01Freeze further configuration changes to the logging pipeline and record the current sampling ratios at every boundary (SDK, agent, collector) so you can compare before and after any experiment.
2. 02Identify one concrete failing request by ID, time, and service, and search the log store for that exact correlation ID and the surrounding two-minute window; the absence of error records establishes the symptom.
3. 03Inspect the in-pipeline telemetry: in the agent UI look at sampled-vs-dropped counters per record class; if errors are in the dropped bucket, sampling is the active boundary.
4. 04Compare the current ratio against the expected ratio for the configured sampler; if expected retention is 10 percent and you observe zero error records across hundreds of failures, head-based sampling is the dominant cause.
5. 05Decide whether to enable a temporary 100 percent sampling rule for the affected service while preserving evidence, and capture the constraint that production traffic must not be impacted by volume.

Evidence to collect

• •The full pipeline topology: which SDK or agent emits records, which collector or processor receives them, and which sink persists them, with explicit sampling processors named.
• •Current configuration of any tail-based sampling processor: priority rules, policy decision wait time, and the attribute keys used to classify records.
• •Dropped-vs-accepted counters from the agent or collector for the affected stream, segmented by severity or record class, over the incident window.
• •At least one identifier that ties traces, metrics, and logs together (trace_id, correlation ID, request ID) and its presence or absence on the supposedly erroring records.
• •The actual emitted record payload from one failing request, captured by a debug sink or a short-lived 100 percent sampler, to confirm the failure existed at the source.

Where to look

• •The OpenTelemetry SDK or agent configuration where the initial sampler is set; this is the earliest evaluation point and decides per-record retention before transport.
• •The collector or processor chain, specifically any tail-based sampling processor that buffers records and applies policy; the decision wait window and the priority attributes live here.
• •The ingest endpoint or log forwarder, including any rate limiter, volume cap, or filter rule applied at the boundary between collection and storage.
• •The log store query interface, where the absence of records for a known correlation ID is the direct evidence that something upstream discarded them.
• •The observability primer semantics for logs versus traces versus metrics; mismatched expectations about what each signal carries often mask that sampling is the cause.

Diagnostic steps

1. 01Pick one confirmed failure from metrics or traces and search the log store for its correlation ID in a tight time window; an empty result establishes that the log pipeline lost the evidence.
2. 02Walk the path from source to storage and list every sampler or rate limit in the chain, recording the current ratio and the attribute or severity rule each one applies.
3. 03At the SDK or agent boundary, enable a debug or always-on sampler for the affected service only and emit a known error event; if it reaches the store, the cause is upstream of the source.
4. 04At the collector boundary, inspect the tail-based sampling processor policy: confirm that severity, error status, or a rule key is in the policy and that policy decision wait does not expire before the error record arrives.
5. 05At the ingest boundary, read the dropped-record counter for error severity during the incident; non-zero drops while the rate limiter reports over-quota confirm volume sampling is the cause.
6. 06Compare expected kept ratio against observed retained ratio per severity; a ratio that matches expectation for INFO but is far below expectation for ERROR implicates severity-blind head sampling.
7. 07Synthesize the result: name the single boundary that, if relaxed, makes the error record appear, and that is the diagnostic answer rather than the broader toolchain.

Common mistakes

• •Assuming the log store query syntax is wrong when the records were dropped at the agent or collector; the data never reached the index, so query tuning cannot retrieve it.
• •Setting a global sampling ratio without per-severity rules, then expecting rare errors to be representatively retained; head-based sampling treats every record with the same probability.
• •Configuring tail-based sampling with a priority rule that only checks span status or trace-level errors, ignoring logs that carry their own severity field, which causes log evidence to be discarded even when a trace is kept.
• •Leaving a low-ratio debug sampler active after a load test and forgetting the change, then shipping that configuration to production where it silently throttles error evidence.
• •Counting dropped records only in aggregate, which hides the fact that errors are over-represented in the dropped bucket while successes comfortably fit the kept quota.

Safe fixes

• •If head-based sampling is the cause: change the sampler to a ratio-based rule that pairs a low base ratio with an always-keep predicate on severity or error status, so error records bypass probabilistic drops while volume stays bounded.
• •If tail-based sampling is the cause: add a policy entry that elevates priority on record class equal to error, on severity equal to error or higher, or on a present exception stack trace, and ensure policy decision wait covers the slowest expected emission.
• •If a rate limiter is the cause: split the budget so error severity has a dedicated reserved quota separate from the informational quota, so error evidence cannot be starved by verbose traffic.
• •Apply each fix on the affected service or stream only, behind a feature flag or scoped config, so a misconfiguration cannot flood shared output for unrelated services.
• •Capture the before-and-after dropped-versus-kept counters per severity so the change is reversible with a documented rollback to the original ratio and rule set.

Prove the fix

1. 01Replay or trigger one known failure and verify that its correlation ID is now retrievable in the log store within the original time window, with at least the error-level record present.
2. 02Check that the dropped-versus-kept counter for error severity during a controlled burst of failures is within a few percentage points of the configured always-keep ratio rather than the base head-based ratio.
3. 03Confirm that total log volume per minute for the affected service did not exceed the original budget by more than an agreed percentage, proving evidence preservation did not flood output.
4. 04Export a small report of kept-versus-dropped records segmented by severity for the next 24 hours, and watch for any return of severity-blind dropping as a regression signal.
5. 05Document the named sampler, processor, or rate limiter that was changed, the rule added, and the rollback command, so the fix is auditable and reversible.

Prevention and next steps

• •Adopt a default pipeline that pairs a low probabilistic sampler with an always-keep predicate on severity error or higher, and codify that as a shared base configuration reviewed on every change.
• •Add an automated dashboard that shows dropped-versus-kept records segmented by severity, with an alert when error-severity drops exceed a small threshold over a rolling window.
• •Require any temporary sampler override such as those used in load tests to expire automatically or to be cleared by a checklist step before merge, so debug ratios cannot leak into production.
• •Define a standard correlation identifier across logs, traces, and metrics and assert its presence in synthetic probes, so a missing identifier is itself an early signal of pipeline loss.
• •Periodically replay a synthetic failure end-to-end and assert that its evidence reaches the log store, treating any miss as a release-blocking regression rather than a normal outage.

Safe commands and checks

grep -R "sampler" /etc/otelcol/ 2>/dev/null | head -n 20
grep -R "tail_sampling" /etc/otelcol/ 2>/dev/null | head -n 20
grep -R "policy:" /etc/otelcol/ 2>/dev/null | head -n 40
grep -R "rate_limiter\|drop_ratio\|keep_ratio" /etc/otelcol/ 2>/dev/null | head -n 20
grep -R "severity\|level" /etc/otelcol/ 2>/dev/null | head -n 30
grep -n "decision_wait\|expected_period" /etc/otelcol/*.yaml 2>/dev/null