Logging evidence checklist

The symptoms

• •Multiple requests appear under the same correlation identifier in logs, making per-request attribution impossible.
• •Logs show an error message but omit which attempt number, retry, or boundary produced it.
• •Search by user ID or session token returns no rows, while the user reports a failed action.
• •Two services log the same request under different IDs, so a trace cannot be reconstructed end-to-end.
• •A failure is reproducible but the relevant log line lacks a timestamp, severity, or structured field required for filtering.

Likely causes

• •Correlation identifiers are not propagated across a service, queue, or HTTP header boundary.
• •Structured logging is disabled or fields are rendered as concatenated strings rather than key-value pairs.
• •Log retention, sampling, or redaction policy strips the identifiers needed for attribution.
• •Retry logic emits a new identifier per attempt without preserving the original request correlation ID.
• •Clock skew between emitters prevents aligning events from different services on a shared timeline.

First ten minutes

1. 01Confirm which signal layer you are inspecting: application log, platform log, or trace span; each carries different identifier fields.
2. 02Identify the boundary where the failure was reported (service entry, queue handoff, downstream call) and locate the log emitter closest to it.
3. 03Capture the exact log line, including timestamp format, severity, and any structured fields present.
4. 04Search the surrounding time window for the same correlation identifier across adjacent services; record which services emit and which do not.
5. 05Record whether identifiers are stable across retries or whether each attempt has a new value.

Evidence to collect

• •A representative failed log line with all fields visible, not truncated or rendered as a single string.
• •The correlation, trace, and request identifiers actually emitted, with the field name used by each service.
• •Timestamp source and format for each emitter, to detect drift between services.
• •Retention and sampling configuration for the log store, since dropped records hide evidence.
• •Whether the identifiers in question are subject to redaction or hashing that prevents cross-service lookup.

Where to look

• •At the HTTP or RPC entry point where the inbound correlation header should be read and recorded.
• •At the queue producer and consumer pair, where identifiers must survive serialization.
• •At retry and circuit-breaker boundaries, since these emit additional attempts with their own identifiers.
• •At the structured logging formatter configuration, which controls whether fields are queryable.
• •At the log router or collector, which may strip, hash, or sample fields before storage.

Diagnostic steps

1. 01Verify each emitter logs a correlation identifier as a structured field, not embedded in a free-text message.
2. 02Compare the identifier field name across services; mismatches indicate propagation was attempted but normalized incorrectly.
3. 03Check whether the inbound header is read before any logging middleware runs; otherwise the first log line lacks the identifier.
4. 04For each retry attempt, confirm whether the original correlation ID is preserved or replaced; the answer determines whether retries are individually attributable.
5. 05Inspect the log formatter output for one record to determine whether fields are key-value, JSON, or flattened; only structured form is reliably queryable.
6. 06Confirm clocks are synchronized within a documented skew budget so timestamps from different services align.

Common mistakes

• •Assuming a trace ID exists in logs when it is only attached to spans in a separate tracing backend.
• •Treating the absence of a correlation field as a logging bug, when the field is intentionally redacted for compliance.
• •Conflating request ID, session ID, and trace ID, then searching with the wrong one across services.
• •Concluding the system is unobservable because one log store is empty, without checking whether the signal was routed elsewhere.
• •Comparing timestamps across services without accounting for documented clock skew or timezone normalization.

Safe fixes

• •Where a correlation identifier is missing at a boundary, add structured-field emission only after confirming the inbound header is read at request entry and forwarded unchanged downstream.
• •Where retries obscure the original request, preserve the parent correlation ID and add an attempt number as a separate structured field, conditional on the retry emitter being reachable and configured to log structured fields.
• •Where timestamps drift, enable a documented time source and record its offset in the log line, conditional on the operating environment supporting it.
• •Where fields are stripped at the collector, adjust the allowlist so the correlation and trace fields pass through, conditional on the routing configuration being under operator control.
• •Where identifiers are redacted, document the redaction scope and provide an authorized unredacted path before relying on cross-service search.

Prove the fix

1. 01A single user action produces a log line per involved service, each carrying the same correlation identifier as a structured, queryable field.
2. 02A simulated retry produces one parent correlation identifier with N distinct attempt fields, and all N lines are retrievable by the parent ID.
3. 03A query by correlation ID returns entries whose timestamps are ordered within the documented skew budget of the configured time source.
4. 04A previously unqueryable log line is now retrievable by at least two independent structured fields (for example, correlation ID and route) without free-text parsing.
5. 05A regression check confirms that redaction, sampling, and retention rules do not remove the correlation or trace fields introduced by the fix.

Prevention and next steps

• •Treat correlation identifier propagation as a contract between services, with documented field names and propagation rules at each boundary.
• •Require structured logging fields for any identifier used for incident attribution, and reject free-text concatenation in code review.
• •Periodically replay a known request and verify that the expected identifier fields appear in each downstream service log.
• •Document the redaction, sampling, and retention policy for identifier fields so on-call engineers know what is queryable during an incident.

Safe commands and checks

grep -n "correlation" <service-config-file>
grep -rn "traceparent" <source-directory>
awk 'NR==<line-number>{print}' <log-file>
jq 'keys' <log-line.json>
jq 'select(.correlation_id=="<id>")' <structured-log-file>
timedatectl show | grep -i 'ntp\|synchronized'