Reading Distributed Traces to Find Latency: A Field Guide

You've got a distributed system. You've got Jaeger, Zipkin, or some other tracing setup. You click on a slow trace and see a waterfall of spans. Now what?

Most engineers I've seen either stare at the waterfall until something stands out, or jump straight to the database query thinking that's always the problem. Both approaches waste time. Over the last few years debugging latency in production, I've developed a systematic approach to reading traces that cuts investigation time by at least half.

Yes, you should look at the span with the longest duration. But that's just the starting point. The real trick is understanding the critical path — the chain of spans that determine the total trace duration. In a trace with parallel calls, the critical path is the longest sequence of parent-child spans. Ignore spans that run concurrently with a longer sibling; they're not the bottleneck.

A common mistake is only looking at span durations. Often, the biggest delay is in the gaps — time not accounted for by any span. This happens when there's unmeasured work (e.g., waiting for a lock, kernel context switching, or a missing span around an HTTP call). I once spent two hours chasing a 500ms span that turned out to be a 50ms span with a 450ms gap caused by a DNS resolution that wasn't instrumented.

A single slow trace can be a fluke. To find systemic latency, compare traces from high-load periods with traces from low-load periods. Look for spans that grow disproportionately under load. For example, if database query duration doubles under load but network latency stays flat, the database is the bottleneck. If network latency increases, the problem might be in the client or server TCP stack.

When debugging latency in a service that calls another service, instrument both sides. I've seen cases where the client span shows 500ms, but the server span shows 200ms. The extra 300ms is client-side overhead — maybe serialization, connection pooling, or retries. Without server-side spans, you'd blame the downstream service.

Most tracing systems use head-based sampling (e.g., trace every 1 in 100 requests). That's fine for overall statistics, but it often misses rare slow traces. Use tail-based sampling to capture all traces whose duration exceeds a threshold (say 500ms). That way, you always have data on the worst cases. I configure Jaeger with a rate-limited sampler that captures all traces > p90 duration.

Over time, you'll learn to spot patterns in trace waterfalls. A "staircase" pattern (each span starts after the previous ends) indicates serial processing that could be parallelized. A "fan-out" with one long span suggests a bad timeout or retry storm. A "gap" after a short span means something is missing. I keep a set of annotated screenshots of common patterns — it helps new team members ramp up faster.

Distributed tracing gives you the data, but it takes practice to read it. Start with these techniques next time you see a slow trace. You'll find the root cause faster than chasing database queries blindly.