Flame graphs are the default visualization for CPU profiling. You've probably seen them: stacked rectangles colored by function call, wider means more CPU time. But most engineers stop at "wider = slower" and miss the subtle patterns that tell you why your code is slow.
I've used flame graphs to debug everything from high-frequency trading systems to Node.js API servers. The difference between a novice read and an expert read isn't knowing what a flame graph is — it's recognizing the topologies that indicate real bottlenecks.
The Basics: What Each Rectangle Tells You
Every rectangle is a stack frame. The x-axis is not time — it's alphabetical order of function names by default, though some tools sort by frame width. The y-axis is stack depth: root at bottom, leaf at top. Width is proportional to CPU time (or samples) spent in that frame and its descendants.
A frame that is wide at the top means that function was on CPU a lot — but it could be because it calls many different functions. A frame that is wide at the bottom (closer to the root) means that function is called from many different code paths. The real hotspot is the widest frame that has no children (a leaf frame), or a frame whose children are all narrow and it itself is wide — that's likely a tight loop.
Pro tip: Always sort by width, not alphabetically. The x-axis order doesn't affect any metric, but sorting by width shows the heaviest call stacks first. In Brendan Gregg's FlameGraph tools, use `--sort=func` to sort by function name; I prefer `--sort=frame` for width-based sorting.
Pattern 1: The Plateau — Lock Contention and Serialization
A plateau is a wide, flat set of frames at the top of the graph, often with many siblings of similar width. This pattern screams lock contention or serialization bottleneck.
Imagine a web server handling requests. Each request calls `handle_request()` which locks a mutex, reads a cache, does work, unlocks. If the mutex is contended, you'll see a wide `pthread_mutex_lock` frame with many callers like `handle_request`, `parse_request`, `send_response`. The mutex itself might be only a few percent, but the plateau of callers all waiting on it tells the story.
# Generate a flame graph from perf data
$ perf record -F 99 -a -g -- sleep 30
$ perf script | ./stackcollapse-perf.pl > out.perf-folded
$ ./flamegraph.pl --sort=frame --width=1200 out.perf-folded > flame.svgIn a real incident at a fintech company, I saw a plateau of `__lll_lock_wait` with many sibling frames: `validate_order`, `check_balance`, `update_ledger`. The lock was a global mutex around a shared cache. The fix was sharding the cache by customer ID, which broke the plateau into multiple smaller peaks.
Pattern 2: The Mountain — Cache Misses and Branch Mispredictions
A mountain has a narrow peak (a single hot function) but a wide base (many different callers of that function). This pattern often indicates that the function itself is fast, but it's being called from everywhere, and the CPU is spending time on cache misses or branch mispredictions due to scattered call sites.
For example, a function like `hash_code` that is called from many different objects. The function itself is tiny, but each call site may have different data patterns causing L1 cache misses. The flame graph shows a narrow `hash_code` peak (because it runs quickly) but a wide base of callers (because many frames call it).
The mountain pattern is a red herring: the function isn't slow, but the calling pattern is causing CPU front-end stalls. Profile with `perf stat -e cache-misses,branch-misses` to confirm.
I once debugged a Redis latency issue where `dictFind` appeared as a narrow peak with dozens of callers. Each caller was a different command handler. The fix was to inline the hash lookup or use thread-local caches to reduce cache line bouncing.
Pattern 3: The Spire — Recursion and Hot Loops
A spire is a tall, thin column of repeated frames — same function name appearing at multiple depths. This is classic recursion or a tight loop with nested calls. The width at each level might be similar, indicating that the recursion depth is consistent.
But be careful: spires can also be tail calls optimized away or inlined functions. If you see a spire of `process_node` -> `process_node` -> `process_node`, check if it's really recursion or a loop that the compiler unrolled. The width tells you how many samples hit that depth.
// Example: recursive tree traversal that shows as a spire
function traverse(node) {
if (!node) return;
traverse(node.left);
process(node); // <-- this line is the leaf
traverse(node.right);
}If you see a spire with decreasing width at each level (wider at bottom, narrower at top), that's likely a recursive algorithm with a wide branching factor — like a quicksort partition. The bottom level is the root call, and the top is the leaf. This is normal for divide-and-conquer algorithms.
Pattern 4: The Alligator — I/O Waiting in Disguise
An alligator has a wide, flat top (like a plateau) but with a long, narrow tail of frames below. This often indicates that a thread is blocking on I/O, and the flame graph shows the call stack that did the blocking call. The wide top is the blocking function (e.g., `read`, `epoll_wait`), and the narrow tail is the code path that led there.
Standard CPU flame graphs won't show I/O wait because the thread is not on CPU. You need an "off-CPU" flame graph or a timeline visualization (flame chart) to see blocking. But if you see a wide `epoll_wait` in a CPU flame graph, it means the thread is spinning on the wait — likely a busy loop.
of production latency issues are I/O-bound, not CPU-bound — yet most teams only look at CPU flame graphs.
To catch I/O, I always generate both on-CPU and off-CPU flame graphs. For off-CPU, use `perf record -e sched:sched_switch` or `bpftrace` to trace context switches. The off-CPU flame graph shows where threads are sleeping, which is often the real bottleneck.
Real-World War Story: The 200ms Spike
200ms latency spike in payment processing
- 14:02Pager: p99 latency on payment service jumps from 50ms to 250ms.
- 14:05CPU flame graph shows a plateau of `pthread_mutex_lock` with callers `process_payment`, `validate_card`, `update_ledger`.
- 14:08Check lock: global mutex around a shared rate limiter. The mutex is held for ~1ms per call.
- 14:12Change rate limiter to per-tenant sharded counters, reduce critical section to 10µs.
- 14:20Deploy fix. p99 drops to 55ms.
Lesson
The plateau pattern directly pointed to lock contention. Without flame graphs, we might have blamed the database or the network. The width of `pthread_mutex_lock` alone was 2%, but the plateau of callers was 40% of samples.
Beyond CPU: Memory and Async Flame Graphs
Flame graphs aren't just for CPU. You can profile memory allocations: use `pprof --alloc_space`, or `valgrind --tool=massif` and convert to flame graph. The patterns are similar: wide frames indicate heavy allocation sites.
For async code (Node.js, Python asyncio, Go goroutines), the stack may not show I/O wait directly. Use async-aware profilers: Node.js's `--prof` with `--interpreted-frames-native-stack`, or Go's `pprof` with `-http` to see goroutine stacks. In async flame graphs, a wide top is often the event loop or scheduler.
One more tip: always look at the "delta" between two flame graphs (before/after a change). Brendan Gregg's `difffolded.pl` script highlights frames that increased or decreased. That's how you prove a fix worked.
# Compare two folded stacks and generate a differential flame graph
$ ./difffolded.pl out-before.perf-folded out-after.perf-folded | ./flamegraph.pl --negate > diff.svgCommon Mistakes When Reading Flame Graphs
- arrow_rightFocusing on the widest frame at the top — it's often just a caller with many children. Look at leaf frames.
- arrow_rightIgnoring the color scale: colors are usually random or by function name. They don't indicate severity.
- arrow_rightNot zooming in: use interactive SVG flame graphs to click and zoom into a frame. The top-level view hides details.
- arrow_rightForgetting sample rate: with low sample rates (like 49 Hz), rare events may not appear. Use 99 Hz or higher.
- arrow_rightComparing flame graphs from different workloads: always profile under the same request pattern.
Flame graphs are a tool, not a diagnosis. The patterns I described — plateau, mountain, spire, alligator — are heuristics, not rules. A plateau could also be a hot loop with many iterations, not necessarily a lock. Always corroborate with other metrics (CPU counters, latency histograms, source code).
The next time you open a flame graph, don't just look for the biggest rectangle. Look at the shape. The shape tells you the story.
Frequently asked questions
What is the difference between a flame graph and a flame chart?
A flame graph aggregates stack traces over time, showing total CPU time per frame. A flame chart shows individual samples over time, which helps visualize concurrency and latency patterns. Use flame graphs for CPU bottlenecks, flame charts for I/O or lock contention.
Why does my flame graph show a wide top frame but the code seems fast?
A wide top often means that function is a caller that invokes many different child functions. If those children are narrow, the parent's width is an artifact of aggregation. Focus on the widest leaf frames (the most bottom frames without children) — those are where CPU is actually spent.
How do I identify a lock contention hotspot in a flame graph?
Look for a wide, flat plateau near the top of the graph — a function like `pthread_mutex_lock` or `futex_wait` that appears as a sibling or parent of many frames. The width indicates time spent waiting, not doing work. Check for locks held across many calls.
Can flame graphs help with memory or GC issues?
Yes, but indirectly. CPU flame graphs won't show memory directly, but you can spot GC threads (e.g., `gc` or `sweep` frames) taking significant width. Use memory flame graphs (heap profiling) or a profiler like `pprof` with `--alloc_space` to see allocation hot spots.