Memory leaks are the silent killers of long-running services. They don't crash your process instantly — they degrade it over days or weeks until ops gets paged at 3 AM because the pod is OOM-killed. I've traced leaks from Node.js microservices, Java monoliths, and even C++ game servers. In every case, the fix was the same: a systematic, repeatable process.
This article walks through my go-to method for finding memory leaks. It's not magic — it's a sequence of steps: observe, snapshot, diff, analyze, fix, verify. I'll include a real war story from a production Node.js service that leaked 200 MB/hour.
Step 1: Confirm You Have a Leak (Not Just High Memory Usage)
High memory usage is not a leak. A service that processes large requests has a legitimate need for memory — it should release it after the request ends. A leak is when memory grows monotonically over time, even during idle periods.
Before diving into heap dumps, check your metrics. I use Prometheus with a histogram of GC activity. If the resident set size (RSS) or heap grows after each full GC, you have a leak. In Node.js, look at `process.memoryUsage().heapUsed` — if it never drops back to baseline after a GC, you're leaking.
Don't confuse a memory leak with a memory bloat from traffic spikes. A leak grows linearly with time, not with request rate. Plot heap size vs. time and look for a trend.
A War Story: The 200 MB/hour Leak
The Case of the Growing EventEmitter
- Day 1Deploy new recommendation engine. Memory grows 200 MB/hour. OOM after 6 hours.
- Day 2Restart every 4 hours as a band-aid. Heap dumps show millions of callback objects.
- Day 3Diff snapshots: each request adds one callback that never gets removed. Root cause: an EventEmitter listener added inside a request handler but never cleaned up.
- Day 4Fix: use once() or removeListener() after completion. Memory stabilizes.
Lesson
Event listeners attached per-request that are not removed are a classic Node.js leak. Always clean up listeners when you're done.
Step 2: Collect Heap Snapshots at Two Points in Time
A single heap dump tells you what's in memory now. It doesn't tell you what's growing. You need two snapshots taken at different times — ideally after a period of steady state activity (e.g., 1000 requests). Then diff them.
For Node.js, I use Chrome DevTools: take a heap snapshot, run traffic, take another, then switch to 'Comparison' view. For Java, use Eclipse Memory Analyzer (MAT) to open two .hprof files and run the 'Compare' query. For C++, valgrind massif can produce snapshots at different points.
# Node.js: trigger heap snapshot via USR2 signal
kill -USR2 <pid>
# waits for a snapshot file in /tmp/heap-*.heapsnapshot
# Then run load test, then trigger another snapshot
kill -USR2 <pid>
# Now you have two files: heap-1.heapsnapshot and heap-2.heapsnapshotWhat to Look For in the Diff
- 1Look for objects whose count increased dramatically between snapshots. Ignore objects that are expected to grow (e.g., LRU caches that have a max size).
- 2Focus on the 'Delta' count — how many more instances exist. A diff of +500,000 Strings could be a leak, but check if they are interned or duplicates.
- 3Trace the retaining path: which object holds the reference to the leaked objects? Often it's a single array, map, or closure scope.
Step 3: Analyze the Retaining Path
Once you spot a class or object type growing, click on it and look at the 'Retainers' or 'GC Roots' path. This shows you who holds a reference to that object, preventing GC from collecting it.
Common patterns: a static HashMap that grows unbounded (Java), a global variable that accumulates event listeners (Node.js), or a closure that captures a variable from an outer function (any JS). In C++, it's usually a std::vector or std::map that never clears.
In Chrome DevTools, select an object and use 'Store as global variable' then inspect its properties in the console. This lets you programmatically walk the tree.
Example: Finding a Closure Leak in Node.js
// Leaky version: the callback closes over 'largeData'
function processRequest(data, cb) {
const largeData = fetchFromDB(); // 10 MB object
someAsyncOperation(() => {
// this callback holds a reference to largeData
cb(largeData);
});
// largeData never gets released until callback fires
}
// Fixed: null out largeData after use
function processRequest(data, cb) {
const largeData = fetchFromDB();
someAsyncOperation(() => {
cb(largeData);
largeData = null; // allow GC to collect
});
}Step 4: Use Automated Tools for C/C++ Leaks
In languages without GC, leaks are harder to spot. You can't just take a heap dump. Instead, use runtime tooling: valgrind memcheck, AddressSanitizer (ASan), or LeakSanitizer (LSan). These tools intercept allocations and track whether they are freed.
I run unit tests under valgrind with `--leak-check=full`. For integration tests, I use ASan with `LSAN_OPTIONS=report_objects=1`. The output tells you exactly which allocation didn't get freed, including the stack trace of the allocation.
# Run valgrind on a test binary
valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes ./my_test
# Or use AddressSanitizer (compile with -fsanitize=address,leak)
clang++ -fsanitize=address,leak -g -o my_test test.cpp
./my_test
# LSan reports leaks at exit with stack tracesStep 5: Verify the Fix
After applying a fix, repeat the same test: run the same load and check if memory growth disappears. But beware: sometimes the fix only hides the symptom. For example, if you reduce the leak rate from 200 MB/hour to 10 MB/hour, that's still a leak — just slower. The goal is zero growth over time.
I always set up a canary deployment with the fix and monitor heap metrics for at least 24 hours. If the heap stays flat (with periodic GC drops), the fix is solid.
Recommended monitoring period after a memory leak fix to confirm zero growth
Preventing Future Leaks
Once you've fixed a leak, add a regression test. For Node.js, I write a test that creates a scenario, forces GC with `global.gc()`, and checks that memory returns to baseline. For Java, use a unit test that calls `System.gc()` and checks heap usage with JMX. For C++, run the test under valgrind as part of CI.
Also, set up production alerts on heap growth rate. If the heap grows at > X MB per hour for Y hours, page someone. This catches regressions before they cause an outage.
A memory leak is a bug that grows over time. Treat it like any other bug — reproduce, isolate, fix, and add a regression test.
Summary
- arrow_rightConfirm growth over time, not just high usage.
- arrow_rightTake two heap snapshots and diff them.
- arrow_rightTrace the retaining path to the root cause.
- arrow_rightUse valgrind or ASan for C/C++.
- arrow_rightVerify with monitoring over 24 hours.
- arrow_rightAdd regression tests to prevent recurrence.
Frequently asked questions
How do I take a heap dump without killing my application?
For Java, use `jmap -dump:live,format=b,file=heap.hprof <pid>`. For Node.js, use `kill -USR2 <pid>` or the inspector protocol. Both create a snapshot without stopping the process.
What is the most common cause of memory leaks in Node.js?
Closures that accidentally capture large objects, and unclosed timers or event listeners that keep references. Also, global caches that never clear.
How can I detect a memory leak in production before it causes an outage?
Monitor heap usage over time with a tool like Prometheus + Grafana. If the heap grows monotonically after each GC cycle, you likely have a leak. Set alerts on heap growth rate.
Is it possible to have a memory leak in a garbage-collected language?
Yes. Any reference that is unintentionally kept prevents GC from collecting the object. This includes forgotten collections, caches, listeners, and thread-local storage.