What this usually means
GC thrashing is a system-level failure where the garbage collector cannot keep up with the allocation rate. The heap fills up faster than the collector can free objects, so it runs continuously in a futile cycle. In JVM terms, this often means the application is allocating objects at a rate that exceeds the throughput of the collector, or the heap is too small for the live set. Common causes include excessive object creation in hot paths, oversized caches that force frequent collections, or a memory leak that gradually reduces the effective free space. The CPU is not doing useful work — it's spending all its time scanning and copying objects that are mostly reachable, only to free a tiny fraction of the heap each cycle.
The first ten minutes — establish facts before touching code.
- 1Run `top -H -p <PID>` on Linux (or `htop`) to identify threads consuming CPU. For Java: `jstack <PID> | grep -A 20 'GC thread'` to confirm GC threads are active.
- 2Enable GC logging immediately: for Java, add `-Xlog:gc*:file=gc.log:time,uptime,level,tags` (or `-verbose:gc` on older JVMs). For .NET, use `dotnet-counters monitor --process-id <PID> System.Runtime[gc-heap-size,gc-collections,gc-pause-time]`.
- 3Check heap usage with `jstat -gcutil <PID> 1s 10` for Java. Look at E (Eden) utilization — if it's near 100% after every young GC and you see constant Full GC, you have thrashing.
- 4Use `jmap -histo:live <PID> | head -20` (Java) or `dotnet-dump analyze <dump> > histo` (.NET) to get a histogram of live objects. Look for unexpected classes with many instances.
- 5For .NET, run `dotnet-gcdump collect -p <PID>` and open the report in Visual Studio or PerfView. Check the 'Large Object Heap' size and pinned object count.
- 6In Kubernetes, use `kubectl top pod` to see CPU and memory at container level, then `kubectl exec` into the pod and run the above commands.
The specific files, logs, configs, and dashboards that usually own this bug.
- searchGC logs (gc.log, gc.log.current) — look for pattern of back-to-back Full GC or concurrent cycles with <1s gaps
- searchHeap dump histograms via `jmap -histo:live` or `dotnet-dump analyze` — identify classes with millions of instances
- searchApplication performance monitoring dashboards (e.g., Datadog, New Relic) — check the 'GC Pause Time' and 'Allocation Rate' metrics
- searchThread dumps taken during high CPU — search for 'GC task thread' or 'Concurrent Mark-Sweep' threads running
- searchCode hot paths — review the top functions in CPU profiler output for object allocations using a profiler like async-profiler or PerfView
- searchJVM flags: `-XX:+PrintGCDetails`, `-XX:+PrintGCTimeStamps`, `-Xmx`, `-Xms`, `-XX:NewRatio` — verify heap sizing configuration
- searchOperating system metrics: /proc/<PID>/status (VmRSS, VmPeak), `sar -r` for swap usage — swap can exacerbate thrashing
Practical causes, not theory. These are the things you will actually find.
- warningExcessive object allocation in a hot loop (e.g., creating a new String or object per request in a high-throughput endpoint) without object pooling
- warningOversized in-memory cache that holds most of the heap, forcing frequent collections and leaving little free space for allocations
- warningMemory leak that gradually fills the heap, reducing effective capacity and forcing GC to run more often
- warningIncorrect heap sizing: heap too small for the live set, or `-Xms` and `-Xmx` set too far apart causing JVM to resize heap repeatedly
- warningUse of `System.gc()` or `GC.Collect()` explicitly triggered by application code or frameworks (e.g., RMI, JNDI) at high frequency
- warningPinned objects (in .NET) or direct buffers (in Java) that prevent compaction, causing fragmentation and triggering full GCs
Concrete fix directions. Pick the one that matches your root cause.
- buildReduce allocation rate: cache immutable objects, use object pools (e.g., Apache Commons Pool), rewrite hot paths to reuse instances, prefer primitives over boxed types
- buildIncrease heap size (if memory is available) to give GC more room. For Java, set `-Xms` equal to `-Xmx` to avoid resizing overhead.
- buildTune GC algorithm: switch from Serial/Parallel to G1GC or ZGC for lower pause times, tune `-XX:G1HeapRegionSize` and `-XX:InitiatingHeapOccupancyPercent` to start concurrent cycles earlier
- buildRemove or resize oversized caches: limit cache size using eviction policies (LRU, TTL) so it doesn't dominate the heap
- buildFix memory leaks: use heap dump analysis to find and fix the leak — common culprits are forgotten static collections, listener registrations, or ThreadLocal variables
- buildDisable explicit GC calls: use `-XX:+DisableExplicitGC` in Java or configure the runtime to ignore manual GC in .NET
- buildFor .NET, use `SERVER_GC=1` environment variable to enable server GC, which uses multiple GC threads and may reduce pause times
A fix you cannot prove is a guess. Close the loop.
- verifiedAfter fix, monitor GC logs for at least 30 minutes under production load — verify that Full GC frequency drops to <1 per hour or less, and pause times are within SLO
- verifiedObserve CPU usage: should return to baseline (e.g., 20-40% for a typical web app) with no more sawtooth spikes
- verifiedRun `jstat -gcutil <PID> 1s 10` (Java) or `dotnet-counters monitor` (.NET) — should see stable heap usage with minor GCs only, no sustained high utilization
- verifiedUse a load test with the same request rate as during the incident — response times should be flat and low, with no periodic latency spikes
- verifiedCheck application health endpoints: they should respond promptly without timeouts
- verifiedReview thread dumps: GC threads should be idle or running infrequently, not active every second
Things that make this bug worse or harder to find.
- warningDo not immediately add more memory without understanding the allocation rate — increasing heap without reducing allocation can make thrashing less frequent but more severe when it occurs
- warningDo not disable GC entirely — that will cause OutOfMemoryErrors or process termination
- warningDo not change GC algorithm without testing on a representative workload — each collector has different throughput and pause characteristics
- warningDo not ignore the application's allocation rate — a truly thrashing system may require code changes, not just tuning
- warningDo not rely on `System.gc()` for heap management — it often triggers Full GCs that worsen thrashing
- warningDo not assume the problem is only in your code — check third-party libraries and frameworks that might be creating objects or calling GC
The 2 AM GC Meltdown: How a 200MB Cache Brought Down Our API Gateway
Timeline
- 01:58PagerDuty alert: API Gateway CPU > 95% for 5 minutes
- 02:03Logged into pod with kubectl exec, ran `top -H` — saw GC threads at 85% CPU
- 02:05Checked GC logs: Full GC every 800ms, heap after GC ~6.5 GB out of 8 GB
- 02:08Ran `jstat -gcutil` — E (Eden) at 99%, O (Old) at 82%, YGCT and FGCT climbing fast
- 02:12Ran `jmap -histo:live` — top class was `com.example.cache.CachedResponse` with 2.5 million instances
- 02:15Found the cache configuration: in-memory map with no eviction, size set to 200MB but actually grew to 6GB due to large response objects
- 02:20Applied hotfix: added LRU eviction with max 1000 entries and TTL of 60 seconds; restarted pods gradually
- 02:30CPU dropped to 30%, GC pauses returned to <50ms every few seconds
I was on-call when the pages hit. The API gateway had been running fine for weeks, but suddenly CPU spiked to 95% and stayed there. My first thought was a traffic surge, but request rates were normal. I jumped into a pod and saw GC threads dominating. That's when I knew we had GC thrashing.
I checked the GC logs and saw Full GCs every 800 milliseconds. The heap was 8GB, and after each GC it was still at 6.5GB. That's insane — the collector was running constantly but barely freeing anything. I ran a live histogram and saw 2.5 million instances of CachedResponse. We had a cache for API responses, but it was supposed to be capped at 200MB. Someone had set the max size but the entries were large (some responses were 500KB), so the cache ballooned to 6GB, filling the old gen and forcing constant Full GCs.
The fix was straightforward: we changed the cache to use an LRU eviction policy with a maximum of 1000 entries and a 60-second TTL. We also set the heap to 12GB to give more breathing room. After restarting the pods, CPU dropped to 30% and GC pauses were back to normal. The lesson: always monitor actual cache size in production, and never trust a cache configuration without testing it with real data sizes.
Root cause
A memory cache with no effective size limit grew to consume most of the heap, forcing continuous Full GCs as the collector tried (and failed) to free memory.
The fix
Applied LRU eviction with max 1000 entries and 60-second TTL; increased heap from 8GB to 12GB as a buffer.
The lesson
Cache configurations must be validated with realistic data sizes and monitored in production. Never assume a 'max size' setting works when individual entries vary in size.
The first thing I do when CPU is high is check GC logs. For Java 11+, use `-Xlog:gc*:file=gc.log:time,uptime,level,tags`. For older JVMs, `-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps`. Look for lines that start with `[Full GC` or `[GC pause (G1 Evacuation Pause) ... (to-space exhausted)`. The key metric is the time between GC pauses. If you see pauses <1 second apart, you are thrashing.
Pay attention to the 'after GC' heap usage. If after a Full GC the heap is still 80%+ occupied, the live set is too large for the heap. Also look for 'to-space overflow' in G1 logs — that means the collector ran out of space to copy surviving objects, forcing a Full GC. This often happens when the heap is too small or when there are many long-lived objects.
In .NET, use EventCounters or dotnet-counters to watch `gc-heap-size`, `gc-collections`, and `gc-pause-time`. A high collection rate (e.g., >10 collections per second) with high pause time >100ms indicates thrashing.
GC thrashing is almost always caused by code that allocates too many objects. To find it, you need an allocation profiler. For Java, async-profiler is my go-to: `./profiler.sh -e alloc -d 60 -f alloc.svg <PID>`. This produces a flame graph of allocations. Look for wide frames at the top — those are the functions allocating the most memory.
Common culprits: boxing (Integer, Long) in hot loops, String concatenation with `+` (use StringBuilder), creating new objects per request instead of reusing them. In one case, I found a logging framework that was allocating a new `Map` for every log message. Switching to a static map reduced allocation rate by 90%.
For .NET, use PerfView with the 'GC Heap Alloc' option, or dotnet-trace with `Microsoft-DotNETRuntime` provider. Look for functions with high 'Allocations MB/sec'.
Once you've reduced allocation rate, tune the heap. The simplest fix: set `-Xms` equal to `-Xmx` to avoid resizing overhead. For G1GC, the default `-XX:InitiatingHeapOccupancyPercent=45` may start marking too early. I often raise it to 60 or even 70 for apps with high allocation rates, but be careful not to trigger to-space exhaustion.
Consider switching to a low-pause collector if you have large heaps. ZGC (Java 11+) or Shenandoah can handle heaps up to 16TB with sub-millisecond pauses. They trade CPU overhead for pause time, so they may not help if CPU is already saturated. In .NET, enable Server GC (`SERVER_GC=1`) for multi-core machines — it uses multiple GC threads and can reduce pause times.
Also check `-XX:NewRatio` and `-XX:SurvivorRatio`. A common mistake is making the young generation too small, forcing frequent minor GCs that promote objects prematurely. Monitor with `jstat -gcutil` — if the old generation grows quickly, you may need to increase the young generation size.
Some frameworks and libraries call `System.gc()` directly. RMI (Remote Method Invocation) does this periodically by default. In Java, you can disable it with `-XX:+DisableExplicitGC`. But be aware that some JVM tools (like JConsole's 'Perform GC' button) and NIO's direct buffer cleanup also rely on explicit GC. If you disable it, ensure those paths are handled otherwise.
In .NET, `GC.Collect()` can be called by third-party libraries. Use tools like `dotnet-trace` to find call stacks. You can override the behavior by setting `GCLatencyMode` to `LowLatency` or `SustainedLowLatency`, but this increases memory pressure.
I once spent hours debugging GC thrashing only to find that a monitoring agent was calling `System.gc()` every 30 seconds. Removing the agent fixed everything. Always check for external triggers.
A slow memory leak can cause GC thrashing after weeks of uptime. The heap gradually fills until the collector can't keep up. To detect leaks, compare heap dumps taken hours apart. Use `jmap -dump:live,format=b,file=heap1.hprof <PID>` and then again later. Analyze with Eclipse MAT or JProfiler. Look for classes whose instance count keeps growing.
Common leak patterns: static `HashMap` used as cache without eviction, ThreadLocal variables not cleaned up, listener registration without deregistration, or `ClassLoader` leaks in web apps.
In .NET, use `dotnet-dump analyze` to look for objects with increasing counts over time. Also check the Large Object Heap (LOH) — if it's growing, you likely have fragmentation or pinned objects. Defragmentation requires a full GC, which can trigger thrashing.
Frequently asked questions
What is the difference between GC thrashing and a normal high CPU situation?
Normal high CPU is caused by busy application threads doing work (e.g., processing requests, computing). GC thrashing shows CPU dominated by GC threads (often named 'GC task thread' or 'Concurrent Mark-Sweep') while application threads are blocked or waiting. A simple check: run `top -H` and look for thread names. If you see multiple GC threads at high CPU, it's thrashing.
Can increasing heap size always fix GC thrashing?
No. If the allocation rate is too high, a larger heap just delays the inevitable — the GC will still run frequently, and pause times will be longer. It's a band-aid. The correct approach is to reduce allocation rate first, then size the heap appropriately. Increasing heap without reducing allocation can make thrashing less frequent but more severe when it occurs (longer pauses).
How do I find which objects are causing the most allocations?
Use an allocation profiler. For Java, async-profiler with `-e alloc` gives a flame graph. For .NET, use PerfView or dotnet-trace with the GCHeapAlloc keyword. Look for the widest frames — those are the functions allocating the most memory. Common culprits: string concatenation in loops, boxing, and object creation in hot paths.
Should I use System.gc() to force garbage collection?
Never in a production application. Calling `System.gc()` triggers a Full GC, which pauses all threads and can cause thrashing. It's a code smell. If you think you need to force GC, you likely have a design flaw. Use `-XX:+DisableExplicitGC` to prevent accidental calls.
Is GC thrashing exclusive to Java?
No, .NET and other garbage-collected runtimes (e.g., Go, Ruby, Python) can also experience thrashing. The symptoms and diagnosis methods are similar: high CPU, frequent collections, and heap sawtooth patterns. The tools differ (dotnet-counters, dotnet-dump for .NET; pprof for Go) but the principles are the same.