What this usually means
In .NET, objects larger than 85,000 bytes go to the Large Object Heap (LOH). Unlike the Small Object Heap, the LOH is not compacted by default. Over time, the allocator leaves gaps (free space) between surviving objects, causing fragmentation. Even if total live memory is modest, the process commits more and more virtual memory to satisfy new large allocations because no single free block is large enough. This looks like a memory leak but is actually allocation pattern pathology. The GC cannot reduce the LOH size because it never compacts, and the OS sees the committed memory as used. This is especially common with buffers allocated per-request that survive a GC (e.g., byte arrays stored in a cache or ASP.NET data protection keys).
The first ten minutes — establish facts before touching code.
- 1Run `dotnet-counters monitor --process-id <PID> System.Runtime` and watch `gc-heap-size`, `gen-2-gc-count`, `loh-size`.
- 2Take a memory dump with `dotnet-dump collect -p <PID>` then load with `dotnet-dump analyze <dump>` and run `dumpheap -stat` to find large objects.
- 3Check LOH fragmentation: in the dump analyze, run `!dumpheap -stat` then `!gcroot <address_of_large_object>` to see if it's rooted.
- 4Use PerfMon counters: ".NET CLR Memory\Large Object Heap size", "# bytes in all heaps", and "% Time in GC" to correlate growth.
- 5Run `dotnet-gcdump collect -p <PID>` and open the .gcdump in Visual Studio or PerfView to visualize LOH fragmentation.
The specific files, logs, configs, and dashboards that usually own this bug.
- searchApplication code: all places that allocate byte[] or other objects > 85 KB (e.g., `ArrayPool<byte>`, `MemoryStream`, image loading, serialization).
- searchConfiguration: ASP.NET data protection key ring size, SignalR buffer settings, Kestrel limits (`MaxRequestBodySize`).
- searchLogs: watch for warnings about large allocations or high GC pressure.
- searchPerformance counters: `\.NET CLR Memory(*)\Large Object Heap size` and `\.NET CLR Memory(*)\# Gen 2 Collections`.
- searchSource code: review class fields that hold byte arrays or collections that could pin objects.
- searchGarbage Collector settings: check if `gcAllowVeryLargeObjects` is enabled or `GCSettings.LatencyMode` is set to `LowLatency` (prevents compacting).
- searchThird-party libraries: especially for image processing (ImageSharp, SkiaSharp) or large JSON (Newtonsoft.Json with large arrays).
Practical causes, not theory. These are the things you will actually find.
- warningAllocating large temporary buffers per request without pooling (e.g., `new byte[100000]` on every call).
- warningStoring large objects in a long-lived cache or static field that never gets cleared.
- warningPinning large objects via `fixed` statement or interop, preventing the GC from moving them (LOH compaction not default, but pinning prevents future compaction).
- warningUsing `BinaryFormatter` or other serializers that allocate large arrays internally.
- warningHigh-frequency allocation of arrays just below 85 KB (e.g., 84,999 bytes) that go to SOH and are promoted to Gen2, causing similar fragmentation.
- warningImproper use of `ArrayPool` or `MemoryPool` (returning arrays too late or not at all).
- warningASP.NET Core Data Protection key ring loading large XML into memory repeatedly.
Concrete fix directions. Pick the one that matches your root cause.
- buildUse `ArrayPool<byte>.Shared` to rent and return large byte arrays instead of allocating new ones each time.
- buildEnable LOH compaction via runtimeconfig.json: `"System.GC.LargeObjectHeapCompactionMode": 1` (or set `GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce` on each GC).
- buildReduce allocation size: chunk large data into smaller pieces (< 85 KB) to use the Small Object Heap.
- buildIf using ASP.NET Data Protection, set `keyRingSize` to a reasonable limit and preload keys.
- buildSwitch to `MemoryStream` with `GetBuffer()` and trim excess capacity or use `RecyclableMemoryStream` from Microsoft.IO.Redist.
- buildReview and fix any code that pins large objects for extended periods.
- buildImplement a custom object pool for your specific large object type if `ArrayPool` doesn't fit.
A fix you cannot prove is a guess. Close the loop.
- verifiedAfter fix, run load test and observe `loh-size` in `dotnet-counters` — it should stay flat after initial warmup.
- verifiedMonitor `# bytes in all heaps` vs private bytes — they should diverge less (private bytes should track heap bytes more closely).
- verifiedCheck `% Time in GC` drops to < 5% under load.
- verifiedTake a memory dump after the fix and confirm no large free regions in LOH with `!dumpheap -stat` and `!verifyheap`.
- verifiedUse `dotnet-gcdump` to compare before/after LOH fragmentation (free space should be minimal).
- verifiedRun for 24 hours in staging and confirm memory is stable.
Things that make this bug worse or harder to find.
- warningAssuming all memory leaks are rooted objects — check LOH fragmentation first if memory grows but heap size is stable.
- warningEnabling full LOH compaction permanently (set to CompactOnce per GC) — full compaction every GC hurts performance.
- warningBlindly increasing GC generation thresholds — this masks the problem and increases pause times.
- warningUsing `GC.Collect()` with `GCCollectionMode.Optimized` — it may not compact LOH.
- warningIgnoring pinned objects — they can prevent compaction even if enabled.
- warningNot verifying with production traffic — staging often has different allocation patterns.
The 4GB Image Processing Service
Timeline
- 09:00On-call alert: App Service memory at 85%, scaling triggered.
- 09:05Check Azure metrics: private bytes 7.2 GB, but .NET heap bytes only 1.5 GB.
- 09:10Take memory dump with `dotnet-dump collect -p 1234`.
- 09:15Analyze dump: `!dumpheap -stat` shows large free blocks in LOH (size field: Free).
- 09:20`!dumpheap -type Free` reveals hundreds of free regions totaling 5 GB.
- 09:30Review code: Image processing allocates `byte[]` for each image, stored in a response cache.
- 09:45Fix: Change cache to store file paths, not byte arrays. Use `ArrayPool` for temporary buffers.
- 10:00Deploy fix, monitor memory: private bytes stabilize at 2 GB.
I got paged at 9 AM because the image processing service was eating memory like candy. Azure was auto-scaling but the new instances were also growing. I checked the metrics: private bytes at 7.2 GB, but the .NET heap reported only 1.5 GB. That 5.7 GB gap was the first clue — something was taking memory outside the managed heap or fragmentation was severe.
I took a memory dump and loaded it into the debugger. Running `!dumpheap -stat` showed a huge amount of Free objects on the Large Object Heap. The LOH was fragmented: many byte arrays had been allocated and freed, but the free blocks were too small to satisfy new allocations. The GC was reserving more virtual memory from the OS, causing the private bytes to balloon.
The root cause was our image response cache: we were caching the processed image as a byte array in memory. Each image was ~500 KB, and the cache held thousands. Even when we evicted old entries, the arrays were collected but left holes. The fix was to store images on disk and cache only file paths. For temporary buffers during processing, I switched to `ArrayPool<byte>`. After deployment, memory dropped to 2 GB and stayed flat.
Root cause
LOH fragmentation caused by caching large byte arrays and allocating temporary buffers without pooling.
The fix
Replace in-memory byte array cache with file-based storage; use `ArrayPool<byte>` for temporary buffers; enable LOH compaction once per GC as a safety net.
The lesson
Not all memory leaks are rooted objects. Always check LOH fragmentation when private bytes exceed heap bytes. Pool large allocations.
The Large Object Heap is not compacted by default because compacting large objects is expensive (it requires copying memory). The GC uses a free list to manage LOH: a linked list of free segments. When you allocate a new large object, the GC scans the free list for a block large enough. If fragmentation is high, it may find no suitable block and extend the segment, committing more memory from the OS. This committed memory is never released until the process ends, even if the free blocks are empty.
Fragmentation occurs when objects of different sizes are allocated and freed. For example, allocating a 1 MB buffer, freeing it, then allocating a 2 MB buffer. The 1 MB free block is too small, so the allocator requests more memory. Over time, the LOH becomes a checkerboard of small free blocks.
Use `!dumpheap -stat` to see the size of Free objects. Look for a large total size of Free entries. Then run `!dumpheap -type Free` to see individual free blocks. If you see many blocks, fragmentation is likely. Run `!verifyheap` to check heap integrity. Use `!gcwhere <address>` to see which generation an object belongs to.
To find what caused the fragmentation, look at the allocation pattern: `!dumpheap -type System.Byte[] -stat` shows byte array sizes. If many arrays are just below 85 KB, they are on SOH but similar fragmentation can occur. For LOH, focus on sizes > 85 KB.
Collect a .NET GC dump with `dotnet-gcdump collect -p <PID>`. Open the .gcdump in PerfView or Visual Studio. In the 'Heap View', look at the 'Large Object Heap' tab. It shows free space as a percentage. A value > 10% indicates potential fragmentation. You can also see the 'Free List' entries.
PerfView's 'GC Heap Alloc' view can show allocation sizes over time. Look for patterns of large allocations and deallocations. If you see a sawtooth pattern in LOH size, fragmentation is likely.
In .NET Core 3.0+, you can request LOH compaction on the next blocking GC: `GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce`. This is a one-time request; after the compacting GC, it reverts to default. You can set it before a known high-traffic period. To enable permanently, add to runtimeconfig.json: `"System.GC.LargeObjectHeapCompactionMode": 1` but beware of performance impact.
Another option is to set `GCSettings.LatencyMode` to `GCLatencyMode.Batch` which triggers full blocking GCs that compact LOH (if compaction is enabled). However, this increases pause times. For most services, compacting once per GC is a good trade-off.
Frequently asked questions
How do I check LOH fragmentation without a memory dump?
Use `dotnet-counters monitor --process-id <PID> System.Runtime` and look for `loh-size`. Compare to `gc-heap-size`. If `loh-size` is a large portion of heap and private bytes are much higher, suspect fragmentation. Also use .NET GC event counters: `dotnet-trace` with `Microsoft-Windows-DotNETRuntime` provider and analyze with PerfView.
What is the 85 KB threshold and can I change it?
The 85 KB threshold is hardcoded in the .NET GC. Objects larger than 85,000 bytes go to LOH. You cannot change it. However, you can design your allocations to stay under 85 KB by chunking data into smaller pieces. This is not always possible, but consider using `ArrayPool` to reuse large arrays.
Does LOH compaction affect performance?
Yes, compacting LOH is expensive because it requires copying large blocks of memory. It increases GC pause time. That's why it's not default. Only enable it if you have fragmentation issues. Use `CompactOnce` to minimize impact.
Can pinned objects cause LOH fragmentation?
Pinning an object prevents the GC from moving it. If you pin a large object, it can cause fragmentation because the free list cannot merge adjacent free blocks. Pinning is necessary for interop, but avoid pinning large objects for long periods. If you must pin, use `GCHandle.Alloc` with `GCHandleType.Pinned` and free promptly.
Is LOH fragmentation the same as a memory leak?
Not exactly. A memory leak means objects are still referenced and cannot be collected. LOH fragmentation means objects are collectable but the memory is not reused efficiently. The symptom (growing memory) looks the same, but the fix is different. Check if live objects are the issue or fragmentation.