What this usually means
The memory allocation fails because the Python process cannot get a contiguous block of virtual memory large enough for the array. This is not just about total physical RAM: it's about address space fragmentation, per-process memory limits (ulimit), swap exhaustion, or the operating system's overcommit policy. For very large arrays (>>10 GB), the dtype matters: float64 arrays are 8 bytes per element, so a 10,000x10,000 matrix is 800 MB, but a 100,000x100,000 is 80 GB. Often, the culprit is using a high-precision dtype unnecessarily, or the array shape calculation is wrong (e.g., reshaping to an unintended huge dimension). Another common cause: the system's vm.overcommit_memory is set to 2 (strict overcommit), so the kernel denies allocations that would exceed physical RAM+swap.
The first ten minutes — establish facts before touching code.
- 1Run free -h or cat /proc/meminfo to check total RAM and swap availability
- 2Check ulimit -a for 'max memory size' (unlimited is ideal) and 'address space limit'
- 3Print the exact dtype and shape: print(f'Shape: {shape}, dtype: {dtype}, bytes: {np.prod(shape) * np.dtype(dtype).itemsize}')
- 4Test allocation with a smaller shape to confirm shape calculation
- 5Check vm.overcommit_memory: cat /proc/sys/vm/overcommit_memory (0=heuristic,1=always,2=strict)
- 6Run with memory_profiler: python -m memory_profiler your_script.py to see peak usage
The specific files, logs, configs, and dashboards that usually own this bug.
- search/proc/meminfo for MemTotal, MemAvailable, SwapTotal, SwapFree
- search/proc/sys/vm/overcommit_memory and /proc/sys/vm/overcommit_ratio
- searchulimit -a output (especially 'max memory size' and 'address space limit')
- searchThe exact line in code where np.array, np.zeros, or np.empty is called
- searchThe dtype and shape variables just before allocation
- searchIf using Docker or container: check the container's memory limit (docker stats)
- searchIf on a cluster: check scheduler memory limits (e.g., SLURM --mem)
Practical causes, not theory. These are the things you will actually find.
- warningUnintended huge shape due to integer overflow or wrong dimension (e.g., reshaping to (100000, 100000) instead of (1000, 1000))
- warningUsing float64 when float32 or int8 would suffice, doubling or 8x memory
- warningvm.overcommit_memory=2 with insufficient swap or overcommit_ratio too low
- warningProcess address space limit (ulimit -v) set too low, especially in HPC environments
- warningMemory fragmentation after repeated allocations/deallocations, preventing contiguous block
- warningRunning inside a container or VM with less memory than expected
- warningSystem has memory but it's reserved by hugepages or other kernel allocations
Concrete fix directions. Pick the one that matches your root cause.
- buildDowncast dtype: use np.float32 or np.int16 if precision allows
- buildUse memory-mapped arrays (np.memmap) to store array on disk
- buildIncrease swap space: sudo fallocate -l 20G /swapfile && sudo mkswap /swapfile && sudo swapon /swapfile
- buildChange overcommit policy: sudo sysctl vm.overcommit_memory=1 (or adjust ratio)
- buildRemove ulimit restrictions: ulimit -v unlimited (or increase in shell config)
- buildProcess data in chunks using np.lib.stride_tricks or dask.array
A fix you cannot prove is a guess. Close the loop.
- verifiedRun the script after fix and confirm no MemoryError
- verifiedMonitor memory usage with htop or free -h while the script runs
- verifiedCheck /proc/meminfo to see that allocation succeeded (available memory decreases as expected)
- verifiedUse np.shares_memory() to verify no unnecessary copies
- verifiedBenchmark with time to ensure performance is acceptable (memmap may be slower)
Things that make this bug worse or harder to find.
- warningBlindly increasing swap without checking if the array size is actually needed
- warningSetting vm.overcommit_memory=1 on a production server without understanding security implications (can cause OOM killer to kill processes)
- warningUsing np.memmap for arrays that are accessed randomly (performance disaster)
- warningForgetting to close memmap files, leading to file handle leaks
- warningAssuming ulimit is the same across all shells; always check in the script's environment
80GB Array Crashing a 64GB Machine
Timeline
- 09:00Deploy model training script to production cluster. Container has 48GB memory limit.
- 09:05Script fails with MemoryError: Unable to allocate 80.0 GiB for an array with shape (100000, 100000) and data type float64
- 09:10Check container stats: memory usage at 2GB before allocation, so not out of memory.
- 09:15Print shape and dtype: shape (100000, 100000) is intentional? Actually, input data has 100000 samples, each with 1000 features, so shape should be (100000, 1000).
- 09:20Find bug: feature dimension expanded during preprocessing due to one-hot encoding with 100 categories, but code used `np.eye(100)[labels]` which produces shape (100000, 100, 100) and then reshaped incorrectly.
- 09:25Fix reshape to (100000, 100000) was wrong; should be (100000, 100).
- 09:30Rerun with corrected shape: allocation now 800 MB, script completes successfully.
I was deploying a model training pipeline to a Kubernetes cluster. The container had a 48GB memory limit, which should be plenty for our dataset. But the script crashed immediately with a MemoryError asking for 80 GiB. That's more than the container limit and even more than the host's 64GB RAM.
First, I confirmed the container memory limit and free memory with `cat /sys/fs/cgroup/memory/memory.limit_in_bytes`. It showed 48GB. Then I added a print statement before the failing allocation: `print(shape, dtype)`. It printed `(100000, 100000) float64`. That's 100,000 x 100,000 = 10^10 elements, times 8 bytes = 80GB. That's huge. Our dataset only had 100,000 samples with 1000 features each, so the expected shape was (100000, 1000).
I traced back the code. The feature engineering step used one-hot encoding on a categorical variable with 100 unique values. The code was `np.eye(100)[labels]` which gave shape (100000, 100, 100). Then a reshape to (100000, 100000) was intended to flatten, but the math was wrong: 100*100=10000, not 100000. It should have been `reshape(100000, -1)` which gives (100000, 10000). The extra zero was a typo. Fixing that reduced the allocation to 800MB, and the pipeline ran fine.
Root cause
Typo in reshape dimensions: wrote (100000, 100000) instead of (100000, 10000), resulting in a 10x larger array than needed.
The fix
Corrected the reshape to `reshape(100000, -1)` which automatically computes the correct second dimension (10000).
The lesson
Always validate array shapes before large allocations. A quick print can save hours. Also, use `-1` in reshape to avoid manual dimension calculation errors.
NumPy arrays require contiguous blocks of virtual memory. When you call np.zeros(shape, dtype), NumPy asks the operating system for a single block of size prod(shape) * dtype.itemsize. If the OS cannot provide that (due to fragmentation, limits, or overcommit policy), a MemoryError is raised.
The Linux kernel uses a heuristic overcommit policy (vm.overcommit_memory=0) by default, which allows allocations larger than physical RAM+swap but may later kill processes with OOM if they actually use it. Setting it to 1 always allows overcommit, while 2 disables overcommit and uses a percentage (overcommit_ratio) of RAM+swap as the limit.
If you see MemoryError despite having enough free RAM, check `cat /proc/sys/vm/overcommit_memory`. If it's 2, the kernel is strict. Check `/proc/sys/vm/overcommit_ratio` (default 50). So with 64GB RAM and 0 swap, the max allocation is 32GB. If you need 40GB, you'll get MemoryError even if 40GB is free. The fix: increase overcommit_ratio, add swap, or set overcommit_memory to 1.
Also check `ulimit -v` (virtual memory limit). If set, it caps the total virtual memory a process can allocate. In HPC environments, job schedulers often set this. Run `ulimit -a` in the script's environment to verify.
Even if total free memory is sufficient, the kernel may not have a contiguous virtual address range of the required size. This is more common on 32-bit systems (limited address space) but can happen on 64-bit with very large allocations (>> 100GB) after repeated mmap/munmap. Python's memory allocator (glibc) can also cause fragmentation.
Workaround: Allocate the array early in the script before fragmentation. Or use np.memmap to map a file, which doesn't require contiguous RAM. For very large arrays, consider using dask.array which splits into chunks.
The dtype directly determines memory footprint. float64 uses 8 bytes per element, float32 uses 4, int8 uses 1. If you can tolerate lower precision, downcasting can reduce memory by 2x-8x. But beware of overflow: int8 max is 127, so if your data exceeds that, use int16 or int32.
Integer overflow can also cause shape calculation errors. For example, `np.prod(shape)` with shape (100000, 100000) returns 10000000000, which fits in int64 but if the code uses int32 somewhere, it could overflow and produce a smaller size, leading to a crash later. Always use Python ints or NumPy's int64 for shape calculations.
In Docker containers, the cgroup memory limit restricts the container's total memory. But the host's overcommit settings still apply inside the container. If the container has a 4GB limit but the host overcommit allows 8GB, the container can allocate up to 4GB (cgroup limit) plus whatever is in its swap? Actually, cgroup memory limit includes swap by default. Check `docker stats` and `/sys/fs/cgroup/memory/memory.limit_in_bytes` inside the container.
On cloud VMs with balloon drivers, available memory can change dynamically. Use `free -h` at the time of error. Also, some cloud instances have memory capacity that is not all available due to reserved memory for the hypervisor. For example, an AWS EC2 r5.large has 16GB RAM but the kernel might reserve 1GB. Always check MemAvailable in /proc/meminfo, not MemTotal.
Frequently asked questions
Why does np.zeros((10000, 10000), dtype=float64) fail with MemoryError when I have 8GB free?
The array size is 10000*10000*8 = 800MB, which is well within 8GB. The error might be due to: (1) ulimit -v set below 800MB, (2) vm.overcommit_memory=2 with overcommit_ratio too low, (3) memory fragmentation preventing a contiguous 800MB block. Run `ulimit -a`, check /proc/meminfo for MemAvailable, and try a smaller allocation to see if it works.
Will using np.memmap fix all MemoryErrors?
No. np.memmap maps a file to virtual memory, so the allocation is limited by disk space and virtual address space, not RAM. But if the entire array is accessed sequentially, it's fine. Random access will be slow because of disk seeks. Also, if the file is on a network filesystem, latency can be high. For many ML workloads, dask.array or h5py are better choices.
How do I check if my system has memory overcommit enabled?
Run `cat /proc/sys/vm/overcommit_memory`. Output 0 = heuristic, 1 = always, 2 = strict. For strict mode, also check `/proc/sys/vm/overcommit_ratio` which is the percentage of (RAM+swap) that can be used. For example, with 64GB RAM, 64GB swap, ratio 50, max allocation = (64+64)*0.5 = 64GB.
I get MemoryError even though the array size is only 500MB. What else could be wrong?
500MB is small; the issue is likely an address space limit. On 32-bit Python, max virtual memory is ~3GB, and other libraries (e.g., TensorFlow) may already be using large chunks. On 64-bit, check `ulimit -v`. Also, if you're in a Jupyter notebook, kernel may have memory limits. Try running the allocation in a fresh Python process.
Is there a way to prevent MemoryError in production code?
Yes. Always validate array shapes before allocation: if the computed size exceeds a threshold, raise a custom exception. Use `try-except MemoryError` to catch it and fall back to a chunked processing mode. Also, use `numpy.errstate` to treat allocation errors gracefully. For critical paths, allocate memory-mapped files from the start.