Docker no space left on device: separate image, layer, and volume pressure

The symptoms

• •The Docker daemon logs an ENOSPC or "no space left on device" error that names a specific path, most commonly /var/lib/docker/overlay2 or /var/lib/docker/volumes, when pulling an image, starting a container, or running an interactive build step.
• •Image pulls terminate with `failed to register layer` or `failed to prepare extract` at a percentage that stops advancing, while the host filesystem still reports free inodes and blocks on df -h, so the error is misread as a false positive.
• •Container starts or `docker exec` writes fail with ENOSPC even though earlier containers on the same image start successfully, indicating that the writable upper directory of overlay2 has been exhausted rather than the read-only lower layers.
• •Build steps inside BuildKit report ENOSPC against /var/lib/docker/buildkit/cache or against a named volume, distinguishing cache pressure from image and volume pressure.
• •Volume-backed databases, log aggregators, and writers fail with ENOSPC while the image that owns them is still under ten layers and `docker image ls` shows no obvious bloat, indicating volume quota or filesystem reachability rather than image-layer pressure.

Likely causes

• •Image pressure: accumulation of pulled images and dangling intermediate layers, where `docker image ls` shows repositories and tags that are no longer referenced by running or stopped containers but still consume storage in the image store.
• •Layer pressure: the overlay2 storage driver reports ENOSPC when the upper directory of a running container fills, often from a process writing logs, scratch files, or uncontrolled output inside the container rather than from the image itself.
• •Volume pressure: named volumes or bind mounts that live on a filesystem smaller than the path Docker expects, where the volume remains attached but cannot extend; the underlying device, not the image graph, has reached its threshold.
• •Build-cache pressure: BuildKit cache mounted into /var/lib/docker/buildkit or held by a custom cache backend, where cache entries from prior builds block new extract steps and trigger ENOSPC at the cache boundary, not the image boundary.
• •Thin-pool pressure with the legacy devicemapper driver, where the metadata and data thin devices have separate thresholds and one can be exhausted while the other is empty, producing ENOSPC even when the host filesystem has space.
• •Filesystem-level quotas (XFS project quotas, ext4 journal limits, tmpfs caps on /var/lib/docker/tmp) that restrict Docker's effective storage independent of the underlying disk's df output.

First ten minutes

1. 01Triage the source of the error by reading the daemon log line in full: identify which path it names, whether it is a /var/lib/docker subpath, a volume target, or a buildkit cache mount, and whether the message is ENOSPC, EINVAL against a missing device, or a generic "exit status 1" wrapper.
2. 02Read storage state with read-only commands that show per-category usage: `docker system df` first, because it reports REPOSITORY, IMAGE, CONTAINER, LOCAL VOLUMES, and BUILD CACHE columns separately; record which column reports RECLAIMABLE as zero versus high, because that labels the pressure point.
3. 03Inspect the filesystem that owns /var/lib/docker (or whatever rootdir is configured) with `df -h` and `df -i` to confirm whether blocks are exhausted, inodes are exhausted, or both; treating ENOSPC as a disk-full event when inodes are exhausted is a common mis-triage.
4. 04Decide the boundary the error names: overlay2 path → layer or image pressure; named volume path → volume pressure; buildkit cache path → build-cache pressure. Do not run `docker system prune` until that boundary is decided, because the command acts across boundaries and may delete evidence needed for the actual fix.
5. 05Capture current state with `docker image ls`, `docker volume ls`, and the size column from `docker ps -s` before any cleanup, so you can compare reclaim against actual freed bytes in the proof-of-fix step.

Evidence to collect

• •Daemon log entry with full path: the exact ENOSPC line from `journalctl -u docker` (or the configured log driver sink) showing the offending path, which is the single highest-value piece of evidence for separating image, layer, volume, and cache pressure.
• •Per-category storage state from `docker system df -v`, which reports total, active, size, and reclaimable bytes for images, containers, local volumes, and build cache; the boundary with reclaimable zero is the pressure boundary.
• •Filesystem state: output of `df -h` and `df -i` for the device that backs /var/lib/docker and for any volume target mount, including percentage used so you can correlate with daemon-reported pressure rather than rely on a single threshold.
• •Container upper-directory size: `du -sh /var/lib/docker/overlay2/<id>/diff` for the offending container, which distinguishes layer pressure (large upper) from image pressure (large diff in /var/lib/docker/overlay2 across many ids).
• •Image inventory: `docker image ls --format` listing of repository, tag, size, and created date for each tag, plus `docker image ls --filter dangling=true` to separate unused images from active ones, so reclaim decisions are scoped rather than blind.
• •Volume inventory: `docker volume ls` and `docker volume inspect` output for each named volume, including mountpoint, so a reclaim decision can be made on a per-volume basis when volume pressure is the labeled boundary.

Where to look

• •Daemon journal: `journalctl -u docker` for the configured container runtime, filtering on ENOSPC, "no space left", and the path printed in the error; the surrounding lines show which step (pull, extract, start, exec) failed and against which storage area.
• •Docker root directory: /var/lib/docker by default, with subdirectories that map directly to the five pressure boundaries — image/ at image pressure, overlay2/ at layer pressure, volumes/ at volume pressure, buildkit/ at cache pressure, and tmp/ at extract-cache pressure.
• •Filesystem boundaries: the device that backs /var/lib/docker (often /dev/sdaN on a cloud image, /dev/xvdN on EC2 instances, or /dev/vdaN on other virtual disks) plus any separate filesystem mounted at a volume target; ENOSPC reported on the device but not the host df usually means a thin pool or quota, not the host disk.
• •Storage driver layer: /var/lib/docker/overlay2 (or the active driver's equivalent path) holds one subdirectory per image layer and per container upper directory; a container's upper dir fills independently of its image, which is why ENOSPC can appear mid-run on an image that started clean.
• •BuildKit cache boundary: /var/lib/docker/buildkit/cache.db and the cache subdirectories of the BuildKit root; ENOSPC here is reported as build failure, not image pull failure, and is the boundary for `--cache-from` or remote cache misconfiguration.
• •Quotas and project ids: for XFS-based hosts, `xfs_quota -c report` shows per-project usage on /var/lib/docker; ENOSPC with low df can mean Docker's project hit its quota before the disk did.

Diagnostic steps

1. 01Read the daemon error line and extract the path it prints; match that path against the pressure-boundary list (image, layer, volume, build cache, tmp) before reading storage state, so the diagnostic is anchored to the failing layer rather than to the topmost reported container.
2. 02Run `docker system df -v` and read the RECLAIMABLE column for each row; if reclaim is large in the IMAGE row but small elsewhere, image pressure is the dominant boundary even if the failing step ran inside a container.
3. 03Run `df -h` and `df -i` on the device that owns /var/lib/docker and on each volume mountpoint; classify the failure as block exhaustion, inode exhaustion, or thin-pool exhaustion — each has different fixes and different destructive-risk profiles.
4. 04Inspect the container that raised the error with `docker inspect <container-id>` to read GraphDriver.Data.UpperDir, RootDir, and DeviceId; a non-empty UpperDir at high utilization indicates layer pressure from process writes inside the container, even when the image is small.
5. 05For BuildKit failures, capture the build flags actually in use: `docker builder ls`, the active builder instance, and `--cache-from` or `--mount=type=cache` mounts that redirect to a separate path; an ENOSPC at a cache mount is not an image-space problem.
6. 06Cross-check with `docker image ls --filter dangling=true` and `docker image ls --format` to see whether unused images exist; presence of unused images plus large reclaim in the IMAGE row corroborates image pressure over layer or volume pressure.
7. 07For volume pressure, run `docker volume inspect` on each named volume to read Mountpoint, then `df -h` on the device backing that mountpoint; ENOSPC tied to a specific mountpoint is volume pressure even if /var/lib/docker has free space.
8. 08Decide one boundary as the cause before any cleanup command, because `docker image prune`, `docker volume prune`, `docker container prune`, `docker builder prune`, and `docker system prune` act on different boundaries and produce different retention guarantees.

Common mistakes

• •Running `docker system prune -a --volumes` as the first response: this command crosses every pressure boundary and removes named volumes that may be the only persistent store for stateful services, so it should not be used until the pressure boundary has been identified and the volume impact has been verified against ownership.
• •Treating ENOSPC as a host disk-full condition and resizing a volume that is not the boundary: the error names /var/lib/docker but the host disk may have free space because Docker is backed by a thin pool, a separate mount, or a quota, so a resize that does not target the boundary will not clear the error.
• •Reading `df -h` alone and missing inode exhaustion: small files such as container logs and metadata shards can fill an inode table while blocks remain free, and the fix (reclaim files, not blocks) is invisible to df-only checks.
• •Pruning images while a long-running build is in progress: another writer to the image store will silently re-fill the space that was just reclaimed, and the ENOSPC recurs without explanation because the layer-pressure component was not addressed.
• •Re-creating a container to "fix" overlay2 upperdir pressure: starting a fresh container does not move data out of the upper dir, so the underlying writer (a runaway log, an unbounded buffer) continues to fill the new upper dir and the error returns within minutes.
• •Assuming devicemapper because of a thin-pool message: with the default overlay2 driver a thin-pool message usually means the host's device-mapper thin pool, not Docker's, and the fix belongs to the kernel/storage layer rather than to Docker commands.

Safe fixes

• •If the boundary is image pressure and dangling images exist, run `docker image prune` first to reclaim only images without a tag and not referenced by any container; verify reclaim against the IMAGE row of `docker system df` after the command, and confirm in `docker ps -a` that no stopped container retains state the prune could invalidate.
• •If the boundary is layer pressure on a single container's upper directory, stop the offending container, inspect with `docker inspect` to confirm UpperDir is the culprit, then prune only stopped containers with `docker container prune`; do not delete the image or its layers until you have ruled out image pressure with `docker system df`.
• •If the boundary is volume pressure on a named volume, identify the mountpoint via `docker volume inspect <name>`, then clean inside the volume using the volume's own service (database truncate, log rotation) rather than `docker volume rm`, because the volume is typically the durable store for that service.
• •If the boundary is build-cache pressure, run `docker builder prune` with `--filter until=<duration>` to retain recent cache while reclaiming old entries, and confirm reclaim against the BUILD CACHE row of `docker system df`; this is safer than `docker system prune` because it leaves images and volumes untouched.
• •If the boundary is filesystem quota (XFS project quota, ext4 journal, tmpfs), the fix is at the filesystem layer: lower the project's reported usage by reclaiming files inside Docker's directories rather than by expanding the disk, which would mask the quota without addressing it.
• •If the boundary is the storage driver's thin pool (devicemapper or LVM thin on the host), the fix is to extend the thin pool's data or metadata LV from the host, not from Docker; Docker will see the additional space on its next operation without requiring a daemon restart in most configurations.

Prove the fix

1. 01Replay the failing operation (image pull, container start, build step) end-to-end and observe completion without an ENOSPC line in the daemon journal during the operation, so the bound that produced the error is verified to be addressed rather than coincidentally cleared.
2. 02Compare `docker system df -v` before and after the fix and confirm that the reclaim on the targeted boundary (images, containers, volumes, or build cache) changed by approximately the size observed in `du -sh` of the freed area, so the reclaim is attributed to the right boundary rather than to a coincidental prune.
3. 03Confirm `df -h` on the device backing /var/lib/docker reports headroom sufficient to absorb at least one further image pull plus one further container start, so a regression is observable before it reaches an ENOSPC rather than after; record the percentage used as a baseline.
4. 04For layer-pressure fixes inside a container, observe the upper dir's size remain bounded across a representative workload, with `du -sh /var/lib/docker/overlay2/<id>/diff` repeated at intervals, so the underlying writer is verified rather than assumed fixed.
5. 05For volume-pressure fixes, observe the volume's reported usage and the underlying device's df both remain bounded across the service's normal workload, with `docker system df` for the VOLUME row returning to a stable reclaimable value after the fix is exercised.
6. 06Confirm no commands targeting other boundaries were issued during the fix window, because a coincidental `docker system prune` will produce a green proof without addressing the named boundary; record the command sequence so the proof is attributable to one boundary only.

Prevention and next steps

• •Schedule a periodic `docker system df -v` capture (for example daily, via a script that writes the report to a log) and a recurring `docker image prune --filter until=720h` plus `docker builder prune --filter until=720h`, so reclaimable storage is reduced on a steady cadence rather than only after an ENOSPC surfaces.
• •Cap per-container upper-directory growth by configuring log drivers with rotation (`max-size`, `max-file`) and by setting process-level limits on known writers (databases, log shippers), because layer pressure is most often caused by an in-container writer rather than by the image itself.
• •Keep named volumes on a filesystem that is monitored independently of /var/lib/docker, so volume pressure is caught before it crosses into the daemon's error path; alert on percentage used, not only on bytes free, because small volumes fail fast.
• •For images that are re-tagged and re-pulled frequently, prefer a registry-mirrored pipeline that supports digest-pinned pulls over `latest` tags, so the image store does not accumulate intermediate tags whose reclaim grows without bound.
• •Decide the storage driver's thin pool size (where applicable) so that the data and metadata devices have a known ratio rather than the defaults, so future ENOSPC messages are not surprises against a thin pool rather than the host disk.

Safe commands and checks

docker system df -v
df -h /var/lib/docker && df -i /var/lib/docker
docker image ls --filter dangling=true
docker image ls --format 'table {{.Repository}}\t{{.Tag}}\t{{.Size}}\t{{.CreatedSince}}'
docker volume ls && docker volume inspect <volume-name>
docker inspect <container-id>
du -sh /var/lib/docker/overlay2/<id>/diff
journalctl -u docker --since '1 hour ago' | grep -i ENOSPC