Docker volume checklist

The symptoms

• •Files written inside the container during a session are gone after `docker stop` and `docker start`, indicating the data landed on the container's writable layer rather than a mounted volume.
• •A directory mounted via `-v /host/path:/container/path` appears empty inside the container, while the host path clearly contains files, suggesting a path mismatch, mount nesting, or permission filter.
• •`docker volume ls` shows unexpected volumes accumulating, or the expected named volume is missing, pointing to anonymous volume creation due to a syntax mismatch in the `-v` flag.
• •Container logs report "Permission denied" when writing to a path that exists on the host, indicating a UID/GID mismatch between the host directory owner and the container process user.
• •Database or stateful service reports "data directory has wrong ownership" on first start, characteristic of an empty named volume being created with root ownership while the service runs as an unprivileged user.
• •`docker inspect --type volume <name>` shows `Mountpoint` pointing under `/var/lib/docker/volumes/...` but the host path is on a different filesystem, indicating driver or storage driver mismatch rather than a missing volume.

Likely causes

• •Anonymous volume auto-created because the `-v` flag used a host path that did not exist at `docker run` time, per the run reference semantics for unspecified volumes.
• •Bind-mount path shadowing: a named volume or anonymous volume defined later in the run arguments overrides the intended bind mount, or an image-defined `VOLUME` instruction causes an anonymous mount that hides the bind source.
• •UID/GID mismatch: the process inside the container runs as a non-root user (e.g., UID 1000) but the bind-mounted host directory is owned by root, so writes fail silently or noisily depending on the application.
• •Empty named volume receiving default ownership from the storage driver (typically root) while the application expects a specific user to own the data directory, causing init scripts to refuse startup.
• •Confusion between `--mount` and `-v` flag syntax: typos in key=value pairs (e.g., missing `target=` or `source=`) silently produce anonymous volumes instead of failing loudly.
• •Tmpfs or overlay coverage inside the container hides the underlying mount under a populated directory, so writes appear to succeed but land on the overlay rather than the volume.
• •Docker Desktop or remote context pointing to a different daemon than expected, so `docker volume ls` queries one host's volumes while the container runs on another.

First ten minutes

1. 01Reproduce deterministically: run `docker ps -a` and identify the failing container's ID, image, and the exact `docker run` command or compose service that started it, capturing the run reference fields used.
2. 02Enumerate all volumes visible to the current Docker context with `docker volume ls` and cross-reference against the container's expected mount list using `docker inspect <container>`.
3. 03Diff the container's filesystem against the volume's contents: `docker exec <container> ls -la /path/in/container` and on the host `ls -la <mountpoint>`; record what is present in one place but missing in the other.
4. 04Inspect the mount record itself with `docker inspect --format '{{json .Mounts}}' <container>` to see whether each mount is a bind, named volume, anonymous volume, or tmpfs, and which path inside the container it covers.
5. 05Check ownership inside the container with `docker exec <container> stat -c '%u:%g %n' /path/in/container` and compare to the host-side `stat` of the bind source or volume mountpoint.
6. 06Rule out context drift by running `docker context show` and `docker info --format '{{.OperatingSystem}}'` to confirm the daemon is the one expected to host these volumes.

Evidence to collect

• •The full JSON output of `docker inspect <container>`, with particular focus on the `Mounts`, `Config.Volumes`, and `HostConfig.Mounts` fields.
• •The output of `docker inspect --type volume <name>` for every volume referenced, capturing the `Driver`, `Mountpoint`, `CreatedAt`, and any `Options` or `Labels` entries.
• •Result of `docker volume ls --format '{{.Name}}\t{{.Driver}}\t{{.Mountpoint}}'` to map names to on-disk mountpoints in the current daemon.
• •File listings both inside the container (`docker exec ... ls -la`) and on the host at the bind source or volume mountpoint, with timestamps and ownership for each entry.
• •Application log lines emitted during the volume mount or first write attempt, captured via `docker logs --timestamps <container>` and filtered for "permission", "readonly", "ownership", or "no such file" tokens.
• •Daemon context and driver metadata from `docker info`, including `Storage Driver`, `Docker Root Dir`, and the active context name from `docker context show`.

Where to look

• •The boundary between the container's writable layer and mounted volumes: inspect `Mounts` in `docker inspect` to see whether a path is covered by a volume, a bind mount, tmpfs, or the overlay layer.
• •The Dockerfile's `VOLUME` instruction, because any path declared there becomes an anonymous mount point at run time and can shadow an external bind mount or named volume you expected to see.
• •The Docker daemon's volume root directory (default `/var/lib/docker/volumes/` on Linux), reachable via `docker info --format '{{.DockerRootDir}}'`, where named volume mountpoints physically live.
• •The `--mount` vs `-v` flag parsing boundary documented by the Docker run reference, where syntax errors (missing `source=`, relative paths, missing `target=`) silently degrade to anonymous volumes.
• •The Docker context configuration (`~/.docker/contexts/` and `DOCKER_HOST`) that determines which daemon the `docker volume` commands are talking to.
• •Filesystem-level mount tables inside the container, via `docker exec <container> cat /proc/mounts` or `mount`, to confirm what the kernel actually considers mounted at a given path.

Diagnostic steps

1. 01Decide whether the failure is "missing data", "shadowed data", or "non-durable data" by comparing writes inside the container before and after a controlled `docker restart`; only non-named mounts typically lose data.
2. 02For each path the application writes to, look up its covering mount in `docker inspect --format '{{json .Mounts}}' <container>` and classify it as bind, named volume, anonymous volume, or overlay-only.
3. 03If the path is overlay-only, the durability failure is by design per the run reference semantics of unspecified volumes; introduce a named volume or bind mount before retrying.
4. 04If the path is a bind mount but the container reports the directory as empty, diff the host path against the container path; a host subdirectory mounted on a non-empty container directory will shadow its contents until the mount is removed.
5. 05If the path is an anonymous volume, decide whether to promote it to a named volume via `docker volume create` and a follow-up `docker run --mount source=<name>,target=<path>` invocation, rather than relying on auto-generated names.
6. 06For permission failures, correlate the in-container UID/GID (`stat -c '%u:%g'` inside the container) with the host-side ownership of the bind source; if they differ, plan a remap via `--user`, a host chown, or a Dockerfile entrypoint that adjusts ownership.
7. 07For ownership failures on an empty named volume, add a one-shot init container or an entrypoint script that performs `chown -R <uid>:<gid> /data` before the main process starts, and verify with `docker exec ... stat`.
8. 08Cross-check the active Docker context with `docker context show`; if it differs from the daemon that originally created the volume, reconnect to the correct context before drawing conclusions from `docker volume ls`.

Common mistakes

• •Trusting the container's view of the filesystem without consulting `docker inspect`'s `Mounts`, so a path that looks empty inside the container is misread as "data lost" when in reality a bind mount is masking the underlying directory.
• •Recreating a container with `docker run` instead of `docker start`, which by the run reference produces new anonymous volumes for any path not explicitly backed, silently discarding prior data.
• •Assuming `VOLUME` in a Dockerfile is documentation; in practice it forces an anonymous mount at run time and will intercept a later bind mount unless the bind mount is specified with a matching `target`.
• •Running `chown` on the host bind source as root without considering that the container's process runs as a non-root UID, leading to "Permission denied" inside the container even though the host side looks correct.
• •Mixing `-v` and `--mount` in the same `docker run` invocation and assuming they are interchangeable, when actually each parses keys differently and a malformed pair silently falls back to anonymous semantics.
• •Inspecting volumes on the wrong context after switching to Docker Desktop or a remote daemon, so the "missing" volume was never present on the daemon the container actually runs on.
• •Treating `docker volume rm` as safe to run during diagnosis, which is a destructive operation and should not appear in read-only triage; only inspect, never remove, until evidence supports a replacement plan.

Safe fixes

• •Once a missing-data failure is classified as overlay-only, stop the container, create an explicit named volume with `docker volume create <name>`, and relaunch with `--mount type=volume,source=<name>,target=<path>` so writes land on the volume's mountpoint under the daemon's root directory.
• •When a bind mount is shadowing an existing in-container directory, change the mount to target an empty subdirectory inside the container (e.g., `/var/lib/<app>/data` instead of `/var/lib`), so the bind source and the image's expected layout coexist.
• •For UID/GID mismatches on a bind mount, either set the container's process to match the host owner via `docker run --user <uid>:<gid>`, or pre-create the directory on the host with the matching ownership so the container process can read and write without `chown`.
• •For ownership failures on an empty named volume, run a one-time init container with the same volume attached and an entrypoint that performs ownership correction against the volume's mountpoint; persist the corrected state by writing at least one file in that path.
• •For syntax-driven anonymous volume creation, convert the `docker run` invocation to `--mount type=bind,source=<host-path>,target=<container-path>` or `--mount type=volume,source=<name>,target=<container-path>`, both of which fail loudly on typos rather than degrading silently.
• •When context drift is the root cause, switch contexts with `docker context use <expected-context>` before any further inspection, and re-run `docker volume ls` against the daemon that actually hosts the container.

Prove the fix

1. 01After applying a named-volume fix, write a sentinel file from inside the container (`docker exec <container> sh -c 'date > /data/sentinel && sync'`), then `docker stop` and `docker start` the container and confirm `docker exec <container> cat /data/sentinel` returns the original timestamp.
2. 02After correcting a bind-mount shadowing, run `docker exec <container> ls -la <expected-path>` and verify every host-side file from the bind source is visible at that path inside the container, with matching sizes.
3. 03After a UID/GID fix, write a file as the application's user inside the container and confirm on the host (at the bind source or volume mountpoint) that the file appears with the expected ownership, using `stat -c '%u:%g %n' <host-path>`.
4. 04After an ownership init on a named volume, start the main service and check that the application's readiness log no longer contains "wrong ownership" or "Permission denied"; capture `docker logs --tail 50 <container>` and grep for those tokens.
5. 05After resolving context drift, run `docker volume ls` on the corrected context and confirm the previously missing named volume appears; cross-check with `docker inspect --type volume <name>` that the `Mountpoint` is under the expected daemon's volume root.
6. 06In all cases, re-run `docker inspect --format '{{json .Mounts}}' <container>` and confirm the mount type, source, and target match the intended fix, with no anonymous volumes that were not part of the plan.

Prevention and next steps

• •Standardize on `--mount` syntax in runbooks and CI, since it fails loudly on missing or malformed keys rather than producing anonymous volumes that are hard to correlate later.
• •Treat every stateful path in a Dockerfile as a candidate for an explicit `VOLUME` declaration plus a named volume in compose or run commands, so a developer cannot accidentally recreate a container without durable storage.
• •Document expected UID/GID ownership for every bind-mounted directory and bake the matching user into the image or the run command, so a fresh checkout on a new host does not silently fail on first write.
• •Pin the Docker context per project using `docker context use <name>` in setup scripts, and record the context name alongside any volume operation in runbooks to prevent cross-daemon confusion.
• •Add a post-deploy verification step that writes a sentinel file through the application and reads it back after a restart, so a regression in volume wiring is caught before it reaches users rather than after data is already missing.

Safe commands and checks

docker ps -a --format 'table {{.ID}}\t{{.Image}}\t{{.Names}}\t{{.Status}}'
docker inspect --format '{{json .Mounts}}' <container>
docker inspect --format '{{json .Config.Volumes}}' <container>
docker volume ls --format '{{.Name}}\t{{.Driver}}\t{{.Mountpoint}}'
docker inspect --type volume <volume-name>
docker exec <container> ls -la <path-in-container>
docker exec <container> stat -c '%u:%g %n' <path-in-container>
docker exec <container> cat /proc/mounts