Docker volume data disappears after restart: compare mount and container paths

The symptoms

• •Files written by the application inside the container are present during the container's lifetime but gone after `docker restart` or `docker stop` followed by `docker start`.
• •A developer confirms a write from inside the container, but a subsequent `docker exec` shows the path is empty or has reverted to an earlier state.
• •Database or uploaded file directories appear to reset to the contents of the original image, despite a `-v` flag being used in `docker run`.
• •Two containers started with seemingly identical `-v` flags contain different file sets, suggesting they resolved to different storage backends.
• •The path inside the container exists and is writable, but `docker volume ls` and `docker volume inspect` do not show any volume that maps to the expected data.

Likely causes

• •The container was started without any persistent volume mount, so writes went to the container's writable layer on the copy-on-write filesystem, which is discarded on container removal.
• •A bind mount was used, but the container path in `-v <host>:<container>` was typed incorrectly, so the application wrote to a different in-container directory that happens to look like the expected data path.
• •The `-v` flag was overridden by `--mount` with a different source, or by a Compose file that redefines the volume mapping for that service.
• •An anonymous volume was auto-created on a path that already had a bind mount, and Docker's precedence rule placed a shadow volume over the bind, hiding the real data.
• •The container was removed and recreated rather than restarted, which destroys any data that lived only on the container's writable layer.
• •Data was written to a tmpfs path such as `/tmp` or to a path inside the image itself, neither of which survives container removal.

First ten minutes

1. 01Capture the exact `docker run` or `docker compose up` invocation that started the container, including every `-v` and `--mount` flag and any Compose `volumes:` block, before changing anything.
2. 02List running and stopped containers with `docker ps -a` and record the container ID for the affected service so subsequent commands target the right instance.
3. 03Run `docker inspect <container_id>` and read the `Mounts` array to record the source, destination, type (`volume`, `bind`, or `tmpfs`), mode (`rw` or `ro`), and whether `RW` is true for each entry.
4. 04Compare the recorded `Mounts` entries against the directory the application claims to write to; a missing or mismatched `Destination` is the primary signal that writes went to the ephemeral layer.
5. 05From inside the container, list the suspect path with `ls -la` and record modification times, then exit without restarting so you can compare after the next restart cycle.
6. 06Check `docker volume ls` and `docker volume inspect <name>` for any named volume referenced in `Mounts`, and confirm the `Mountpoint` resolves to a directory that still contains the expected files on the host.

Evidence to collect

• •The full `docker inspect <container_id>` output, with focus on the `Mounts` array fields `Type`, `Source`, `Destination`, `Mode`, and `RW`.
• •The exact `-v`, `--mount`, `--volume`, or Compose `volumes:` declaration that started the container, including any `driver`, `driver_opts`, or `source` overrides.
• •The contents of the host directory that the bind mount or named volume points to, captured with `ls -la` and `stat` on representative files.
• •Modification times of files inside the container at the suspect path, taken before and after a controlled `docker restart`, to demonstrate whether the path is persistent.
• •The output of `docker volume ls` and `docker volume inspect <name>` for every volume referenced by the container, including the `Mountpoint` field on the host.
• •Whether the container was stopped and restarted with `docker restart` or removed and recreated with `docker rm` followed by `docker run`, since recreate destroys the writable layer.

Where to look

• •The boundary between the host filesystem and the container's mount namespace, which `docker inspect` `Mounts` describes as the `Source` to `Destination` mapping.
• •The container's writable layer at `/var/lib/docker/overlay2/<layer>/diff`, which holds any writes that did not land on a mounted volume and is discarded on container removal.
• •The named volume directory under `/var/lib/docker/volumes/<volume_name>/_data`, which is the persistence target for `docker volume create`-style mounts.
• •The Compose project metadata, typically in `docker-compose.yml` or `compose.yaml`, where `volumes:` and `services.<name>.volumes` can silently override CLI flags.
• •The Dockerfile `VOLUME` directive, which causes Docker to auto-create an anonymous volume on a path even when the operator did not ask for one.
• •The container logs for the application entrypoint, to confirm the application is actually writing to the path you think it is rather than a configured default such as `/tmp` or `/data`.

Diagnostic steps

1. 01Run `docker inspect <container_id>` and locate the `Mounts` array. If `Mounts` is empty for the suspect container path, the application is writing to the writable layer and the data will not survive container removal.
2. 02For each entry in `Mounts`, verify that `Destination` exactly matches the path the application writes to, including trailing slashes and case, because a one-character mismatch routes writes to an unmounted path inside the container.
3. 03If `Type` is `volume`, run `docker volume inspect <Source>` and compare `Mountpoint` on the host to the directory the application expects; a mismatch indicates the volume was created against a different source than the operator assumed.
4. 04If `Type` is `bind`, resolve `Source` on the host with `ls -la` and `stat` to confirm it points to the intended directory and is not a typo, a symlink to a different filesystem, or a path inside another container's overlay.
5. 05Test persistence by creating a sentinel file inside the container at the suspect path, then issuing `docker restart <container_id>` and re-listing the path; presence of the sentinel confirms the mount is persistent, absence confirms the writes are ephemeral.
6. 06Cross-check against any Dockerfile `VOLUME` directive, since Docker auto-creates an anonymous volume on those paths and Compose may mount a named volume on top, leaving the anonymous volume orphaned and shadowed.
7. 07Inspect Compose definitions with `docker compose config` for the project and confirm the resolved `volumes:` block for the service matches the operator's intent, noting that Compose silently merges CLI and file definitions.

Common mistakes

• •Assuming that specifying `-v` on the command line overrides a Compose `volumes:` block; Compose's file definition takes precedence over ad-hoc CLI flags for the same service.
• •Confusing container restart with container recreate; `docker restart` preserves the writable layer for the same container ID, but `docker rm` followed by `docker run` produces a new container with a fresh writable layer.
• •Writing to `/tmp` or another tmpfs-backed path inside the container, which is not a mount target the operator can inspect from the host and is cleared on container removal.
• •Trusting the path the application logs claim it is writing to without verifying the in-container mount table, since the application may use a default that differs from the configured data directory.
• •Treating anonymous volumes created by Dockerfile `VOLUME` directives as the same as named volumes created by the operator, when in fact anonymous volumes are orphaned when the container is removed and the volume's contents become unreachable.
• •Ignoring the `RW` field in `Mounts`, which can be `false` even when the mount appears correct, causing writes to silently fail or be redirected.

Safe fixes

• •If `Mounts` shows no entry for the suspect container path, stop the container, then start a new container with an explicit `-v <named_volume>:<container_path>` or `--mount type=volume,source=<named_volume>,destination=<container_path>` so the writes target a persistent volume rather than the writable layer.
• •If `Mounts` shows the wrong `Destination`, stop the container, then restart it with the corrected container path in the `-v` flag, and re-run `docker inspect` to confirm the new `Destination` matches the application's actual write path.
• •If a bind mount's `Source` resolves to an unexpected directory on the host, stop the container, then restart it with the corrected host path and verify with `ls -la` on the host that the expected files are visible at the new `Source`.
• •If a Dockerfile `VOLUME` directive has shadowed the operator's bind mount, stop the container, remove the orphaned anonymous volume with `docker volume rm <name>` after confirming it contains no needed data, then restart with a bind mount to the intended host path.
• •If Compose definitions override the CLI flags, edit the Compose file so the service's `volumes:` block matches the intended host path or named volume, then run `docker compose up -d` to apply the change, and verify with `docker compose config` before recreating.

Prove the fix

1. 01Write a uniquely named sentinel file inside the container at the suspect path, capture its name and modification time, then run `docker restart <container_id>` and confirm the sentinel is still present with the same modification time.
2. 02After the restart, run `docker inspect <container_id>` and confirm `Mounts` contains an entry whose `Destination` exactly matches the application write path and whose `RW` is `true`.
3. 03For a named volume, run `docker volume inspect <volume_name>` and confirm `Mountpoint` on the host contains the sentinel file with the expected modification time.
4. 04For a bind mount, list the host `Source` directory and confirm the sentinel file appears there with the expected modification time.
5. 05Remove the container with `docker rm <container_id>`, then start a new container with the same `-v` declaration and confirm the sentinel and all prior data are present without any data-restore step.

Prevention and next steps

• •Standardize on named volumes created with `docker volume create` rather than anonymous volumes, and document the volume name and mount destination in the service runbook.
• •In Compose, treat the `volumes:` block as the authoritative definition for each service and avoid mixing CLI `-v` flags with Compose-managed services for the same data path.
• •Before deploying, run `docker inspect` on a test container and confirm every critical write path appears in `Mounts` with the expected `Destination` and `RW: true`.
• •Avoid relying on `docker restart` alone as a persistence guarantee; the writable layer is preserved only for the same container ID, so include an explicit recreate-and-verify step in any disaster recovery drill.
• •Avoid writing application data to `/tmp` or other tmpfs paths inside the container, and configure the application to write to a path that is explicitly mounted as a volume or bind mount.

Safe commands and checks

docker ps -a --filter id=<container_id>
docker inspect <container_id>
docker volume ls
docker volume inspect <volume_name>
docker exec <container_id> ls -la <container_path>
docker exec <container_id> stat <container_path>
docker compose config
docker restart <container_id>