How to verify Docker volume persistence across replacement

The symptoms

• •Files written inside a container vanish after `docker rm` followed by a recreated container, even though the previous run appeared to persist state.
• •A service "loses memory" between deploys: configuration, database files, or uploaded assets reappear as defaults after a restart on a new container ID.
• •Two containers started with different `-v` flags report different on-disk content for what was assumed to be the same data directory.
• •Inspected mounts on a running container do not match the path documented in the run command or compose file, suggesting the mount was not actually applied.

Likely causes

• •An anonymous volume was created on first run and discarded when the container was removed, because no named volume or host bind mount was specified.
• •A bind mount was declared but the source path on the host was empty, so writes were staged in the container's writable layer instead of on the host filesystem.
• •Volume driver mismatch: the new container was started against a different volume driver or storage backend than the original, isolating the data.
• •Path typo or trailing-slash mismatch caused the source directory to be mounted as a parent of the intended path, leaving the target inside the container effectively unwritable for persistence.
• •Container recreate used `--rm` or was orchestrated by tooling that drops the anonymous volume reference before relaunch.

First ten minutes

1. 01Stop further container recreation cycles; additional churn can make evidence irreproducible and obscure which mount actually carried the data.
2. 02Capture the exact `docker run` (or compose) command that originally started the container, including every `-v`/`--volume` and `--mount` flag and the image tag, so the claim of persistence can be checked against the documented syntax.
3. 03List all volumes with `docker volume ls` and cross-reference names against the run command; anything not named there is an anonymous volume attached to the container's lifecycle.
4. 04Record the container ID of the previous run and inspect its mount table with `docker inspect <container_id>` filtered to the Mounts field, to record what was actually attached versus what was intended.
5. 05Write a sentinel file with a unique timestamp into the suspect path inside the still-running container, so survival across replacement can be proven rather than asserted.
6. 06Sketch the expected boundary: named volume → Docker-managed directory under the volumes root; bind mount → explicit host path; anonymous volume → ephemeral, tied to container ID.

Evidence to collect

• •The literal run command string, with image, tag, every `-v` and `--mount` clause, and the container name used for the previous instance.
• •Output of `docker volume ls` and `docker volume inspect <name>` for each named volume referenced in the run command, capturing Driver, Mountpoint, and Name.
• •The Mounts array from `docker inspect <container_id>` for both the old and the replacement container, showing Source, Destination, Type, Mode, and RW fields.
• •A timestamped sentinel file written through the container into the suspect mount, plus a separate sentinel written directly on the host bind-mount source if applicable.
• •Diff result between the host bind-mount source directory and the path the container reports inside its filesystem after replacement.

Where to look

• •At the run-command boundary: the exact `-v`/`--mount` flags decide whether persistence is even possible; anything absent here caps the rest of the investigation.
• •At the Docker volume boundary: named volumes live under the driver-defined volume root; inspect the Mountpoint and confirm it is on storage that survives container removal.
• •At the container mount boundary: `docker inspect` Mounts shows what the kernel actually mounted into the container's filesystem namespace, which is the only ground truth.
• •At the host filesystem boundary for bind mounts: the Source path must exist, be writable by the container's UID, and be on storage that is not pruned by container removal.
• •At the orchestrator boundary: if Compose or another tool drove the replacement, compare its effective mount definition with the literal flags observed in `docker inspect`.

Diagnostic steps

1. 01Confirm mount type using `docker inspect <container_id>` Mounts: Type is `volume` for managed volumes and `bind` for host paths; mismatches with the run command explain most "data lost on recreate" reports.
2. 02For Type `volume`, read the Name field and run `docker volume inspect <name>` to record the Mountpoint; this is the directory whose contents must survive container removal.
3. 03For Type `bind`, verify that Source is an absolute host path, exists on the host, and is writable; empty or read-only Source paths explain why writes never reached persistent storage.
4. 04Remove only the test container with `docker rm <container_id>` (without `-v` to preserve anonymous volumes) and create a fresh container using the same `-v`/`--mount` syntax; any divergence in the Mounts array between the two inspects pinpoints the cause.
5. 05Inside the replacement container, list the destination path and compare against the sentinel recorded before removal: presence proves persistence, absence disproves it.
6. 06For bind mounts, additionally diff the Source directory on the host against the container's destination path to rule out a mount that points one level above the intended target.
7. 07Repeat the test with the new container's `--rm` flag removed and with the named volume explicitly reused; if persistence appears only with explicit reuse, the previous run was relying on an anonymous volume.

Common mistakes

• •Assuming any `-v` flag creates a named volume: a single argument without a colon-prefixed name creates an anonymous volume tied to the first container that uses it.
• •Trusting that a container's writable layer persists: union filesystem layers are discarded with the container unless a mount captured the writes.
• •Confusing the destination path inside the container with the source on the host; reversing the two arguments silently re-points persistence to a different location.
• •Running `docker rm` with `-v` during cleanup, which removes anonymous volumes and removes the only evidence of where data may have lived.
• •Re-tagging an image and assuming the volume follows it; volumes are independent of image identity, so recreation must reuse the volume by name to retain data.
• •Treating `docker volume ls` as proof of persistence: presence only confirms a volume object exists, not that the container's writes reached it.

Safe fixes

• •If the run command lacked a name, replace it with an explicit named-volume form such as `-v <volume_name>:<destination>` so the volume survives independent of any container ID, per the `-v`/`--volume` syntax documented in the run reference.
• •If a bind mount was intended, add or correct `--mount type=bind,source=<absolute_host_path>,target=<container_path>` and verify the host path exists and is writable before restarting the container.
• •If anonymous volumes are suspected, recreate the container with `--volumes-from <original_container_id>` or with the same named volume attached, and re-run the sentinel test to confirm survival.
• •If the Mounts array in `docker inspect` disagrees with the run command, treat the inspect output as authoritative and adjust the run command until both agree before claiming persistence.
• •If path mismatch is suspected, simplify to a single test directory on the host, mount it at a single container path, and re-validate before reintroducing layered configuration.

Prove the fix

1. 01With the corrected run command in place, write a uniquely named sentinel file through the running container into the mount destination and record its name and timestamp.
2. 02Stop and remove only the container with `docker rm <container_id>` (no `-v`), then start a new container using the identical `-v`/`--mount` clause and image tag.
3. 03Inside the replacement container, list the destination path and confirm the sentinel file is present with the original content; for bind mounts, also confirm the same file appears at the host Source path.
4. 04Run `docker inspect <new_container_id>` and verify the Mounts entry matches the original Mounts entry by Type, Source, Destination, and RW; any drift invalidates the proof.
5. 05Repeat the write-remove-replace sequence once more to show reproducibility; persistence must hold across two independent replacement cycles, not just one.

Prevention and next steps

• •Standardize on named volumes for any data that must outlive a container, and keep a registry of volume names alongside the run commands that own them.
• •Treat `docker inspect` Mounts as the canonical record of what is actually mounted, and diff it against the run command in code review or CI before deployment.
• •Avoid `docker rm -v` in cleanup scripts for any volume that carries application state; use explicit `docker volume rm` against named volumes only.
• •Pin image tags and mount syntax in version control so replacement containers cannot silently drift to a different mount configuration between deploys.

Safe commands and checks

docker volume ls
docker volume inspect <volume_name>
docker inspect <container_id>
docker rm <container_id>
docker run --rm -v <volume_name>:<container_path> <image:tag> ls -la <container_path>
docker run -v <host_source_path>:<container_path> <image:tag> sh -c 'echo <sentinel> > <container_path>/<sentinel_file>'
diff -r <host_source_path> <reference_path>