What this usually means
Multi-stage builds rely on stage aliases (`AS builder`) and exact `COPY --from=builder /path /path` syntax. When the artifact is missing, either the builder stage never produced the file (compilation failure, wrong working directory), the alias is misspelled (e.g., `builder` vs `build`), or the path in COPY --from doesn't match the actual output path. BuildKit may cache old layers, masking the issue. The error often surfaces only at runtime because build succeeds with a smaller image.
The first ten minutes — establish facts before touching code.
- 1Run `docker build --no-cache .` to bypass cache and confirm the failure reproduces
- 2Add a temporary `RUN ls -la /expected/path` in the builder stage to verify the file exists
- 3Inspect the builder stage image: `docker build --target builder -t debug-builder . && docker run --rm debug-builder ls -la /app`
- 4Check the final stage's COPY line: `docker build -t final . && docker run --rm final sh -c 'ls -la /expected/path'`
- 5Examine build output for warnings: look for 'COPY --from=builder failed' or 'file not found'
- 6Use `docker history final-image` to see which layers contributed files
The specific files, logs, configs, and dashboards that usually own this bug.
- searchDockerfile: stage aliases (`AS builder`) and COPY --from lines
- searchBuild output log: especially lines with 'COPY --from=' and any '[WARNING]'
- searchBuilder stage image (intermediate): `docker build --target builder -t debug-builder .` then shell in
- searchDockerfile context: ensure compiled output path matches COPY source path exactly (trailing slashes matter)
- searchCI/CD pipeline logs: sometimes the artifact is built in a different working directory
- search.dockerignore: accidentally excluding the build output directory
Practical causes, not theory. These are the things you will actually find.
- warningMisspelled stage alias in COPY --from (e.g., `--from=build` vs `--from=builder`)
- warningBuilder stage compiles to a different path than COPY expects (e.g., `/go/bin/app` vs `./app`)
- warningMissing `AS` label on the builder stage: `FROM golang:1.20` without `AS builder`
- warningBuild step fails silently: compilation error ignored due to `&&` chaining or `|| true`
- warningCOPY --from uses a numeric stage index that changes when stages are reordered
- warningFile permissions: artifact exists but can't be copied due to OS restrictions (rare)
- warningBuildKit caching: old builder image with missing artifact is reused
Concrete fix directions. Pick the one that matches your root cause.
- buildExplicitly name every intermediate stage with `AS <name>` and reference by name
- buildUse `COPY --from=builder /app/binary /app/binary` with absolute paths; avoid relative paths
- buildAdd `RUN --mount=type=cache,from=builder` or use `docker build --no-cache` to invalidate cache
- buildBreak compilation into separate RUN commands to isolate failures (e.g., `RUN go build -o /app/binary .`)
- buildValidate artifact existence in builder stage with a RUN command that fails if file missing: `RUN test -f /app/binary`
- buildPin stage names with comments if using BuildKit experimental features
A fix you cannot prove is a guess. Close the loop.
- verifiedBuild with `--no-cache` and check `docker run final-image /app/binary --version`
- verifiedAdd a health check that runs the missing binary and confirm it works
- verifiedCompare final image file listing against builder stage: `docker diff` on intermediate containers
- verifiedRun `docker build --target builder -t debug . && docker run --rm debug ls -la /app` to confirm output
- verifiedCheck image size increased appropriately (e.g., from 5 MB to 50 MB after fix)
- verifiedAutomate verification in CI: `docker run final-image sh -c '[ -f /app/binary ]'`
Things that make this bug worse or harder to find.
- warningUsing `.` as COPY source — it copies the whole context, not just the artifact
- warningAssuming COPY --from works with relative paths; always use absolute paths in builder stage
- warningAdding `--from=builder` to a stage that doesn't have `AS builder`
- warningRelying on cached layers without testing with `--no-cache` after changing stage names
- warningIgnoring build warnings about 'file not found' — they are silently skipped
- warningUsing the same stage alias in multiple FROM lines (aliases must be unique)
Missing Go Binary in Production Docker Image
Timeline
- 09:15Deploy to staging — container exits immediately with 'exec /app/server: no such file or directory'
- 09:18Check Dockerfile: multi-stage with 'FROM golang:1.21 AS builder' and 'COPY --from=builder /app/server /app/server'
- 09:25Run 'docker build --no-cache .' — build succeeds with no errors, final image is 6 MB
- 09:30Inspect builder stage: 'docker run --rm debug-builder ls -la /app' — shows 'server' exists, 40 MB
- 09:35Inspect final stage: 'docker run --rm final ls -la /app' — directory exists but is empty
- 09:40Notice COPY line: 'COPY --from=builder /app/server /app/server' — typo: 'builder' vs 'builder' (should be 'builder')
- 09:42Fix typo: change '--from=builder' to '--from=builder' (stage alias is 'builder')
- 09:45Rebuild — final image is 46 MB, container runs successfully
We pushed a new microservice to staging and immediately saw the container crash with 'exec /app/server: no such file or directory'. The Dockerfile used a multi-stage build: a builder stage compiled the Go binary, and a scratch stage copied it. The build succeeded locally, but the binary wasn't in the final image. I checked the builder stage — binary was there. I checked the final stage — empty. That narrowed it to the COPY command.
I stared at the Dockerfile for five minutes before seeing it: the stage alias was 'builder' (with an 'i'), but the COPY --from referenced 'builder' (with an 'l'). A single-character typo. Because the 'builder' stage didn't exist, Docker silently failed to copy. The build succeeded because COPY --from from a non-existent stage is treated as a warning, not an error.
I fixed the typo, rebuilt with --no-cache, and the final image jumped from 6 MB to 46 MB. The container ran perfectly. I added a CI step to verify the binary exists in the final image after build. That typo cost us 25 minutes of debugging — all because Docker doesn't fail on missing stage references.
Root cause
Typo in COPY --from stage name: 'builder' referenced instead of 'builder' (stage alias was 'builder'). Docker does not fail build on missing stage alias — it silently produces an empty copy.
The fix
Corrected COPY --from=builder to COPY --from=builder. Added a verification step in CI: 'docker run final-image test -f /app/server'.
The lesson
Always stage aliases match COPY --from references exactly. Never rely on silent warnings — add explicit file existence checks in CI.
Docker resolves `COPY --from=<name>` by looking for a stage whose alias matches `<name>`. If no stage matches, Docker issues a warning: `[Warning] One or more build-args [from] were not consumed` (for older versions) or a more specific file-not-found warning. Crucially, the build does NOT fail — it just copies nothing. This is by design for backward compatibility but is a common pitfall.
The alias is set with `AS <name>` after the FROM line. Names are case-sensitive. You can also reference stages by index (0-based), but this breaks if you reorder stages. For example, `COPY --from=0 /app /app` copies from the first stage. Indexes are fragile and should be avoided.
BuildKit (default since Docker 18.09) caches intermediate layers aggressively. If the builder stage hasn't changed, it reuses the cached layer — even if the source code changed. This can cause the artifact to be missing if the cache is stale. Always use `--no-cache` when debugging multi-stage issues, or invalidate cache by changing the FROM digest.
Additionally, BuildKit's concurrent stage execution may produce inconsistent states if stages share volumes or have race conditions. While rare, it's worth checking if the artifact is produced conditionally (e.g., only on certain build args).
COPY --from interprets trailing slashes differently. `COPY --from=builder /app/ /app/` copies the contents of /app/ into /app/, while `COPY --from=builder /app /app` copies the directory itself. If the artifact is at /app/server, copying /app/ will include server, but copying /app will create /app/app/server. This subtlety causes many missing artifacts.
Best practice: use absolute paths and copy the file directly, not the directory: `COPY --from=builder /app/server /app/server`. This avoids ambiguity and makes the intent explicit.
To inspect the builder stage, build it as an image: `docker build --target builder -t builder-image .`. Then run a shell: `docker run --rm -it builder-image sh`. This gives you the exact filesystem the builder stage produces.
For the final stage, use `docker diff` to see which files were added by each layer. First, get the container ID from a `docker run -d final sleep 60`, then run `docker diff <container>` to list added/modified/deleted files. This quickly shows if the COPY command actually added files.
Frequently asked questions
Why does docker build succeed even though the artifact is missing?
Docker treats a `COPY --from` referencing a non-existent stage as a warning, not an error. The build continues, but no files are copied. This is historical behavior for backward compatibility. You can enforce strict checking with `--check` (experimental in Docker 24+) or by adding a `RUN test -f` in the final stage.
Can I use numeric stage indices instead of names?
Yes, but it's fragile. Indexes are zero-based and depend on order. If you add a new stage before the referenced index, the index shifts, and you copy from the wrong stage. Always use named aliases for clarity and reliability.
How do I verify the artifact exists in the builder stage before copying?
Add a RUN command that fails if the file is missing: `RUN test -f /app/binary` right after the build step. This ensures the build fails early if the artifact isn't produced. You can also use `ls -la` for debugging, but test -f is cleaner for CI.
What if the artifact is a directory?
Copying a directory works similarly: `COPY --from=builder /output /output` copies the directory's contents. To copy the directory itself (creating /output/output), use `COPY --from=builder /output /`. Be explicit about trailing slashes to avoid surprises.
Can .dockerignore cause missing artifacts?
Yes, if the artifact is generated outside the build context (e.g., downloaded from a URL) and then copied. But more commonly, .dockerignore excludes source files needed for compilation, causing the builder stage to produce nothing. Check that your .dockerignore doesn't exclude the source directory.