How to verify a build is reproducible from the same commit

The symptoms

• •Two clean builds from the same commit produce artifacts whose content hashes differ, even though the source tree is unchanged and the build command is identical.
• •CI artifacts hash inconsistently between reruns of the same workflow on the same commit, while single-run builds look fine.
• •A local build reproduces a remote artifact's hash, but a colleague's machine or a CI runner produces a different hash from the same commit.
• •Cache entries for a task are invalidated on every run despite no source changes, showing as constant cache misses for a task that should be deterministic.
• •A content-addressable cache reports the same logical key but stores two different outputs, causing restore to pick a hash that does not match what a fresh build produces.

Likely causes

• •Undeclared inputs leak into the build: timestamps from the filesystem, absolute paths, embedded user or hostname strings, or generated files written into the source tree before compilation.
• •Non-deterministic ordering from parallel workers, hashmap or set iteration, filesystem listing order, or glob expansion that the build system does not sort.
• •Toolchain drift between environments: different compiler versions, locale settings, timezone, or package manager lockfiles that pin to different transitive resolutions on different machines.
• •Cache key inputs are narrower than the real inputs: the task hashes only source files and omits environment variables, toolchain versions, or configuration files that affect output.
• •Side effects between tasks: an earlier task mutates a shared directory that a later task reads, so the second task's inputs depend on whether the first ran and in what order.

First ten minutes

1. 01Pin the exact commit under test: record the full commit hash, not a branch name, and confirm both builds check out that same hash with no working-tree modifications.
2. 02Inventory declared inputs: list source files, lockfile contents, toolchain versions, environment variables, and configuration files that the build reads. Treat this list as the candidate input set for the reproducibility check.
3. 03Capture one full build run end-to-end: command line, working directory, environment snapshot, and the resulting artifact hashes. Save this as the reference for comparison.
4. 04Wipe all caches and derived outputs, then rebuild in the same environment and diff hashes against the reference. A mismatch inside one environment points at task nondeterminism; a match points at cross-environment drift.
5. 05Compare two fresh builds in a second environment using the same recorded inputs. If hashes diverge only across environments, the cause is environmental; if they diverge within one environment, the cause is nondeterministic task output.

Evidence to collect

• •Commit hash, working tree status (clean or dirty), and the exact build command and arguments used for both runs.
• •Content hashes of every output artifact from each build, computed with a stable algorithm so hashes can be compared directly.
• •Resolved versions of the build tool, compilers, and any codegen tools, plus locale and timezone settings for each environment.
• •Hash inputs that the build system recorded for each task: source file globs, environment variables, lockfile contents, and toolchain pins.
• •Build logs showing task execution order and any warnings about undeclared inputs or hash mismatches between cache key and actual inputs.

Where to look

• •The boundary between the source tree and the build's read set: which files outside the declared source globs does the task touch, including dotfiles, generated files, and dependency caches.
• •The boundary between the host environment and the build: shell environment variables, locale, timezone, and user or hostname fields that the toolchain may embed.
• •The boundary between the cache key and the real inputs: what the build system hashes to identify a cacheable result versus what actually influences the output bytes.
• •The boundary between tasks: shared directories, intermediate output paths, and any filesystem ordering that propagates from one task into the next.
• •The boundary between declared and actual toolchain: lockfile contents versus what package managers actually resolve to on disk in each environment.

Diagnostic steps

1. 01Compute the symmetric difference of the two output file sets; identical file lists with different contents point at nondeterminism, while different file lists point at order-dependent generation.
2. 02For each differing file, run a binary diff and identify which byte ranges diverge; timestamps, embedded paths, and random or counter fields usually cluster in small, identifiable regions.
3. 03Hash every input the task declares it consumes, then hash the actual filesystem reads the task performed during one run; mismatches reveal undeclared inputs leaking into the build.
4. 04Replay the build with a frozen environment snapshot in two containers and compare hashes; if hashes now match, the cause is environmental drift rather than task logic.
5. 05Re-run the same build N times in one environment with caches cleared between runs; any variation across runs confirms within-environment nondeterminism and rules out environment drift as the sole cause.
6. 06Inspect the cache key the build system computed for the failing task against the actual inputs the task used; a key that omits a real input will reuse stale outputs and produce hashes that look correct only by coincidence.

Common mistakes

• •Comparing only one artifact instead of every output; one matching file can hide a divergent binary elsewhere in the build graph.
• •Hashing the wrong thing: using modification timestamps instead of content hashes, or comparing compressed archive bytes that include nondeterministic metadata.
• •Trusting a single rerun: a second run that matches the first can still be nondeterministic if both runs share the same source of variation, such as the same clock or the same hostname.
• •Cleaning only the build output directory while leaving dependency caches intact, so the second build reads stale generated artifacts and looks reproducible by accident.
• •Assuming the lockfile pins everything: lockfiles pin versions, not the resolved binary contents, and two machines can compile different binaries from the same locked source.

Safe fixes

• •If undeclared filesystem inputs are leaking, narrow the task's input glob to the files that genuinely affect output, and exclude generated and timestamp-bearing files from the input set.
• •If environment variables influence output, add the relevant variables to the task's hash inputs so cache keys reflect them, and document which variables must be frozen for verification.
• •If toolchain drift is the cause, pin the toolchain by hash rather than by version string and verify that the resolved binary content matches across environments before comparing output.
• •If parallel execution introduces ordering bugs, force serial execution for the affected task as a verification step; if hashes then match, the fix is to sort inputs or stabilize worker ordering, not to leave the task parallel.
• •If the cache key omits a real input, widen the key to include that input; only do this after confirming the input actually affects output bytes, since over-broad keys reduce cache hit rates.

Prove the fix

1. 01Run the build N times in a clean environment with caches cleared between runs and confirm that every output artifact's content hash is identical across all runs.
2. 02Run the build in two independently provisioned environments using the same frozen inputs and confirm that output hashes match across environments, not just within one.
3. 03Inspect the build system's recorded hash inputs for each task and confirm that any variable known to affect output (toolchain version, locale, relevant environment variables) appears in the key.
4. 04Re-introduce one suspected source of variation at a time and observe that the output hashes change in the expected direction, confirming the verification is sensitive to the inputs it claims to cover.
5. 05Capture a reference artifact hash for the commit and store it alongside the input manifest so that future builds can be checked against a known-good reproducibility baseline rather than against each other.

Prevention and next steps

• •Treat the input manifest as a first-class artifact: version it alongside the code, and review it whenever a task's behavior changes.
• •Run reproducibility checks on a scheduled cadence in CI so that nondeterminism is detected at the commit where it was introduced, not months later.
• •Pin toolchains by content hash and rebuild them from the same source on every environment, so that "same version" implies "same bytes."
• •Avoid embedding paths, hostnames, timestamps, or random values into outputs; when they are unavoidable, funnel them through a single code path that can be audited and frozen.
• •Document the verification procedure itself: the exact commands, environments, and hash algorithms, so that "reproducible" means the same thing to every engineer who checks.

Safe commands and checks

git rev-parse HEAD
git status --porcelain
git diff --quiet HEAD && echo CLEAN || echo DIRTY
sha256sum <artifact-path>
find <output-dir> -type f -print0 | sort -z | xargs -0 sha256sum
comm -23 <(cd <build-a-output> && find . -type f | sort) <(cd <build-b-output> && find . -type f | sort)
cmp -l <build-a-artifact> <build-b-artifact> | head -n 20
env | sort > <env-snapshot-path>
locale; date +%Z