Same commit produces different output: identify non-hermetic inputs

The symptoms

• •Cache hash for a given commit changes between two otherwise identical local builds with no source edits.
• •Remote CI artifact bytes differ from a local artifact produced from the exact same commit SHA.
• •Reproducible-build checksum (e.g., diffoscope, sha256 of tarball) mismatches across reruns while inputs remain constant.
• •Build logs contain absolute paths, hostnames, user names, timestamps, or locale-specific formatting embedded in compiled output.
• •Dependency lockfile resolves to different versions across machines or across successive runs despite a pinned lockfile.
• •Generated code or assets (protobuf, OpenAPI, GraphQL, codegen) emits different bytes on each run due to non-deterministic ordering.

Likely causes

• •Wall-clock or timezone leakage: SOURCE_DATE_EPOCH not set, or build tools embed __DATE__, __TIME__, build timestamps, or filesystem mtimes into artifacts.
• •Unpinned or floating dependency versions (e.g., npm semver ranges, Maven SNAPSHOTs, Go pseudo-versions, Cargo git refs) that resolve differently across runs.
• •Environment leakage: PATH, LANG/LC_ALL, USER, HOSTNAME, HOME, or shell environment variables are baked into compiled output or generated files.
• •Network or registry non-determinism: package fetched from a remote registry without an integrity hash, or a CDN-served asset that changes between fetches.
• •Non-deterministic file ordering: directory walks, glob expansion, or tar/zip steps that emit entries in filesystem order rather than sorted order.
• •Filesystem metadata leakage: timestamps, UIDs, GIDs, or permission bits carried into archives or compiled into resources.

First ten minutes

1. 01Confirm the variance is real by capturing sha256 of two consecutive outputs of the same commit and computing a diff; this rules out caching-miss misdiagnosis.
2. 02Inspect the build tool's hash-input manifest (Turborepo's hashing inputs, Bazel's --show-inputs, Nix derivation hash) to see which files it claims affect the hash.
3. 03Capture the full environment of one build with `env -i` plus the documented required vars, then compare against a second build's environment to detect drift.
4. 04Sort and compare the contents of any generated source files (e.g., *.pb.go, generated.ts, *_generated.rs) byte-for-byte to localize the diff to a specific generator stage.
5. 05Check lockfiles for resolution timestamps, registry URLs, or platform-specific resolution fields that may indicate lockfile non-determinism.
6. 06Disable cache and run the build twice in succession; if outputs still differ, the variance is from inputs, not from a stale cache.

Evidence to collect

• •sha256 sums of the final artifact from two runs of the same commit, plus byte-level diff of differing regions.
• •Hash-input manifest from the build tool listing every file path and env var that contributed to the cache key.
• •Sorted list of environment variables in both builds, with values, to detect any drift in PATH, locale, or user-specific vars.
• •Resolved dependency versions from the lockfile plus the registry URL and any integrity/SHA fields.
• •Generator tool version and configuration (proto compiler, openapi-generator, schema codegen) used in each run.
• •Order of files emitted by any glob, find, or tar step, comparing sorted versus actual output order.

Where to look

• •Build tool configuration boundary: the inputs declared to the cache key (Turborepo inputs in turbo.json, Bazel --host_action_env, Nx task inputs).
• •Source generator boundary: any code generator invoked during pre-build, including its version, flags, and schema inputs.
• •Dependency resolution boundary: lockfile parser output, package manager cache, and registry integrity verification.
• •Filesystem walk boundary: glob/find/tar invocations whose order is not explicitly sorted.
• •Process environment boundary: env vars read at build time, especially those passed through to compilers or codegen tools.
• •Output archive boundary: any post-build packaging step (tar, zip, jar, deb) that may embed filesystem metadata.

Diagnostic steps

1. 01Run the build twice with the cache disabled and capture sha256 of both artifacts; equal hashes rule out environmental drift as the source of variance.
2. 02Extract and sort the build tool's hash inputs (e.g., `turbo run build --summarize` output) to identify which file paths are hashed but are not in version control.
3. 03Bisect generated versus compiled regions of the differing artifact: if the diff is confined to a generator's output, isolate the generator; if it spans the binary, suspect env or path leakage.
4. 04Re-run the build with `SOURCE_DATE_EPOCH` set to a fixed integer and compare hashes; unchanged variance means timestamps are not the source.
5. 05Re-run with all locale and user environment unset (`LC_ALL=C`, `USER=builder`, `HOME=/tmp/builder`) and compare hashes to test for env-var leakage.
6. 06Inspect lockfile resolution metadata; if it contains floating ranges, registry fetch timestamps, or platform-specific fields, dependency drift is a candidate.
7. 07Sort the input list to any archive or codegen step explicitly and verify the output bytes stabilize; non-deterministic ordering is confirmed if sorting fixes the diff.

Common mistakes

• •Assuming the build is hermetic because cache hits return the same artifact, without verifying that cache misses also produce identical artifacts.
• •Hashing only the final binary while ignoring intermediate artifacts, generator outputs, or source maps that may embed non-deterministic content.
• •Trusting lockfiles as deterministic without checking for floating versions, registry-specific resolution metadata, or platform-conditional resolution.
• •Diffing artifact names rather than contents, which can mask variance introduced by time or hostname embedded in metadata.
• •Resetting only PATH or LANG while leaving other locale, timezone, or user-specific variables uncontrolled.
• •Reproducing on the same machine in the same shell, which masks env-var leakage that appears across different CI runners or developers.

Safe fixes

• •Condition: diff is localized to timestamps or embedded dates. Fix: export `SOURCE_DATE_EPOCH=<fixed_unix_seconds>` and configure generators to honor it; re-run and confirm hash stabilizes.
• •Condition: lockfile shows floating ranges or unconstrained refs. Fix: regenerate the lockfile with `--frozen-lockfile` semantics and pin every transitive to an exact integrity hash; verify with a clean cache.
• •Condition: hash-input manifest includes env vars not in version control. Fix: declare only the inputs the task truly depends on (per Turborepo's inputs configuration) and remove env vars that do not affect output.
• •Condition: file ordering causes archive or codegen variance. Fix: pipe directory walks through `LC_ALL=C sort` before feeding the next stage; confirm archive entries are emitted in sorted order.
• •Condition: locale-dependent formatting appears in output. Fix: set `LC_ALL=C` and `LANG=C` for the build process; verify generated strings are byte-identical across locales.
• •Condition: package fetched from network without integrity check. Fix: add an integrity/SRI hash to the fetch step or vendor the dependency; verify offline rebuild produces the same hash.

Prove the fix

1. 01Build the same commit twice in isolated environments with the declared inputs fixed and confirm the final artifact hashes and manifests match.
2. 02Compare the remaining artifact differences byte by byte and verify that timestamps, paths, dependency resolutions, and generated-file ordering no longer vary.
3. 03Run the build in CI and locally from the same lockfile revision, then record the matching output hash as the regression baseline.

Prevention and next steps

• •Declare explicit, minimal inputs for every cacheable task and review them on PRs so accidental inputs (logs, timestamps, env) cannot enter the hash.
• •Pin all dependencies to exact versions with integrity hashes; reject builds that resolve floating ranges in CI.
• •Standardize the build environment via a hermetic image or container with fixed locale, timezone, and PATH; require local builds to match.
• •Adopt reproducibility assertions in CI: run each release build twice, compare sha256, and fail the pipeline on mismatch.
• •Add a pre-commit or CI step that diffs generated source files across two runs to catch non-determinism before merge.

Safe commands and checks

sha256sum <artifact_path>
diff -q <artifact_run_1> <artifact_run_2> && echo IDENTICAL || echo DIFFERENT
LC_ALL=C sort <unsorted_file_list> > <sorted_file_list>
env -i HOME=<build_home> PATH=<build_path> LC_ALL=C SOURCE_DATE_EPOCH=<fixed_unix_seconds> <build_command>
tar -tf <archive> | LC_ALL=C sort > <archive_sorted_manifest>