Reproducible-build checklist

The symptoms

• •Hash of a built artifact changes across two runs of the same commit on the same machine with no code edits, while file timestamps differ between runs.
• •Remote cache hit ratio drops to zero for a previously cached task even though the package inputs hash did not change.
• •Two engineers produce different binary sizes or byte counts for the same tagged commit, but their source trees agree under git diff.
• •Build succeeds locally but a clean CI run of the same commit emits artifacts whose digests differ from a previously recorded golden output.
• •Adding an unrelated workspace to a Turborepo pipeline causes selected downstream tasks to be invalidated in a way that is not reproducible across machines.

Likely causes

• •Undeclared inputs into the hashing function: environment variables, OS-specific paths, or timestamps read inside the build script and not captured by the task's input glob.
• •Non-deterministic file ordering when tools walk a directory, producing different embedded metadata or symbol tables even when file contents are identical.
• •Toolchain drift: package manager lockfiles or compiler toolchains diverge between environments, so the same source produces different object code.
• •Floats in filesystem metadata: mtime, inode, or permission bits influence archive creation or signing steps and leak into the final artifact.
• •Cache key collisions due to package boundaries or roots defined too broadly, so a logically unrelated edit appears to change the hash of an unrelated task.

First ten minutes

1. 01Capture the exact task invocation, working directory, and commit SHA used to produce the divergent artifact, and confirm the source tree is identical with git diff against the recorded SHA.
2. 02Read the cache hit and miss events from the Turborepo logs and record which task keys reported hits versus misses, so the boundary of the divergence is known before any deeper analysis.
3. 03Inspect the canonical inputs that Turborepo declares for a task: package.json glob, source glob, environment variables, and the dependencies list, and compare them against the actual files read during the build.
4. 04Hash the produced artifact with a stable digest function and compare two runs from the same commit; record whether the divergence is byte-level or only metadata-level.
5. 05Quarantine the failing task by disabling remote cache for it briefly so local-only behavior can be observed without polluting shared caches.

Evidence to collect

• •The Turborepo task key, its full inputs specification, and the recorded hash for the artifact at the previous known-good run.
• •List of environment variables in scope during the build, distinguishing variables explicitly declared in the task config from those only present in the shell.
• •Output of a deterministic archive operation over the source tree, and a side-by-side byte comparison of the two divergent artifacts limited to header or metadata sections.
• •Resolved versions of the relevant toolchain components as seen by the build, captured from the package manager lockfile and resolved at execution time.
• •Filesystem metadata snapshot for the input set, including timestamps and permission bits, to check whether ordering or filesystem variation is a factor.

Where to look

• •At the boundary between the build graph and the filesystem: the globs and package paths that Turborepo reads to form a task hash, since these define what counts as an input.
• •At the boundary between the build script and the environment: the set of environment variables actually read at runtime, compared to the variables documented as inputs.
• •At the boundary between the producer and the cache: the artifacts directory and the cache storage layout, since divergence often appears first as a miss followed by a near-match.
• •At the boundary between the source tree and the build tool: the directory walk order used by the tool, which can encode non-determinism even when file contents are stable.
• •At the boundary between the package manager and the toolchain: the lockfile and resolved binary versions, which set the baseline for byte-level reproducibility.

Diagnostic steps

1. 01Confirm the source identity by checking that the recorded commit SHA matches the working tree and that no untracked files are being included via a broad glob pattern.
2. 02Decompose the task hash by comparing the documented inputs to the actual files read during the build, and flag any path, variable, or pattern that exists at runtime but is absent from the declared inputs.
3. 03Diff two artifact hashes produced from the same commit and classify the difference as either affecting payload bytes, embedded metadata, or both, since each class points to a different likely cause.
4. 04Run the task twice with a controlled environment that exposes only the declared variables, and compare hashes; if they now match, the divergence was driven by an undeclared input.
5. 05Replay the task with the toolchain pinned to the lockfile-resolved versions and compare hashes; if divergence persists, the cause is downstream of the toolchain and likely in archive or sign steps.
6. 06Restrict the cache key to a single, well-defined package boundary and re-run, to determine whether the original divergence was caused by an over-broad root definition rather than a genuine input change.

Common mistakes

• •Assuming a missing cache hit implies a code change, when in fact an undeclared environment variable or filesystem timestamp is invalidating the hash.
• •Treating two different artifact sizes as proof of non-determinism, when the actual variation is in metadata, embedded timestamps, or symbol ordering rather than in the compiled code.
• •Mutating the build script to remove timestamps without first confirming the timestamp is read on the input path, which can hide the symptom without addressing the root cause.
• •Promoting a wider glob into the inputs list to fix a single miss, which then causes unrelated tasks to be invalidated together and destroys the reproducibility property.
• •Comparing hashes across machines before stabilizing the toolchain, so that legitimate version drift is recorded as a reproducibility defect.

Safe fixes

• •After decomposing the task hash and identifying a missing input, add the specific path or environment variable to the task's documented inputs so the hash reflects the new dependency on that input.
• •When filesystem metadata is the cause, replace path-based or time-based inclusion with a content-addressed manifest, so the input identity is derived from bytes rather than metadata.
• •When directory walk order is the cause, sort the input list before hashing and document the sort order in the task definition so the order is stable across machines.
• •When toolchain drift is the cause, pin the toolchain via the package manager lockfile and re-verify hash equality with the lockfile-resolved versions before changing any build script.
• •When package boundaries are too broad, prune the inputs glob to the specific package subtree and re-run the cache hit test to confirm the boundary is now narrow enough to be stable.

Prove the fix

1. 01Run the same task twice from the identical commit with the documented inputs and a pinned toolchain, and observe that the two artifact hashes are byte-equal or metadata-equal as determined by the diagnostic classification.
2. 02Restore remote cache for the task and observe that the recorded cache key matches the new task hash, and that a subsequent run with identical inputs reports a hit rather than a miss.
3. 03Introduce a deliberate, isolated change to a documented input and observe that exactly the affected task is invalidated while unrelated tasks remain cached, confirming the input boundary is now precise.
4. 04Record a golden artifact hash for the commit in a known-good manifest and confirm that the post-fix build reproduces it on a second machine without manual intervention.

Prevention and next steps

• •Treat the task inputs specification as a contract: any path, variable, or tool that influences the artifact must appear in the inputs list, and changes to that list require a review of the cache impact.
• •Pin all toolchain components through the package manager lockfile and verify that builds are reproducible from the lockfile alone, without depending on ambient system versions.
• •Run a periodic reproducibility check on a small set of representative tasks, recording hashes into a manifest so drift is detected before it reaches production.
• •Keep package boundaries narrow and well named so that unrelated edits cannot accidentally fall into the input set of an unrelated task.

Safe commands and checks

git diff --quiet <commit-sha> -- . ':!build' ':!dist' && echo SOURCE_IDENTICAL || echo SOURCE_DIFFERS
tar --sort=name --mtime='UTC 1970-01-01 00:00:00' --owner=0 --group=0 --numeric-owner -cf <source-archive>.tar <source-dir>
sha256sum <artifact-A> <artifact-B>
cmp -l <artifact-A> <artifact-B> | head -n 50
turbo run <task-name> --summarize --dry=json 2> <summarize-log>
turbo run <task-name> --force 2> <forced-run-log>
turbo run <task-name> --filter=... 2> <filtered-run-log>