CI-only failure debugging checklist

The symptoms

• •The same commit and test command exits 0 on a developer's laptop but exits non-zero on the remote CI runner, producing a red pipeline status.
• •Failure logs from the runner reference files, paths, environment variables, or binaries that do not exist on the local machine used for the green run.
• •The pipeline failure only appears on a specific runner OS (for example ubuntu-latest vs windows-latest) or only after the runner caches or restores a previous job's state.
• •Re-running the failed job succeeds intermittently while the underlying code has not changed, suggesting non-deterministic runner state rather than a deterministic code defect.
• •The local environment was set up manually (brew install, apt-get, manual PATH edits) instead of through the workflow's setup steps, masking missing provisioning from the CI definition.

Likely causes

• •Missing or implicit dependencies on the local machine that are not declared in the workflow's install/setup steps, so the runner never installs them and the command fails at first use.
• •Environment variable differences: secrets or values injected via the CI configuration are absent locally, or local shell exports are missing in the runner, changing behavior of tools that read configuration from the environment.
• •OS, shell, or filesystem case-sensitivity divergence between the developer's OS and the runner image, especially when paths, globs, or archive entries are case-sensitive on Linux runners but case-insensitive on macOS or Windows.
• •Stale caches or restored build artifacts that are out of sync with the current commit, causing old binaries, compiled outputs, or lock files to be reused and produce mismatched results.
• •Time, locale, or timezone-sensitive logic: tests that assume local wall-clock time, a fixed timezone, or a specific locale will diverge on runners that default to UTC and the POSIX locale.
• •Network differences: the runner can or cannot reach external services the developer's machine can, or DNS, proxy, or TLS certificate stores differ between the two environments, causing fetches to fail only in CI.

First ten minutes

1. 01Capture the exact failing step name, the command it ran, and the runner OS label from the CI run summary before touching any code, so subsequent comparisons reference a stable identifier.
2. 02Open the job log for the failing step and isolate the first non-zero exit line and the last successful line, then copy those two lines plus 5–10 lines of surrounding context into a local note for diffing.
3. 03Reproduce the failing step locally using the identical command shown in the log, ideally inside a clean shell with a minimal environment (no interactive rc files), and record whether the exit code matches the runner.
4. 04Diff the effective environment between local and runner for PATH, HOME, LANG/TZ, and any tool-specific variables the failing command consults, listing only the deltas rather than full dumps.
5. 05Diff the dependency manifest versus the workflow's install step, and flag any tool, version, or system library the manifest assumes but the workflow does not install.
6. 06Disable caches and re-run the failing job once to determine whether the failure is tied to restored cache state rather than the current commit, before modifying project code.

Evidence to collect

• •The job log of the failing step, including the exact command line, exit code, and the first stack trace or error message emitted before exit.
• •The workflow file (for example .github/workflows/*.yml) for the job: runner OS label, setup/install steps, environment variables declared, and cache keys used.
• •A local environment snapshot for PATH, HOME, LANG, TZ, and tool-specific variables, captured in the same shell where the local green run succeeds.
• •The dependency manifest (lockfile, requirements, package.json, go.mod, etc.) and the list of system packages the workflow's setup step actually installs.
• •The cache key, cache hit/miss status, and the path scope of any cache action used in the job, to test whether restored artifacts match the current commit.

Where to look

• •The runner OS image boundary: the job runs on a fresh virtual environment whose contents are defined entirely by the workflow's setup steps; anything not installed there is absent, even if present on the developer's machine.
• •The workflow file boundary: each step declares its own shell and working-directory, so a passing local shell script may execute under a different shell or directory on the runner.
• •The environment variable boundary: secrets and vars injected at job or step scope are visible to the step but not to local shells, and vice versa, which is the most common source of behavior divergence.
• •The cache and artifact boundary: restored caches and downloaded artifacts carry state from previous jobs or branches and can override outputs produced by the current commit's steps.
• •The external network boundary: runner egress can be restricted, can resolve DNS differently, or trust different certificate authorities than the developer machine, causing fetches and TLS handshakes to fail only in CI.

Diagnostic steps

1. 01Add a temporary diagnostic step that prints the effective shell, working directory, PATH, and key tool versions, then compare the output to the same command run locally to localize divergence to a specific variable.
2. 02Re-run the failing job with caches disabled; if the failure disappears, the divergence is in restored cache contents, not in the current commit's code or workflow.
3. 03Re-run the failing job on a different runner OS label that the project also supports; if the failure is OS-specific, the cause is OS-dependent tooling, paths, or case-sensitivity rather than the workflow structure.
4. 04Pin dependency versions in the workflow's install step to the exact versions present on the local machine that produced the green run, and re-run; a green run under pinning isolates the cause to unpinned or drifting versions.
5. 05Run the same step locally inside a minimal container that mirrors the runner image (using the same base image tag) to reproduce the runner's filesystem; a reproduced failure locally confirms the cause is environmental, not workflow-only.
6. 06Inspect the failing command's exit code and signal: an exit code from the tool itself indicates the tool ran and decided to fail, while a signal such as SIGSEGV or SIGKILL indicates the runner terminated the process, often for memory or time limits.

Common mistakes

• •Editing application code to work around a missing dependency, instead of adding the dependency to the workflow's install step, which leaves the workflow definition inaccurate and the local-green/CI-red gap unresolved at its source.
• •Assuming the local shell and the runner shell behave identically; they often differ in default shell, login vs non-login, and whether rc files are sourced, producing divergent PATH and alias state.
• •Blaming the tool version when the actual divergence is case-sensitivity, line endings, or path separators between the developer's OS and the Linux runner image.
• •Clearing caches without changing the cache key or path scope, which lets the next run restore the same stale state and produce the same misleading failure.
• •Adding retry loops around network calls without diagnosing whether the runner can reach the endpoint at all, masking the real boundary failure behind intermittent success.

Safe fixes

• •If the diff shows a missing tool or library on the runner, add the explicit install step to the workflow for the failing job and re-run, only after confirming the tool is genuinely absent from the runner image rather than simply not on PATH for that step.
• •If the diff shows divergent environment variables, declare the required variables explicitly in the workflow at job or step env scope and remove any reliance on locally-exported shell variables, then re-run to confirm green.
• •If cache state is the suspect, change the cache key to include a hash of the lockfile or a relevant input so restored contents match the current commit, then re-run; do not simply delete the cache without updating the key.
• •If the failure is OS-specific, either add a runner-matrix entry to cover the failing OS with a corrected setup, or document and drop the unsupported OS so the matrix reflects only validated environments.
• •If time, locale, or timezone is implicated, set the relevant variable explicitly in the workflow step (for example TZ=UTC, LANG=C.UTF-8) and remove any test code that reads local wall-clock time without injecting these values.

Prove the fix

1. 01The previously failing job returns exit code 0 in the CI run summary for the same commit that previously produced the red status, with no manual re-runs required.
2. 02A subsequent commit that changes only unrelated code also produces a green run, demonstrating that the fix is not a one-shot artifact of a particular cache state.
3. 03If the cache key was changed, the cache hit/miss log shows the new key pattern in use and the restored contents correspond to the current lockfile, not to a prior branch or commit.
4. 04The diagnostic print step, if added during triage, is removed or moved behind a conditional flag so it does not leak environment values into routine job logs.
5. 05If an OS-specific failure was the cause, the same commit is green across the full matrix of runner OS labels declared for the job, confirming the boundary is closed for every supported environment.

Prevention and next steps

• •Keep the workflow's install steps as the single source of truth for tool and library provisioning, and document in the repository README which local prerequisites are required so developers and the runner start from the same baseline.
• •Pin dependency versions in the workflow install steps and use lockfiles committed to the repository, so the runner and local machines converge on identical versions without manual coordination.
• •Use cache keys that include a hash of the relevant lockfile or input set, and scope cache paths narrowly, so restored contents cannot outlive the inputs that produced them.
• •Run a periodic job that intentionally executes the project's primary commands inside a clean container image matching the runner, so environmental drift is detected before it surfaces as a CI-only failure.

Safe commands and checks

env -i PATH=/usr/bin:/bin HOME=$HOME LANG=C.UTF-8 TZ=UTC <command-from-failing-step>  # run the failing step's command under a minimal environment to test for reliance on inherited shell state
git rev-parse HEAD  # capture the exact commit SHA under test for reproducible comparison between local and CI runs
grep -nE 'runs-on|setup-.*|actions/cache|env:' .github/workflows/*.yml  # list runner labels, setup actions, cache actions, and env declarations from the workflow files
diff <(printenv | sort) <(remote-runner-env-snapshot | sort)  # diff local versus runner environment, after capturing the runner snapshot via a temporary diagnostic step
stat -c '%n %i' <path-from-failing-log>  # inspect inode and path of a file referenced in the failure to detect case or path-separator divergence on the runner
date -u +%Y-%m-%dT%H:%M:%SZ  # confirm the runner's UTC clock so time-sensitive tests can be compared against local wall-clock assumptions