CI passes locally but fails remotely: compare execution context

The symptoms

• •Tests or build steps pass on a developer laptop with the same commands but the CI workflow reports failures on the same commit.
• •CI fails only on certain runners (for example, ubuntu-latest vs macos-latest) while passing on others or locally on a similar OS.
• •Error messages in CI logs reference missing tools, modules, files, or permissions that are present locally.
• •CI passes on the first job and fails on a downstream job that depends on artifacts uploaded by the first.
• •Failures are intermittent or timing-related in CI but never reproduce during a single local run.
• •Network-dependent steps (package install, API calls, container pulls) time out or fail in CI but succeed locally.

Likely causes

• •Dependency drift: the remote runner uses a different language runtime version or locked package set than the local machine.
• •Missing or mis-scoped secrets and environment variables: values present in the developer's shell are not declared in the workflow file's env or secrets.
• •Working-directory or repository-root mismatch: a relative path resolves differently when CI checks out into a workspace with a different layout.
• •OS and shell differences: line endings (LF vs CRLF), case-sensitive vs case-insensitive filesystems, GNU vs BSD userland tools, and POSIX vs PowerShell semantics.
• •Permissions and user identity: the runner executes as a different UID, in a different sandbox, or with different filesystem ACLs than the developer.
• •Concurrency and timing: parallel jobs, shared caches, or rate-limited services expose race conditions that a single local run masks.
• •Resource limits: the runner has less CPU, memory, or disk than the local machine, surfacing OOM or disk-full errors.

First ten minutes

1. 01Open the failed CI run and read the failing step's log end-to-end; note the exact command, exit code, and error line rather than the summary status.
2. 02Confirm the failure is on the same commit SHA you tested locally; rule out a stale local branch or unpushed change.
3. 03List what your local shell has that the workflow does not declare: tools, env vars, dotfiles, daemons, and any background services.
4. 04Capture the runner's effective context: OS, runner image label, Node/Python/Java versions, and any actions referenced in the workflow.
5. 05Reproduce the same command from CI in a clean local shell (no aliases, no exported vars, fresh terminal) to establish a baseline gap.
6. 06Identify the smallest CI job and step that fails, and treat that as the boundary between "matches local" and "differs from local."
7. 07Decide which axis (dependencies, env, files, OS, permissions, timing) is most plausible given the error text before changing anything.

Evidence to collect

• •The exact workflow file contents under .github/workflows/ and any reusable workflows it references.
• •The full log of the failing step, including the runner image label printed at job start and any set -x or debug output enabled via ACTIONS_STEP_DEBUG.
• •Local runtime versions for every tool invoked by the workflow (for example, node --version, python --version, go version).
• •The contents of lockfiles and version pins (package-lock.json, poetry.lock, go.sum, Gemfile.lock) and any .nvmrc, .python-version, or tool-versions file in the repo.
• •A diff of environment variables between the local shell and the CI job, restricted to names not containing secrets.
• •Output of file or stat on suspected binary or path-sensitive files, including line-ending bytes if the failure mentions parsing or shebangs.

Where to look

• •The workflow YAML under .github/workflows/ for env, defaults, permissions, and uses clauses that shape the job environment.
• •The job header and step "Set up" lines for the chosen actions, which report the runner image and tool versions actually installed.
• •Repository files that pin tooling: .nvmrc, .python-version, .tool-versions, .ruby-version, and language-specific lockfiles committed to the repo.
• •The CI run's "Annotations" and "Artifacts" tabs for uploaded logs, coverage reports, or crash dumps produced only on failure.
• •Settings for the repository's Actions secrets and variables, to confirm which names are available to the workflow at runtime.
• •The runner's documented environment matrix in the GitHub Actions documentation to compare against what you have locally.

Diagnostic steps

1. 01Read the workflow file top to bottom and write down, in order, every command the failing step runs; this is the contract the runner follows.
2. 02Execute the same sequence in a clean local shell with a fresh login session, removing aliases, exports, and shell startup files, and compare the exit code of each command.
3. 03Diff the workflow's pinned runtime versions against the local toolchain versions; any non-zero delta is a candidate cause.
4. 04Inspect declared vs undeclared environment: every name referenced in CI but missing from the workflow's env block is a likely missing input.
5. 05Check working-directory and repository layout: confirm whether checkout runs with lfs, sparse-checkout, or submodules, since these alter what files exist on disk.
6. 06For OS-sensitive failures, run the step on each runner label the workflow uses (ubuntu-latest, windows-latest, macos-latest) to localize the divergence.
7. 07For timing-sensitive failures, enable ACTIONS_STEP_DEBUG, rerun the job, and capture timestamps at step boundaries to detect stalls or rate limits.
8. 08For permission-sensitive failures, add a debugging step that runs id, pwd, and ls -la on the working directory to record the runner's effective identity and ACLs.

Common mistakes

• •Assuming the local green run proves correctness and treating the CI red as flaky noise, instead of treating the workflow file as the only authoritative spec.
• •Re-running the CI job hoping it will pass, without first identifying which environmental axis differs from the developer machine.
• •Pinning an action by mutable tag rather than by commit SHA, so a third-party change silently alters behavior between local reasoning and remote execution.
• •Hiding secrets behind local .env files that are gitignored, then forgetting to declare the same names in the workflow's env or repository secrets.
• •Mixing shell dialects between local and CI, for example writing a script that uses bashisms locally but executing it via sh on the runner.
• •Trusting GUI test runners locally that mask filesystem casing, CRLF endings, or path separators which the CI runner handles strictly.
• •Mutating shared caches (npm, pip, Docker) on CI in a way that is not represented locally, creating drift that only appears on remote runners.

Safe fixes

• •Add explicit working-directory and shell declarations on every step so paths and dialect are unambiguous regardless of runner.
• •Pin language runtimes using repo-committed files (.nvmrc, .python-version, .tool-versions) and reference them via official setup actions so local and CI versions converge.
• •Reference third-party actions by full-length commit SHA, not by tag, to lock the behavior the workflow performs remotely.
• •Declare all required env names in the workflow's env or secrets block; never rely on ambient shell state from the developer machine.
• •Add a diagnostic step that runs uname -a, node --version or the relevant runtime version command, and id, then uploads the output as an artifact for future debugging.
• •Normalize line endings with a .gitattributes file so CRLF/LF differences cannot silently change script parsing between platforms.
• •Run sensitive steps inside a container or service whose image is pinned by digest, so the local Dockerfile and the CI job resolve to the same bytes.

Prove the fix

1. 01The previously failing step exits with code 0 on at least three consecutive CI runs against the same commit, without any code change between attempts.
2. 02A diagnostic artifact captured from the job shows the same runtime versions, working directory, and effective user as the developer machine, or a documented intentional difference.
3. 03The same shell command sequence, when executed in a clean local shell with no exported variables, also exits 0, demonstrating parity with the workflow.
4. 04All env names referenced by the failing step are present in the workflow's env block or repository secrets, verified by a printed diff during the run.
5. 05If the failure was timing-related, repeated reruns under ACTIONS_STEP_DEBUG show no step exceeding the documented runtime budget and no rate-limit responses from external services.

Prevention and next steps

• •Keep the workflow file as the single source of truth: every tool, version, and env name it uses must be declared there, not implied by the developer's shell.
• •Commit lockfiles and runtime version files, and configure dependabot or equivalent to keep them current without silent drift.
• •Use a devcontainer or Codespaces configuration that mirrors the CI runner image, so local commands and CI commands share the same filesystem and tool versions.
• •Reference all third-party actions by commit SHA and review updates deliberately, rather than tracking floating tags.
• •Add a minimal smoke-test job that runs on every push and fails fast on environmental drift, so local/CI mismatches surface before they reach the main pipeline.

Safe commands and checks

uname -a && node --version && python3 --version # record local toolchain versions to compare against CI job header
git rev-parse HEAD # confirm the exact commit SHA being tested locally matches the CI run
git status --porcelain # verify there are no uncommitted or unpushed changes that could mask the divergence
env | sort > /tmp/local-env-sorted.txt # capture non-secret local env vars; redact any line whose name contains TOKEN, SECRET, KEY, or PASSWORD before sharing
cat .nvmrc 2>/dev/null; cat .python-version 2>/dev/null; cat .tool-versions 2>/dev/null # show repo-pinned runtime versions used by setup actions
file <path-to-failing-script-or-binary> # identify whether the file has CRLF line endings or a Windows shebang that the CI runner rejects
ls -la # inside the CI step, record the working directory permissions and ownership to compare with local stat output
grep -nE 'uses:|with:|env:|working-directory:|shell:' .github/workflows/<workflow-file>.yml # list every external dependency the workflow introduces