CI times out while local tests pass: compare runner capacity and waiting resources

The symptoms

• •Job exceeds its declared timeout-minutes and is cancelled mid-step, even though the same suite completes locally in a fraction of the time.
• •Progress logs show long idle gaps between stages, suggesting the runner is waiting on an external resource rather than executing code.
• •Duration drifts across runs: the same workflow takes 8 minutes on Monday and 32 minutes on Wednesday with no code change.
• •Local runs use a dedicated database or container, while CI jobs point at a shared managed service that reports high connection or query latency.
• •Post-job cleanup steps are skipped or truncated, leaving caches, containers, or test artifacts that accumulate between runs.

Likely causes

• •Runner concurrency limit reached: queued jobs wait for an available hosted runner, inflating wall-clock time without any test slowdown.
• •Shared external services (databases, object stores, registries, package mirrors) are contended or throttled, so install and fetch steps block on network I/O.
• •Missing or skipped teardown leaves orphan processes, database connections, or cache locks that slow subsequent steps.
• •Cache key churn causes cache misses on every run, forcing full dependency reinstall and asset download on each job.
• •Resource class mismatch: default runner has less CPU or memory than local, so the same workload takes several times longer.
• •Test suites that rely on local filesystem speed, port availability, or /dev/shm behave differently on ephemeral runner images.

First ten minutes

1. 01Capture the full job log and record the declared timeout-minutes versus the elapsed time at cancellation.
2. 02Identify the step where the wall-clock stops advancing and inspect timestamps before and after that step.
3. 03Check the queue time before the first step ran: a long queue suggests runner capacity exhaustion rather than a slow test.
4. 04List every external service the workflow contacts (registry, database, cache backend, artifact store) and note whether it is shared or job-dedicated.
5. 05Compare resource class labels between local and CI: CPU count, memory, disk type, and any ephemeral storage limit.
6. 06Skim the post section for cleanup hooks, trap handlers, or always() steps that may be skipped after a timeout.

Evidence to collect

• •Per-step start and end timestamps from the CI provider's job timeline view, including queue-to-start duration.
• •Resource class, runner image identifier, and region of the runner that executed the job.
• •Cache hit or miss log lines for dependency caches, and the cache key expression used.
• •Connection or query latency reported by managed services accessed during the job.
• •List of background processes, open file handles, or open ports captured before the runner is torn down.
• •Exit code and signal reported for the cancelled step, plus the exact reason string from the runner.

Where to look

• •CI provider's workflow syntax documentation for timeout-minutes, jobs.<job_id>.runs-on, and jobs.<job_id>.timeout-minutes semantics.
• •Runner provider's concurrency and queueing dashboard, which shows pending and running jobs per runner class.
• •Managed service observability panels (database, registry, object store) for the time window of the failed job.
• •Job artifact or log archive for the post section, where cleanup hooks and always() steps would have run.
• •Repository's workflow file and any reusable workflow, focusing on services blocks, container blocks, and cache steps.

Diagnostic steps

1. 01Compare queue time to step execution time: if queue time dominates, the issue is runner capacity, not test speed.
2. 02For each external dependency, time a single round-trip from inside the runner using the provider's built-in timing rather than a custom script.
3. 03Re-run the same job on a larger resource class and compare end-to-end duration; a near-linear drop indicates CPU or memory starvation.
4. 04Re-run with a fixed cache key to isolate cache miss cost from test cost.
5. 05Inspect the post section to see whether always() or cleanup hooks executed; skipped hooks point to a missing teardown path.
6. 06Run the job on a self-hosted runner with the same OS image and compare durations to isolate hosted-runner specific I/O behavior.
7. 07Re-run after disabling optional integrations (code coverage upload, security scanning, artifact upload) to see whether they are contributing to the wait.

Common mistakes

• •Increasing timeout-minutes without evidence: this only hides a real capacity or I/O problem and inflates queue back-pressure.
• •Assuming local speed reflects CI speed, when the runner image, filesystem, and network path are materially different.
• •Trusting green local tests as proof of correctness and skipping service contention analysis for shared backends.
• •Re-running the same workflow repeatedly without changing a variable, which produces the same timeout and wastes queue capacity.
• •Ignoring the post section because the job timed out, missing the root cause that lives in a skipped cleanup or artifact upload.

Safe fixes

• •If queue time dominates, move the job to a runner class with available capacity or split it into smaller jobs that schedule independently.
• •If shared service latency dominates, replace the shared backend with a job-scoped service container or an ephemeral test database, and add a readiness probe before the test step.
• •If cache misses dominate, pin the cache key to a stable input (lockfile hash) and add a fallback restore-keys list to recover partial hits.
• •If resource class is the constraint, raise the class for the affected job only and re-measure; do not raise it globally without data.
• •If teardown is skipped, move cleanup into an always() step or a post job so it runs even when the main step times out.
• •Add a step that logs free disk, free memory, and open file descriptor count at job start and before the suspected slow step, so future timeouts have evidence.

Prove the fix

1. 01The job completes under its declared timeout-minutes across at least five consecutive runs with no manual intervention.
2. 02Per-step durations are stable within a documented variance band, with queue time representing a small, bounded fraction of total duration.
3. 03Cache hit ratio for dependency caches is high and reproducible, and a forced cache miss no longer pushes the job past the timeout.
4. 04The post section runs to completion on timed-out paths as well, with cleanup hooks and always() steps executing in the log.
5. 05Managed service latency for the job's time window is within the documented SLO for that backend.

Prevention and next steps

• •Document the expected per-step duration in the workflow file as comments, so future timeouts are compared against a baseline rather than a guess.
• •Set timeout-minutes per job and per step, and alert when jobs exceed a threshold such as 1.5x the rolling p50 duration.
• •Use job-scoped services for any backend the tests depend on, and reserve shared services for build-time fetches only.
• •Pin cache keys to lockfile hashes and review them when dependency manifests change, to avoid silent cache invalidation.
• •Track runner utilization and queue depth in the same dashboard as test results, so capacity issues surface alongside flaky test signals.

Safe commands and checks

gh run list --workflow <workflow-file> --limit 10 --json databaseId,conclusion,createdAt,updatedAt,headBranch
gh run view <run-id> --json jobs,createdAt,updatedAt --jq '.jobs[] | {name, startedAt, completedAt, conclusion, steps: [.steps[] | {name, startedAt, completedAt, conclusion}]}'
gh workflow view <workflow-file-or-id> --yaml
grep -nE 'timeout-minutes|runs-on|resources|cache|services|always' .github/workflows/<workflow-file>.yml
stat -f '%m' <cache-restore-path-on-runner> && df -h <workspace-path-on-runner>
ps -eo pid,etime,pcpu,pmem,comm --sort=-pcpu | head -n 20
cat /proc/<pid>/status | grep -E 'VmRSS|Threads|State'
tail -n 200 <runner-log-path> | grep -iE 'timeout|cancelled|signal|killed|queue'