What this usually means
Silent pipeline failures typically fall into one of three categories: JVM-level crashes (OOM, segfault) that kill the agent before it can flush logs; Groovy CPS deserialization errors that cause the pipeline to abort before reaching error handlers; or pipeline step timeouts that expire without a visible exception because the step never yielded control back. When you see a build that shows 'Failure' but the last console line is a normal echo or a successful shell command, that's a strong indicator the failure happened outside the Groovy sandbox — in the JVM itself or in a plugin's internal state machine.
The first ten minutes — establish facts before touching code.
- 1Check Jenkins master log (`/var/log/jenkins/jenkins.log` or System Log in UI) for OOM errors or plugin exceptions at the exact build timestamp
- 2Look at the agent's JVM logs: if using SSH agents, check `/var/log/jenkins-slave/` or the agent's stdout file for crashes
- 3Enable pipeline step debug: add `enableDiagnosticReporting=true` to `JENKINS_OPTS` and re-run the build to get additional logging
- 4Use Pipeline Syntax → 'Declarative Directive Generator' to add `options { timestamps() }` and `options { timeout(time: 5, unit: 'MINUTES') }` to narrow down where the hang occurs
- 5Run the build with a minimal Jenkinsfile (remove stages one by one) to isolate the failing step
- 6Check plugin compatibility: go to Manage Jenkins → Plugin Manager → Updates and ensure all plugins are on supported versions
The specific files, logs, configs, and dashboards that usually own this bug.
- search`/var/log/jenkins/jenkins.log` — master-side errors, plugin crashes, OOMs
- searchAgent stdout/stderr (e.g., `java -jar agent.jar` terminal output or `~/.jenkins/remoting/logs/`) — agent JVM crashes or network disconnects
- searchPipeline stage view in UI — if a stage shows 'Running' but no log lines, it's likely a hang
- search`${JENKINS_HOME}/jobs/<job>/builds/<build>/log` — raw console log file (may contain truncated exceptions that UI clips)
- searchScript Approval page (`/scriptApproval/`) — if your pipeline uses `build()` or other methods that require approval, the failure may be a silent rejection
- searchThread Dump from Manage Jenkins → System Information → Thread Dumps — if the build is hanging, look for threads in `BLOCKED` or `WAITING` state on pipeline locks
Practical causes, not theory. These are the things you will actually find.
- warningGroovy CPS deserialization error after a restart — pipeline state is saved but the class definition changed
- warningJVM Metaspace exhaustion on the agent due to long-running builds loading many classes
- warningPipeline step that calls a plugin method that throws an undeclared checked exception, aborting the pipeline without printing
- warningNetwork timeout between master and agent during `sh` or `bat` step — the step never returns, so no error is logged
- warningInsufficient heap for the agent JVM causing `OutOfMemoryError` that kills the process before writing to stderr
- warningPipeline library (shared library) that has a syntax error in a method imported at runtime — the error only appears on execution, not validation
Concrete fix directions. Pick the one that matches your root cause.
- buildAdd `options { retry(3) }` around flaky stages to tolerate transient agent disconnects
- buildSet explicit JVM heap limits for agents: add `-Xmx512m -Xms256m` to the agent startup command to prevent OOM
- buildUse `wrap([$class: 'AnsiColorBuildWrapper'])` or `catchError` to ensure exceptions are captured and printed
- buildPin plugin versions in your Jenkins configuration-as-code (JCasC) to prevent incompatible updates
- buildReplace `sh` with `sh(returnStdout: true)` combined with `timeout` to detect dead commands
- buildMigrate from declarative pipeline to scripted pipeline with explicit try-catch blocks for better error handling
A fix you cannot prove is a guess. Close the loop.
- verifiedRe-run the build with the fix and check that the stage that previously failed now shows a clear error message
- verifiedVerify that the console log ends with 'Finished: SUCCESS' or the expected error, not an abrupt truncation
- verifiedCheck that agent logs show no OOM errors or unexpected thread exits for at least 10 consecutive builds
- verifiedRun a stress test: trigger 5 concurrent builds of the same job and confirm no silent failures
- verifiedAdd a `post { always { echo 'Build ended' } }` block and confirm it always appears — if it doesn't, the agent died
Things that make this bug worse or harder to find.
- warningDon't restart Jenkins without checking the pipeline state — you may lose the half-finished build entirely
- warningDon't disable the Workflow CPS plugin globally; it's required for pipelines. Instead, update it to the latest version
- warningDon't ignore plugin update warnings — a plugin that is 'incompatible' with your Jenkins version can cause silent failures
- warningDon't assume a 'Process killed' message means external intervention — it's often the agent JVM crashing internally
- warningDon't set `timeout` on a stage that contains `input` step — the timeout will not interrupt the input prompt
The Phantom Failure: A Jenkins Build That Died Without a Trace
Timeline
- 09:15Deploy job triggered for service 'payment-api' — build #184
- 09:17Stage 'Build Docker Image' shows green, last log line: 'Successfully tagged payment-api:latest'
- 09:18Stage 'Push to Registry' starts — no output for 90 seconds
- 09:19Build marked as FAILURE. Console log ends with 'Successfully tagged payment-api:latest' — no error message
- 09:20Team checks Jenkins master log — finds 'java.lang.OutOfMemoryError: Metaspace' at 09:18:45
- 09:21Checked agent pod logs — agent JVM exited with exit code 1 due to Metaspace
- 09:30Applied fix: increased agent JVM metaspace to 256m and added -XX:MaxMetaspaceSize=256m
- 09:35Re-ran build #185 — all stages succeed, console log shows clean exit
I was paged for a failing deploy job that looked normal. The console showed 'Successfully tagged' and then nothing — the build just died. The team thought it was a Kubernetes pod eviction, but no events showed that. I checked the Jenkins master log and saw 'OutOfMemoryError: Metaspace' right at the timestamp. The agent pod had been running for 48 hours and had accumulated classloaders from multiple builds.
The agent startup command had no memory limits. I added `-Xmx512m -Xms256m -XX:MaxMetaspaceSize=256m` to the JVM args. But the real fix was to add a `timeout` on the push stage so that if the agent dies, the pipeline doesn't hang forever. I also added a `catchError` block around the push step to capture the exit code.
The lesson: never trust a console log that ends cleanly. Always check the master log and agent logs for JVM errors. And set explicit memory limits on agents, especially in Kubernetes where pods can be long-lived.
Root cause
Agent JVM exhausted Metaspace because no limit was set, causing the agent process to die silently without flushing logs.
The fix
Added -Xmx512m -Xms256m -XX:MaxMetaspaceSize=256m to agent JVM arguments and added a stage timeout of 5 minutes.
The lesson
Always set memory limits on Jenkins agents and add timeouts to pipeline stages to detect silent agent deaths.
When a Jenkins pipeline is resumed after a restart, Workflow CPS serializes the Groovy stack and reloads it. If the class definition of a step or shared library method has changed, you get a `ClassCastException` or `NoSuchMethodError` that is swallowed by the pipeline engine. The build fails with no console output because the error occurs during deserialization, before any step runs.
To catch this, add a `try-catch` at the pipeline root: `try { // pipeline } catch (Exception e) { echo "Pipeline failed: ${e}"; throw e }`. Also, check the Jenkins log for 'CPS deserialization' errors. Pin your shared library versions to prevent class drift.
A JVM crash (SIGSEGV, OOM, StackOverflow) kills the agent process immediately. The pipeline sees a broken channel and marks the build as failure, but the console log only shows content up to the last flush. No error message appears because the agent never sent one. This is especially common in Kubernetes ephemeral agents that run out of memory.
Enable JVM crash logging: add `-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/heapdump.hprof` to agent JVM args. Also, monitor agent memory usage via Prometheus or `kubectl top pod`. Set resource limits in the pod template: `containerTemplate(..., memory: '512Mi')`.
Some pipeline steps, like `input` or `sh` with a long-running command, are not interruptible by the `timeout` option if the underlying plugin doesn't support it. You get a build that hangs indefinitely with no log output. The console shows 'Running' on that stage but no new lines.
Use `timeout` at the stage level with `activity` type: `options { timeout(time: 10, unit: 'MINUTES', activity: true) }`. This kills the step if no output is received for the specified duration. For `sh` steps, use the `timeout` command inside the shell: `sh 'timeout 30 ./long-script.sh'`.
A plugin update can introduce an API change that breaks your pipeline without any error in the console. The plugin throws an `AbstractMethodError` or `LinkageError` that the pipeline interpreter catches but doesn't print. You'll see the failure in the Jenkins log under 'severe' or 'warning'.
Use the Jenkins Plugin Manager's 'Check Now' to verify compatibility. Set up a test pipeline that runs after each plugin update. Use JCasC to freeze plugin versions in production. If you suspect a plugin, disable it temporarily and rerun the pipeline.
Frequently asked questions
Why does my Jenkins pipeline fail but the console log shows no error?
Common reasons: 1) Agent JVM crash (OOM or segfault) that kills the process before it can send error output; 2) Groovy CPS deserialization error after a restart; 3) Pipeline step that throws an unchecked exception that is swallowed by the engine. Check the Jenkins master log (`/var/log/jenkins/jenkins.log`) and agent logs for clues.
How do I debug a pipeline that hangs on a stage?
First, add `options { timestamps() }` to see when the last log line was printed. Use `options { timeout(time: 5, unit: 'MINUTES', activity: true) }` to auto-kill the stage if no output. Then, take a thread dump from Manage Jenkins → System Information → Thread Dumps and look for threads in `BLOCKED` or `WAITING` state on pipeline locks.
What is the Workflow CPS plugin and why does it cause silent failures?
Workflow CPS (Continuation Passing Style) is the plugin that serializes and restores the Groovy pipeline state. It can cause silent failures when class definitions change between runs, leading to deserialization errors. The error is logged in Jenkins master log but not in the job console. Fix by pinning plugin versions and avoiding dynamic class loading in shared libraries.
How do I prevent Jenkins agent OOM crashes?
Set explicit JVM memory limits when starting the agent: `-Xmx512m -Xms256m`. In Kubernetes pod templates, set resource limits: `containerTemplate(name: 'jnlp', memory: '512Mi', cpu: '500m')`. Also, add `-XX:+HeapDumpOnOutOfMemoryError` to capture heap dumps for analysis.
Can a plugin update cause my pipeline to fail silently?
Yes. A plugin update may change its API, causing `NoSuchMethodError` or `AbstractMethodError` that is not printed to the console. Always test pipeline jobs after plugin updates. Use Jenkins Configuration as Code to lock plugin versions in production.