Deployment is green but health checks fail: separate build success from serving

The symptoms

• •CI or deployment pipeline finishes with a green status badge, but the application's health endpoint replies non-2xx or times out after rollout.
• •Container image or artifact build completes successfully, yet the running instance cannot bind the configured port or crashes during startup before any health probe responds.
• •Ingress or load balancer reports the backend as unhealthy (502/503/504) while build logs and deploy hooks show no error and the rollout is marked successful.
• •Liveness and readiness probes return the requested path but the response body or status does not match what the platform's health check policy expects.
• •Pre-deploy health checks pass on a warm instance, but post-rollout health checks fail because routing, DNS, or readiness gating changed after traffic shifted.

Likely causes

• •Build and runtime configurations diverge: the artifact that passed CI was assembled with environment A settings, but the deployed runtime uses environment B (different config maps, secrets, or feature flags).
• •Health endpoint contract mismatch: the probe calls a path, method, or schema that the running process does not serve, or expects a status code the application does not emit on startup.
• •Startup ordering failure: required dependencies (database, cache, downstream API) are not reachable from the new instance, so readiness never flips to true even though the binary started.
• •Binding failure: the process listens on the wrong interface, the wrong port, or the port collides with another container, so the platform's probe cannot reach it even though the binary is alive.
• •Routing layer stale state: ingress, service mesh, or load balancer holds the old endpoint reference and probes the previous revision, or its health check settings (timeout, threshold, path) disagree with the application.
• •Probe timing misconfiguration: initial delay, period, or failure threshold is shorter than real cold-start time, so the first probes always fail and the instance is marked unhealthy.

First ten minutes

1. 01Confirm the symptom is at the serving boundary, not the build boundary: identify which probe (liveness vs readiness vs external LB) is failing and the exact non-success response or timeout it produced.
2. 02Capture the running instance's view of configuration: read the env vars, mounted config, and resolved secrets actually present in the deployed process, not the ones used during build.
3. 03Check process liveness versus endpoint reachability: verify the binary is running and not crash-looping, then separately verify it is listening on the address and port the platform will probe.
4. 04Reproduce the probe locally with the same path, method, headers, and status expectation the platform uses; compare against what the application actually serves.
5. 05Map the routing chain: trace from external health check -> ingress/load balancer -> service -> pod/container, and note where each hop's view of "healthy" is sourced.
6. 06Record initial-delay, period, timeout, and failure-threshold values used by the platform probe, and compare them to measured cold-start time from logs.

Evidence to collect

• •The deployment identifier, process start timestamp, resolved listen address and port, and the first health-check result after the process started.
• •The platform's probe request and response details, including the target path, status code, latency, and which routing hop marked the instance unhealthy.
• •Runtime logs from cold start through the first failed probe, correlated with the load balancer or ingress health events for the same instance.

Where to look

• •Artifact boundary: the exact build output and release identifier that the deployment promoted, separated from the process that serves it.
• •Process boundary: startup logs, resolved bind address, port, health-handler registration, and dependency initialization state inside the running instance.
• •Routing boundary: ingress, load balancer, service discovery, and platform probe configuration that turns the health response into a traffic decision.

Diagnostic steps

1. 01Classify the failure response code: a connection refused/timeout at the LB indicates binding or routing; a 5xx from the upstream indicates the application responded but rejected readiness; persistent non-response indicates startup or dependency stall.
2. 02Diff build-time and runtime configuration for the same release identifier; any divergence is a candidate root cause until disproven by a passing health check.
3. 03Replay the platform probe verbatim inside the deployed instance (same path, method, headers, expected status) and compare the observed response to what the probe expects.
4. 04Measure cold-start latency from process start to "server listening" log line and compare against the probe's initialDelay; if startup exceeds initial delay, the first probe is guaranteed to fail.
5. 05Trace the routing chain hop by hop: external LB -> ingress -> service -> pod IP -> container port, and identify the first hop that returns a non-success verdict.
6. 06Inspect dependency readiness: verify each required downstream is reachable from the new instance and that readiness logic gates on dependency state, not just process liveness.
7. 07Compare current probe spec against a known-good previous release; if only the spec changed, the application contract is the suspect, not the binary.

Common mistakes

• •Treating a green pipeline as proof of serving correctness; the pipeline validates artifact creation, not runtime behavior under the platform's probe contract.
• •Curling the health endpoint from a workstation and assuming it represents what the platform probes; the probe origin, network path, and headers often differ.
• •Restarting the deployment repeatedly without changing anything, hoping for a transient race to clear; this masks timing misconfiguration and burns evidence.
• •Editing probe failure thresholds upward to silence alerts; this hides real readiness gaps and converts a recoverable incident into a serving outage.
• •Assuming the load balancer is wrong because the application "starts"; in many cases the application never reaches a state where its health endpoint reflects true readiness.
• •Rolling back the artifact when the root cause is a runtime configuration or routing change that will reappear on the next deploy of any artifact.

Safe fixes

• •If evidence shows build-time vs runtime configuration divergence, correct the runtime configuration source so the deployed process sees values consistent with what was tested in CI, then redeploy without altering the artifact.
• •If evidence shows the probe contract disagrees with the served endpoint, align the probe's path, method, and expected status code with the application's actual health response, or update the application to serve the contracted endpoint.
• •If evidence shows cold-start exceeds probe initialDelay, raise initialDelay to a value strictly greater than observed worst-case startup, and re-verify with a fresh rollout that the first probe lands after the server is listening.
• •If evidence shows binding to the wrong interface or port, fix the application's listen address (or the platform's published port mapping) so the probe target matches an actually open socket inside the instance.
• •If evidence shows a dependency stall blocks readiness, introduce or restore an explicit readiness signal that flips only after required dependencies are confirmed reachable, rather than relying on process liveness alone.
• •If evidence shows ingress or LB health check settings disagree with the application, update the LB probe definition (timeout, threshold, path) to match the application's contract; do not weaken thresholds to mask failures.

Prove the fix

1. 01After a fresh rollout with no warm-up, the platform's readiness probe transitions to passing within the configured period and the load balancer reports the backend as healthy without operator intervention.
2. 02The probe response captured from the platform's origin (not a side channel) matches the application's expected health status code and body schema, recorded across at least one full probe period.
3. 03A repeat rollout that intentionally restarts the process from cold shows the same passing sequence, demonstrating the fix is not dependent on warm caches or prior traffic.
4. 04Runtime configuration observed inside the deployed instance matches the configuration that was validated in CI for the same release identifier, verified by reading the resolved env or config inside the process.
5. 05Dependency readiness gates remain stable across a forced dependency restart: the new instance's health endpoint reflects dependency state rather than reporting ready prematurely.
6. 06Routing chain logs show a consistent verdict at every hop for the same probe request, with no hop disagreeing about the backend's health status.

Prevention and next steps

• •Treat build success and serving correctness as separate gates: require an explicit runtime smoke that exercises the platform's actual probe path against a freshly started instance before declaring a release healthy.
• •Version configuration alongside the artifact and fail the rollout if the resolved runtime configuration diverges from the configuration that was validated during build.
• •Keep probe definitions and application health contracts in a single reviewed source so path, method, and expected status cannot drift between platform spec and served code.
• •Measure and record cold-start latency per release; set probe initialDelay from observed worst-case startup plus a margin, and alert when startup trends cross the configured delay.
• •Use readiness signals that reflect dependency state, not just process liveness, so a started binary that cannot serve traffic is correctly marked unready.

Safe commands and checks

ps -o pid,etime,cmd -p <pid>
cat /proc/<pid>/environ | tr '\0' '\n' | grep -E '^(HEALTH|PORT|BIND|ADVERTISED_)'
ss -ltnp | grep <pid>
kubectl describe pod <pod-name> | sed -n '/Liveness:/,/Events:/p'
kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[*].state}'
kubectl logs <pod-name> --previous --tail=200
kubectl get endpoints <service-name> -o yaml
diff <(kubectl get configmap <cm-build> -o yaml) <(kubectl get configmap <cm-runtime> -o yaml)