The pattern
The application is correct, but the wiring around it differs between local, CI, and production: a health check that always returns 200, a port mismatch, an empty artifact, a default that only exists on laptops. Deployment bugs are diffs between environments, so the fastest path is to diff the environments.
( 01 )Symptoms
How this failure announces itself.
- warningThe orchestrator restarts services that are actually healthy, or keeps dead ones alive.
- warningThe deployed artifact is empty, stale, or missing files that exist in CI.
- warningA container quietly runs with local defaults because production env injection never happened.
- warningTraffic routes to a port nothing is listening on.
( 02 )First moves
The first ten minutes — establish facts before touching code.
- 1Get a shell (or one-off pod) in the failing environment and inspect reality: env vars, listening ports, and the artifact's file listing.
- 2Diff that reality against a working environment, or against what the manifest claims should be true.
- 3Curl the health endpoints from inside the network and compare with the orchestrator's view of them.
- 4Trace one config value from its source (CI secret, manifest) to the process - find the hop where it drops.
( 03 )Where to look
The code and config that usually owns this bug.
- searchHealth check routes - does liveness accidentally depend on downstream services (that is readiness's job)?
- searchDocker and Kubernetes manifests - ports, env blocks, volumes, and the exact casing of every key.
- searchCI artifact paths - what the build step wrote versus what the deploy step actually uploaded.
- searchRuntime env injection - the orchestrator's env block versus what the process actually received.
( 04 )Common fixes
Fix the cause, then make the regression impossible.
- buildSeparate liveness (process is up) from readiness (dependencies reachable) and wire each to the right probe.
- buildRender and assert manifests in tests so port, env, and path drift fails CI instead of production.
- buildValidate required env on boot and exit nonzero - orchestrators handle crashes better than zombies.
- buildAssert artifact contents (file count, entry points) before deploy, not after rollout.
( 05 )Prove the fix
A fix you can't demonstrate is a guess. Close the loop.
- verifiedDeploy to staging and watch the probes: healthy pods stay up, and a deliberately broken dependency flips readiness without triggering restarts.
- verifiedThe previously failing config value now reaches the process - verified from inside the container, not from the manifest.
- verifiedCI fails when you intentionally break the manifest or empty the artifact.