Docker healthcheck flapping: separate process health from probe timing

The symptoms

• •Service alternates between "healthy" and "unhealthy" in `docker ps` over short intervals (seconds to a couple of minutes) with no restart.
• •Upstream orchestrators or load balancers deregister and re-register the container repeatedly, producing connection storms.
• •Application logs show the workload completes successfully each time, yet the daemon still reports a failed probe.
• •Increasing CPU or latency in the container correlates with the unhealthy transitions, suggesting probe timing rather than a logic fault.
• •The container's exit code is 0 on graceful shutdown, but `docker inspect` shows `Status: unhealthy` immediately before exit, blocking ordered drain.

Likely causes

• •Probe interval shorter than the worst-case response time of the health endpoint, so a slow but successful response is counted as a failure.
• •Timeout set below realistic network, GC, warm-up, or cold-cache latency for the process under load.
• •Retries value too low, so a single transient blip (DNS, disk, JIT) cascades into an unhealthy state.
• •`start_period` shorter than genuine warm-up time, so the probe races initialization and reports unhealthy before readiness.
• •Health endpoint shared with readiness or liveness logic, causing the probe to flip on transient dependencies the process can still serve.
• •Healthcheck command itself is flaky: shell parsing, missing tool inside the image, or a port/socket path that differs between runtime and build.

First ten minutes

1. 01Capture the current health state and recent transition log with `docker inspect --format '{{json .State.Health}}' <container>` to see `Status`, `FailingStreak`, `Log` entries with timestamps and exit codes.
2. 02Reproduce the flip deterministically by sending a single probe against the same command the Dockerfile declares, then comparing its exit code and latency to the values shown in the health log.
3. 03Time the in-process health endpoint under representative load (using a load tool or a recorded trace), measuring p50, p95, and p99 latency, to see whether the configured timeout is reachable.
4. 04Map the transition timestamps against application logs and host metrics (CPU, GC pauses, disk I/O) to determine whether each "unhealthy" reading coincides with a slow probe or a real failure.
5. 05Record the configured `interval`, `timeout`, `retries`, and `start_period` from the Dockerfile or compose file, and compare them against the measured readiness latency and worst-case response time.

Evidence to collect

• •`docker inspect` health block: `Status`, `FailingStreak`, and the `Log` array showing each probe's `Start`, `End`, `ExitCode`, and `Output`.
• •Dockerfile or compose file HEALTHCHECK line with the exact options string and command, including any shell wrappers.
• •Measured response time distribution of the in-process health endpoint under the same load the container normally serves.
• •Application-side readiness signal (for example a `/ready` route, a marker file, or a readiness file written by the entrypoint) and the time it takes to flip from "starting" to "ready".
• •Host or container metrics for CPU, memory pressure, GC, and I/O at the timestamps where the state flipped to unhealthy.

Where to look

• •Dockerfile `HEALTHCHECK` directive, or the `healthcheck` block under the service in docker-compose.yml, where interval, timeout, retries, and start_period are declared.
• •`docker inspect <container>` JSON, specifically `State.Health.Status`, `State.Health.FailingStreak`, and the `State.Health.Log` array that records each probe outcome.
• •The application process boundary: the HTTP or TCP handler that the healthcheck command reaches, including any middleware that may delay or fail the response.
• •Container logs (for example `docker logs`) filtered to the timestamps of each `unhealthy` transition, to see whether the workload actually errored or merely slowed.
• •Daemon-level health log on the host, if the daemon writes probe events to its own log, for an independent record of probe start, end, and exit code.

Diagnostic steps

1. 01Read `State.Health.Log` and confirm whether every unhealthy transition has `ExitCode` non-zero, or whether some are timeouts recorded as non-zero even though the process eventually responded.
2. 02Compare each probe's `End - Start` duration to the configured `timeout`; values at or near the timeout indicate the probe is being killed, not the application failing.
3. 03Run the healthcheck command directly inside the container with `docker exec` and time it under realistic load; if it succeeds but exceeds the configured `timeout`, the contract is unstable rather than the process unhealthy.
4. 04Cross-check probe timestamps with application readiness events; if readiness completes after `start_period`, the probe window begins before the process is honestly ready.
5. 05Verify the probe command exists in the image (for example, `wget`, `curl`, or `nc`) and that the target port or socket path matches what the process binds during normal operation.
6. 06Reproduce a single flap cycle in an isolated test container with the same healthcheck options and load profile, then vary one option at a time to identify which parameter triggers the flip.

Common mistakes

• •Treating every unhealthy reading as a logic bug, when the probe timeout is simply shorter than the slowest legitimate response.
• •Reusing a liveness or readiness endpoint as the Docker HEALTHCHECK target, conflating orchestration-level signals with Docker's per-second probe timing.
• •Setting `retries: 1` so that any single slow probe immediately promotes the container to unhealthy.
• •Forgetting that `start_period` only delays the first probe, not subsequent ones; warm caches after the first request can still time out later probes.
• •Assuming the Docker daemon retries the command inside the container's filesystem context; a missing binary, wrong PATH, or different shell in a slim image produces false negatives that look like flapping.

Safe fixes

• •Set `timeout` to a value comfortably above the measured p99 latency of the health endpoint under load, with documented headroom for GC and cold cache.
• •Set `interval` longer than `timeout`, and ensure `interval * retries` exceeds the worst observed stretch of slow responses so transient blips do not cascade.
• •Introduce or lengthen `start_period` to cover genuine warm-up, and gate readiness on a process-internal signal rather than the first probe response.
• •Use a dedicated, lightweight health endpoint that performs a shallow check (process up, event loop responsive, dependencies reachable) instead of a full dependency sweep used by orchestrator readiness probes.
• •Rebuild the image after any change to the HEALTHCHECK directive so the daemon re-evaluates options; verify with `docker inspect` that the new values are recorded under `Config.Healthcheck`.

Prove the fix

1. 01Run the container under representative load for a defined observation window (for example 30 minutes) and confirm `docker ps` shows `healthy` continuously with no transitions.
2. 02Confirm `docker inspect --format '{{json .State.Health}}' <container>` shows `FailingStreak` remains 0 and the `Log` array contains only entries with `ExitCode: 0`.
3. 03Inject a controlled fault (for example, a brief dependency pause) lasting longer than `timeout * retries` and confirm the container transitions to `unhealthy` exactly once, then back to `healthy` after recovery, with no oscillation.
4. 04Verify that a graceful shutdown still records a clean exit and that the last health entry before exit is `healthy`, allowing orchestrators to drain connections in order.

Prevention and next steps

• •Define the healthcheck contract independently from orchestration probes, with documented SLOs for response time and availability of the health endpoint.
• •Keep the healthcheck command minimal and self-contained inside the image, using binaries or built-in shell features that are guaranteed to exist at runtime.
• •Right-size `interval`, `timeout`, `retries`, and `start_period` from measured p99 latency and warm-up data, and revisit them whenever the workload's profile changes.
• •Record healthcheck parameters and their measured latency budgets alongside the Dockerfile so reviewers can spot a probe contract that drifts below real performance.
• •Treat `docker inspect` health fields as a first-class signal in CI, alerting on `FailingStreak > 0` or on non-zero `ExitCode` entries in `State.Health.Log`.

Safe commands and checks

docker inspect --format '{{json .State.Health}}' <container>
docker inspect --format '{{json .Config.Healthcheck}}' <container>
docker ps --format 'table {{.Names}}\t{{.Status}}'
docker logs --since <timestamp> --until <timestamp> <container>
docker exec <container> sh -c 'time <healthcheck-command>'