API circuit breaker open: separate dependency recovery from caller pressure

The symptoms

• •All outbound calls to a specific dependency fail within milliseconds with a client-local fast-fail error, while the dependency itself returns 200 or 502 on parallel probes from other clients.
• •Breaker metrics show state OPEN or HALF_OPEN with an elevated failure ratio and a non-zero recent request count, while the same window shows successful health probes from independent clients.
• •Logs contain repeated fast-fail events with no upstream TCP connection attempts for the protected dependency, and a separate stream shows successful probes hitting the same hostname and port.
• •Latency for the protected dependency drops to a flat near-zero floor instead of the dependency's normal response time distribution, because requests never leave the client.
• •Upstream returns HTTP 502 Bad Gateway for the dependency, but the client never observes 502 itself; it observes its own short-circuit marker instead.

Likely causes

• •The breaker opened on caller-side metrics (timeout, 5xx ratio, slow-call ratio) and has not yet satisfied its sleep-window or probe threshold to transition to half-open.
• •The dependency is healthy for low-volume probes but still degraded under the caller's traffic shape (concurrency, payload size, header volume), so the breaker correctly stays open against this caller.
• •Shared breaker scope across multiple endpoints causes one endpoint's failure burst to short-circuit unrelated endpoints, masking the actual recovery of the target dependency.
• •Breaker thresholds are tuned to a dependency whose error budget is exhausted (e.g., upstream returns 502 on overload), so the breaker treats caller pressure as dependency failure.
• •Retry policy inside the breaker scope multiplies failure events, pushing the failure ratio past the threshold even though the underlying call count is small.

First ten minutes

1. 01Identify which client, process, and dependency are protected by the open breaker; record the breaker name and scope from configuration, not from inference.
2. 02Read the breaker's current state, opened-at timestamp, failure-window size, sleep window, and permitted half-open probe count from its metrics endpoint.
3. 03Confirm whether requests are leaving the client at all: check connection-level metrics (TCP connect attempts, TLS handshakes) for the dependency target, not just application logs.
4. 04Issue a controlled probe from a separate client (different process or different credentials) against the same dependency path and status codes to establish an independent recovery baseline.
5. 05Compare the breaker's failure-definition (timeout value, 5xx list, slow-call threshold) against the dependency's current response profile before touching any threshold.
6. 06Capture the caller's traffic shape (concurrent in-flight, request rate, payload size) for the window in which the breaker tripped, since shape often explains why the breaker disagrees with probes.
7. 07Decide between dependency recovery (probe shows 2xx, breaker logic should advance to half-open on its own) and caller pressure (probe is fine but caller shape differs) before any configuration change.

Evidence to collect

• •Breaker state-transition log lines with timestamps, state names (CLOSED, OPEN, HALF_OPEN), and the counters that triggered each transition.
• •Connection-level counters: outbound TCP connect attempts, TLS handshake successes, DNS resolution results, and socket-read timeouts for the protected dependency hostname.
• •Independent probe results from a non-circuited client against the same path and method, including status code, total time, and connection reuse behavior.
• •Caller-side traffic shape for the trip window: in-flight count, request rate, p50/p95/p99 latency, payload size distribution, and retry count per logical request.
• •Upstream responses recorded at a proxy or sidecar in front of the dependency, including 502 Bad Gateway occurrences and the time-to-first-byte for successful responses.
• •Configuration snapshot of the breaker: failure threshold, slow-call threshold and duration, sliding window type and size, sleep window, and permitted half-open probe count.

Where to look

• •The client process boundary: the resilience library or middleware that owns the breaker (its metrics exporter and its internal state machine), not the application business logic.
• •The network boundary between the client and the dependency: connection-pool metrics, DNS cache, and TLS session reuse counters scoped to the dependency's hostname and port.
• •The dependency's own ingress boundary: the proxy or gateway that fronts it, where 502 Bad Gateway responses originate when the dependency is overloaded.
• •The configuration boundary: the file, environment variable, or service registry entry that defines breaker thresholds, scope, and which routes share the breaker.
• •The observability boundary: dashboards and log streams that include the breaker name as a dimension, so independent probes can be filtered separately from caller traffic.

Diagnostic steps

1. 01Confirm the breaker is actually OPEN by reading the state metric, not by reading error counts alone; a high failure count with state CLOSED means the breaker is not the source of the short-circuit.
2. 02Verify requests are short-circuited client-side by checking that TCP connect attempts to the dependency target dropped to zero at the moment the breaker opened, while DNS resolutions continued.
3. 03Run an independent probe from a non-circuited client and record the dependency's actual response status and latency; this isolates whether the dependency is recovering.
4. 04Diff the caller's request shape against the probe's request shape; if concurrency, payload size, or header count differs, the breaker's view of the dependency may be correct for this caller even when probes pass.
5. 05Inspect upstream 502 occurrences at the dependency's ingress proxy for the trip window; 502s indicate the dependency's own capacity is being exceeded and are consistent with caller pressure rather than network failure.
6. 06Check breaker scope and shared state; if the breaker is shared across endpoints, isolate the breaker per endpoint and observe whether the target endpoint's failure ratio alone justifies the OPEN state.
7. 07Compare the breaker's failure definition against observed failures; if retries are counted as separate failures, recompute the failure ratio with retries collapsed to one logical failure to test whether the threshold was crossed legitimately.
8. 08Decide recovery path: if probes pass and caller shape matches, allow the breaker's sleep window to elapse and observe a controlled half-open transition; if probes fail or shape differs, do not manipulate the breaker state directly.

Common mistakes

• •Manually forcing the breaker to CLOSED because an independent probe returned 200, without checking whether the caller's traffic shape is the reason the breaker tripped in the first place.
• •Reading application error logs as evidence of dependency failure when those logs only show the client's short-circuit marker and contain no upstream response.
• •Sharing one breaker across multiple endpoints and then attributing all fast-fails to the "main" endpoint, masking that a side endpoint caused the threshold breach.
• •Counting retries as independent failures in the failure-ratio calculation, which inflates the ratio and makes the breaker appear more sensitive than its configuration suggests.
• •Lowering the failure threshold to make the breaker close sooner, which trades one failure mode (premature open) for another (silent dependency degradation) without evidence.
• •Conflating a healthy dependency (probes pass) with a healthy dependency under this caller's load; an upstream 502 on overload is still a dependency problem, not a caller-only problem.

Safe fixes

• •If probes pass and caller shape matches the tripped window, do not change thresholds; wait for the breaker's sleep window to elapse and confirm a HALF_OPEN transition with a bounded probe count before traffic resumes.
• •If probes pass but caller shape differs (higher concurrency, larger payload), reduce caller pressure first (lower concurrency, shed non-critical traffic), then let the breaker evaluate under the new shape; do not edit breaker thresholds as the first action.
• •If probes fail, treat the dependency as not recovered; keep the breaker OPEN, raise an alert on the dependency's ingress proxy 502 rate, and do not attempt to override the breaker state from configuration.
• •If the breaker scope is shared across endpoints, narrow the scope so the affected endpoint has its own breaker and unaffected endpoints are not short-circuited; verify with a config reload that does not require restarting traffic.
• •If retries inflate the failure ratio, change the breaker's recording policy so retries are recorded once per logical request, and verify with a controlled fault-injection test that the ratio matches the underlying call count.
• •If the breaker's failure definition includes status codes the dependency returns on overload (for example, 502), keep the breaker strict and instead address overload at the caller or proxy boundary; weakening the definition hides real upstream failures.

Prove the fix

1. 01Breaker state transitions from OPEN to HALF_OPEN on schedule, with the permitted probe count executed and recorded, and then to CLOSED only after the success ratio in the half-open window meets the configured threshold.
2. 02TCP connect attempts to the dependency target resume at the moment of the HALF_OPEN transition, and DNS resolutions remain stable, confirming requests are no longer short-circuited client-side.
3. 03Latency for the protected dependency returns from the flat near-zero fast-fail floor to the dependency's normal response time distribution, with p95 within the dependency's documented healthy range.
4. 04Independent probes and caller-driven requests both report the same status code distribution over a fixed post-recovery window, with no divergence indicating that the breaker is still filtering by shape.
5. 05Upstream 502 rate at the dependency's ingress proxy remains within the dependency's documented tolerance for the recovery window, so the breaker is not re-opening on caller-induced overload.
6. 06A regression check that briefly exceeds the failure threshold in a controlled environment causes the breaker to OPEN again with the expected transition log and counters, confirming the fix did not disable the breaker.

Prevention and next steps

• •Define breaker scope per endpoint or per dependency contract, not per client process, so a burst on one path cannot short-circuit unrelated paths.
• •Record breaker state transitions and the counters that triggered them as first-class events, with the breaker name and the dependency target as searchable dimensions.
• •Keep the breaker's failure definition aligned with the dependency's documented failure semantics, including which status codes count as failures and how retries are counted.
• •Separate caller pressure controls (concurrency limits, load shedding, payload caps) from breaker configuration so each can be tuned against its own evidence rather than coupled.
• •Run periodic controlled failure-injection exercises that verify the breaker opens, recovers via half-open, and closes, so the recovery path is known before a real incident.

Safe commands and checks

Read the breaker state metric for a given breaker name (replace <breaker_name> with the configured breaker identifier): query the metrics endpoint for the gauge labeled with the breaker state dimension and value OPEN, HALF_OPEN, or CLOSED.
Inspect transition events for a breaker in the log stream (replace <breaker_name> and <window> with the configured identifier and time range): filter for events whose event field equals the breaker's state-change event and record the timestamp, previous state, new state, and counters.
Check TCP connect attempts for the dependency target (replace <dependency_host> and <dependency_port> with the actual values from configuration): query the connection-pool metric for outbound connect attempts in the trip window and confirm whether the count dropped to zero when the breaker opened.
Verify the breaker's failure-definition configuration (replace <config_path> with the path to the resilience configuration file or key): read the failure-threshold, slow-call threshold and duration, sliding-window size, sleep window, and permitted half-open probe count without modifying the file.
Compare the upstream 502 rate at the dependency's ingress proxy (replace <proxy_name> with the proxy or sidecar identifier) for the trip window against the same window from the prior day to establish whether the dependency itself was overloaded.