Debugging Provider-Wide Outages

How this failure announces itself.

• warningMultiple user journeys that share one external provider start timing out or returning connection errors at roughly the same time.
• warningThe provider reports a regional or datacenter incident, but your own impact is partial: some accounts, protocols, regions, or operations still work.
• warningHealth telemetry marks the primary path unhealthy while application logs show traffic continues to select it.
• warningA configured secondary path receives no traffic, fails authentication, or lacks current policy and network configuration when operators try to use it.
• warningAn apparent recovery produces flapping, duplicate operations, a growing retry queue, or failures that persist only inside your application.

The first ten minutes — establish facts before touching code.

• 1Open an incident timeline and record one precise failing operation: timestamp, source region or network, account, protocol, target hostname, latency, and returned error. Preserve raw evidence without customer message contents or credentials.
• 2Check the provider’s official status, then independently probe the same operation from another network or region. An official incident supports the provider hypothesis; it does not prove every local symptom shares that cause.
• 3Resolve the target with authoritative and recursive DNS and compare it with the endpoint the application actually uses. Do not change DNS, MX records, credentials, or client settings while the evidence still indicates a broad provider incident.
• 4Build a quick blast-radius table: user journey, dependency capability, affected regions or accounts, current state (unavailable, degraded, delayed, unaffected), reversibility, and owner.
• 5Inspect routing telemetry. Confirm which region was selected, which health sample informed that decision, how old the sample was, and whether the secondary passed authentication and a harmless operation.
• 6Choose containment explicitly: hold traffic, queue work, enable a documented degraded mode, or fail over. State the correctness risk and rollback trigger before moving traffic.

The code and config that usually owns this bug.

• searchProvider boundary evidence - official incident updates, exact affected services, timestamps, and your synthetic probes from more than one independent network or region.
• searchDNS path - authoritative records, recursive resolver answers, TTLs, cached targets, split-horizon rules, and whether the application resolves the same endpoint you tested manually.
• searchApplication boundary - recent deploys, credentials, certificates, proxy and firewall rules, connection pools, timeout budgets, retry policy, and provider-specific error classification.
• searchBlast-radius map - every user journey that depends on inbound traffic, outbound traffic, authentication messages, callbacks, support channels, billing notifications, or administrative access.
• searchRouting policy - where primary priority is converted into current eligibility, how regional health is consumed, and whether an unhealthy route can remain selected because it appears first in configuration.
• searchSecondary readiness - credentials, allowlists, templates or policy, suppression and compliance state, quotas, replicated data, callback URLs, and capacity under realistic load.
• searchControl and recovery paths - whether operators can still change routing, read health, communicate with users, and reconcile queued work without relying on the failed provider domain.

Fix the cause, then make the regression impossible.

• buildSeparate provider, DNS, network, credential, and application hypotheses. Change production only after a boundary test points to the layer that owns the fault.
• buildMake route selection health-aware and fail closed: prefer the healthy primary, select an eligible healthy secondary when needed, and enter the defined degraded mode when neither route is safe.
• buildUse layered health checks that test DNS, connection, authentication, a harmless protocol operation, and an isolated end-to-end transaction where appropriate. Record region and failure layer instead of collapsing everything into one boolean.
• buildAdd hysteresis to routing decisions: require sustained failure before failover and sustained health before failback so a recovering region does not attract and lose traffic repeatedly.
• buildBound timeouts and retries. Preserve stable operation identity, reconcile ambiguous outcomes, and prevent a failover attempt from replaying the same side effect in both regions.
• buildDefine degraded behavior per user journey. Queue reversible work durably, expose delayed state honestly, provide an independent recovery channel for critical access, and pause operations whose correctness cannot be guaranteed.
• buildKeep secondary-path configuration continuously warm through safe synthetic traffic and scheduled drills. A standby that is tested only during an outage is an assumption.
• buildMaintain a dependency inventory with owners, failure domains, official status channels, routing controls, queue limits, user messaging, and recovery criteria. Review it whenever regions or providers change.

A fix you can't demonstrate is a guess. Close the loop.

• verifiedProvider versus local fault is supported by independent evidence: official scope, multi-vantage probes, DNS answers, direct protocol checks, and application-only checks no longer contradict one another.
• verifiedA deterministic test with an unhealthy primary and healthy secondary selects the secondary exactly once; a healthy primary remains preferred; two unhealthy routes fail closed.
• verifiedThe secondary path passes authentication, policy, network, quota, state-consistency, and safe synthetic end-to-end checks before production traffic increases.
• verifiedStaged traffic shows acceptable error rate and latency, bounded queue age, no duplicate operations, and expected delivery or completion confirmations.
• verifiedQueued and ambiguous operations are reconciled with stable identifiers before retries return to normal rates.
• verifiedThe primary remains healthy for the agreed observation window before failback, and failback is staged with the same measurements as failover.
• verifiedThe incident closes with a timestamped recovery proof, an updated dependency and blast-radius map, and a scheduled exercise for any untested recovery path.