API returns 502 in one region: compare regional upstream paths

The symptoms

• •502 responses observed from a single geographic region or POP while identical requests from other regions return 200 or 3xx as expected.
• •Health checks or synthetic probes from the failing region fail, while the same probe executed from a healthy region succeeds against the same origin host.
• •Users report 502 with normal latency from one country or ISP, while logs from the application servers show no corresponding 5xx at the origin tier.
• •CDN or load balancer logs for the affected POP show 502 entries paired with empty or unexpected upstream responses, with no matching application-level error.
• •Routing or DNS resolution from the failing region returns a different IP, ASN, or hop count than resolution from a healthy region for the same hostname.

Likely causes

• •Divergent DNS resolution in the failing region, where an authoritative or resolver cache returns a stale, decommissioned, or region-specific IP that no longer answers.
• •Regional routing or peering problem where traffic exits the failing POP toward an upstream that is unreachable from that network path (BGP, anycast split, or transit issue).
• •TLS handshake failure specific to the regional path, for example an SNI mismatch, an unsupported cipher on that hop, or an expired intermediate certificate served by an intermediate proxy.
• •WAF, bot protection, or rate-limiter rule scoped to a region or ASN returning a synthetic 502 when it should return 403 or 429.
• •Origin or backend instance in the region-specific deployment segment is unhealthy, overloaded, or removed from the load balancer pool, producing a bad gateway at the regional tier.
• •Regional CDN shield or mid-tier cache cannot reach origin due to a private link, VPC peering, or allowlist configuration that excludes the failing region's egress IP.

First ten minutes

1. 01Confirm scope: separate 502s at the edge (CDN, WAF, gateway) from 502s at the origin by reading the edge log fields status and upstream_status, and check whether the same request id appears in both.
2. 02Reproduce from the failing region using an out-of-region probe (a hosted synthetic or a known-good ISP) and from a healthy region against the same hostname, comparing status, upstream IP, and TLS handshake outcome.
3. 03Resolve the hostname from each region using public DNS and compare answer records, TTL, and resolver; a divergent A/AAAA record is the most common regional 502 root cause.
4. 04Trace the regional path with a network probe tool that returns per-hop ASN and RTT; a sudden RTT increase or new ASN before the origin often points to a peering or egress issue.
5. 05Inspect the regional edge configuration for recent changes: anycast routing, custom origin hostname, shield POP, or TLS settings; revert or disable the change if the regional 502 began after the change.
6. 06Compare health-check definitions and pool membership between the failing and healthy regions; mismatched thresholds or pool sizes are a frequent source of regional 502s.

Evidence to collect

• •Edge access log entries from the failing POP and a healthy POP for the same request id, with fields client_ip, host, status, upstream_status, upstream_addr, tls_version, and ssl_protocol.
• •DNS resolution results for the API hostname from at least two vantage points, including record type, TTL, and the AS number announcing the returned IP.
• •Network probe output from the failing region showing hop count, ASN per hop, and RTT, alongside an identical probe from a healthy region for comparison.
• •TLS handshake details from each region including negotiated cipher, certificate chain, and any SNI-related warnings or handshake errors.
• •WAF or bot-management rule evaluation logs filtered to the failing region to detect synthetic 502 responses generated by security policy.
• •Origin pool or backend inventory snapshot showing which instances are healthy, draining, or removed, for the regional deployment segment implicated by the trace.

Where to look

• •At the regional edge: CDN POP logs, anycast routing tables, shield POP selection, and TLS termination settings that apply only to that region.
• •At the DNS boundary: authoritative NS records, EDNS Client Subnet responses, and any geo-DNS or traffic-manager policies that return different records per region.
• •At the regional network boundary: egress IPs, peering agreements, transit provider status, and BGP announcements for the destination prefix as seen from the failing region.
• •At the regional origin tier: load balancer pools, health check endpoints, autoscaling group state, and instance health for the deployment segment serving that region.
• •At the security boundary: WAF, bot protection, and rate-limiter rules that vary by region, ASN, or client IP, and that may emit synthetic 502 responses.
• •At the control plane: recent configuration changes to routing, origin, TLS, or pool definitions that were applied to the failing region and not to healthy regions.

Diagnostic steps

1. 01Isolate the failure layer by comparing edge upstream_status values across regions: a missing or 0 value points to a connect failure, a non-2xx upstream value points to an origin issue, and a missing edge log points to a client-to-edge problem.
2. 02Run identical resolution and TCP-level probes from two vantage points and diff the returned IPs, ASNs, and TCP handshake outcomes; a divergent IP or ASN localizes the problem to DNS or routing.
3. 03Perform a TLS handshake from each region against the same hostname and compare the negotiated cipher and certificate chain; an older POP with a deprecated cipher suite can produce 502 only on that path.
4. 04Replay the request through the regional edge with verbose origin logging enabled to capture the upstream_addr hit and the response bytes; if upstream_addr is missing, the regional edge could not reach any pool member.
5. 05Test origin reachability directly from a host in the failing region's network using a public probe service, bypassing the edge, to confirm whether the upstream host is reachable from that network at all.
6. 06Diff configuration snapshots between the failing region and a healthy region, focusing on origin hostname, custom certificates, shield POP, and health check thresholds, to find the first divergent setting.
7. 07Replay the request with regional security rules temporarily disabled in a canary, comparing 502 rates before and after, to determine whether a regional WAF or bot rule is emitting the 502 synthetically.

Common mistakes

• •Concluding that the application is broken because origin logs show no errors, when the 502 is generated by the regional edge before the request reaches origin.
• •Assuming DNS is correct because a local resolve works, without testing from a vantage point inside the failing region's network where EDNS Client Subnet or geo-DNS may return a different record.
• •Rebooting origin instances in response to a regional 502, which does not change the regional path and can amplify impact by removing healthy capacity.
• •Increasing timeouts or relaxing health checks across all regions to mask a regional pool issue, which hides the real divergence between regions and delays diagnosis.
• •Trusting a single synthetic probe from the engineer's own network, which may be in a healthy region, and missing the regional divergence that only appears from the affected POP.

Safe fixes

• •If DNS divergence is the evidence, lower TTL on the affected record, pin clients in the failing region to the healthy-region answer via an override, and coordinate an authoritative record refresh; verify by repeating the two-vantage-point resolution.
• •If the regional origin pool is empty or unhealthy, restore pool membership by re-enabling drained instances or expanding capacity in that region, and confirm via health check fields before re-serving traffic.
• •If a regional WAF rule is producing synthetic 502, narrow the rule to the affected region only and switch its action from synthetic 502 to a logged 403 so the real upstream is not masked during diagnosis.
• •If TLS is the divergence, force the regional edge to use a modern TLS profile with explicit SNI, and pin the intermediate chain; verify by repeating the TLS handshake probe from the failing region.
• •If the regional egress path is unreachable, add a backup transit or temporarily steer the failing region's traffic through a healthy region's edge, and confirm via edge logs that the new path serves 200s.

Prove the fix

1. 01Repeat the two-vantage-point resolution and probe; the failing region's answer must now match the healthy region's answer and yield a 2xx response for the same request.
2. 02Confirm edge logs from the previously failing POP show status 200 with a non-empty upstream_addr for the same request id that previously returned 502.
3. 03Watch synthetic probes from the failing region over a defined window; 502 rate must drop to zero and the regional origin pool health ratio must return to the same level as the healthy region.
4. 04Diff configuration between regions to confirm the divergent setting that caused the 502 has been aligned or explicitly scoped, and that no new divergence was introduced.
5. 05Observe that users in the previously affected region report successful responses in support channels or telemetry, with no 502 entries in the same time window after the change.

Prevention and next steps

• •Standardize per-region configuration under version control and alert on drift between regions for origin, TLS, shield POP, and health check settings.
• •Run multi-region synthetic probes continuously so regional 502s are detected before user reports, and route alerts to the team that owns the regional deployment segment.
• •Prefer low TTLs on DNS records that may need to change per region, and document the EDNS Client Subnet or geo-DNS policy so future edits cannot silently diverge.
• •Test any regional change in a canary POP first and compare its 502 rate against a control POP before promoting the change to additional regions.
• •Document the runbook for regional 502 with the specific log fields, probes, and configuration keys used here, so on-call engineers do not improvise under time pressure.

Safe commands and checks

dig +short <api.example.com> @<resolver-ip> # resolve the API hostname from a chosen resolver; replace <api.example.com> and <resolver-ip> with the production hostname and a public resolver IP you choose
dig +short <api.example.com> @<resolver-ip> # repeat the resolution from a second vantage point by using a different public resolver IP, to diff answers across regions
nslookup -debug <api.example.com> <resolver-ip> # capture TTL, EDNS Client Subnet, and authoritative flags; run from hosts in different regions to compare
traceroute -A <api.example.com> # observe per-hop ASN and RTT to the upstream from the failing region; compare against the healthy region output
openssl s_client -connect <api.example.com>:443 -servername <api.example.com> -tls1_2 # capture the negotiated cipher and certificate chain from a vantage point in each region
curl -sS -o /dev/null -w '%{http_code} %{remote_ip} %{ssl_verify_result}\n' https://<api.example.com>/<health-path> # run from each region to compare status code, remote IP, and TLS verification result