The healthy secondary no one routes toFailover & RoutingHardPreparing

00:00

Incident Dossier

The healthy secondary no one routes to

A regional provider outage raises error rates even though the secondary is healthy. Trace health state, route caching, provider clients, and deployment configuration to find why failover never takes effect.

Initial failing signalTraffic remains on an unhealthy primaryExpected behavior: A fresh health failure routes traffic to the healthy secondary. · actual behavior: The cached primary route is reused after health state changes.

Codebase incident

Evidence desk

Codebase Incidents · 6

At 18:03 UTC, requests to the primary payment region begin failing. The secondary provider is healthy and passes direct probes, but customer traffic keeps selecting the primary. A recent routing optimization added a short-lived route cache. Investigate the interaction between cached route choice and current health state; do not mechanically copy the existing provider-failover challenge.

Customer symptom

Requests fail during a primary provider outage even though a healthy secondary exists.

Expected

Cached route choices are valid only while the selected provider remains healthy.

Actual

The router returns a cached primary choice without rechecking current health.

Architecture

Traffic routerHealth stateRoute cachePrimary provider clientSecondary provider clientFailover config

Traffic router → Health state · current status
Traffic router → Route cache · reuse decision
Traffic router → Primary provider client · selected route
Traffic router → Secondary provider client · fallback route
Health state → Failover config · thresholds

Logs2

Routing decision

18:03:02 health primary=degraded secondary=healthy 18:03:02 route selected=primary cache=hit 18:03:02 provider primary status=503

router.route

Route cache timeline

18:02:30 cache set key=payments value=primary ttl=30s 18:03:02 health update primary=degraded 18:03:02 cache get key=payments value=primary

router.cache

Metrics1

Provider health

primary probe success=12%; secondary probe success=100%; primary traffic=100%; secondary traffic=0%

provider.health

Recent changes

Route decisions were cached for 30 seconds to reduce health-store reads.
Primary and secondary clients now expose the same request interface.
Health probes were changed to update state immediately on failure.

Plausible hypotheses

Primary outage is larger than the probe — Requests fail at the primary while its health signal may be too narrow.
Secondary is misconfigured — Failover can appear broken when the alternate client has invalid credentials or endpoint config.
Route cache ignores changed health — The incident follows a routing optimization and logs show cache hits after a health transition.
Failover threshold is too conservative — A threshold or cooldown can intentionally keep traffic on a degraded provider.

Progress

1Reproduce the bugRun checks

2Edit the fixOpen

3Pass checksOpen

4Submit fixOpen

Reproduction Steps

Route once while both providers are healthy.
Mark the primary degraded and keep the secondary healthy.
Request again before the route cache expires.
Make the route decision honor current health and prove recovery after restoration.

Focus Mode

The primary was selected earlier; check whether a cached choice is still trusted after health changes.

Start

README.md
src/routing/router.js
src/routing/health.js

Likely area

src/routing/router.js
src/routing/cache.js
src/routing/health.js

Investigation path

1Confirm direct health for both providers.
2Compare route selection with route-cache state.
3Trace the health transition and cache invalidation boundary.
4Use current health for cached choices and verify both failover and recovery.

Repo scale

15 meaningful files
router, health state, cache, provider clients, request path, metrics, and deployment context
One health-aware route fix

Challenge Coach

Hints reveal broad direction first, then narrow toward the suspect path without showing hidden tests or exact solution code.

TopicFailover & Routing

DifficultyHard

Est. time40 min

Workspacepreparing

Explorer

Loading files

Evidence Map

Failing signal loaded from challenge API
Workspace files loaded from workspace API
Results rendered only after API run/submit

Autosaved

workspace

Outputread-onlyready$ prepare workspace

Preparing workspace