Debugging Apollo Federation Subgraph Errors: A Field Guide

The first ten minutes — establish facts before touching code.

• 1Enable Apollo Federation tracing on the gateway: set `APOLLO_GRAPH_REF` and `APOLLO_KEY` environment variables, then check `Apollo Studio` > `Traces` for the exact subgraph query that failed.
• 2Run `rover subgraph check <graph-ref> --name <subgraph-name> --schema ./schema.graphql` to validate schema composition. If this passes, the error is runtime, not schema.
• 3Inspect the gateway's response headers: if you see `graphql-federation-include-keys: true`, the gateway is sending entity keys — verify the subgraph receives them.
• 4Add a health check endpoint on the subgraph that echoes back the `@key` fields it expects. Compare with what the gateway sends.
• 5Check subgraph logs for `__resolveReference` calls: if there are none, the gateway isn't calling it — likely a schema composition issue.

The specific files, logs, configs, and dashboards that usually own this bug.

• searchGateway logs: `grep 'RESOLUTION_FAILURE' /var/log/gateway/access.log`
• searchSubgraph logs: `journalctl -u subgraph-service -f | grep 'resolveReference'`
• searchApollo Studio performance tab for subgraph latency and error rates
• searchSchema registry diff: `rover supergraph fetch <graph-ref> > current.graphql` and compare with the composed schema in CI
• searchGateway config file (YAML/JSON) for `@apollo/gateway` or `@apollo/subgraph` version and `experimental_didResolveReference` callback
• searchSubgraph's `package.json` or `Gemfile` for the federation library version (e.g., `@apollo/subgraph`, `apollo-federation`)
• searchNetwork proxy logs (e.g., Envoy, Nginx) for request/response body size — large entity lists can trigger timeouts

Practical causes, not theory. These are the things you will actually find.

• warning`__resolveReference` not implemented or returns null for valid keys
• warningSchema composition succeeds but `@key` fields differ between subgraph and gateway (e.g., `@key(fields: "id")` vs `@key(fields: "id tenantId")`)
• warningGateway doesn't forward authentication context (e.g., `Authorization` header) to subgraph entity requests
• warningSubgraph library version mismatch: `@apollo/subgraph` v2.4 vs v2.5 changed internal `__resolveReference` signature
• warningEntity keys contain null values or are serialized incorrectly (e.g., JSON number vs string)
• warningSubgraph returns an error in a list entity resolution — gateway treats the entire list as failed
• warningRequest timeout too short: subgraph takes >5s to resolve a batch of entities, gateway aborts

Concrete fix directions. Pick the one that matches your root cause.

• buildImplement `__resolveReference` with proper null-checking: `if (!reference.id) throw new GraphQLError('Missing key')`
• buildAlign `@key` directives across all subgraphs that extend a type — use `rover subgraph check` to enforce
• buildConfigure gateway to forward headers: set `buildService({ url }) => { return new RemoteGraphQLDataSource({ url, willSendRequest({ request, context }) { request.http.headers.set('authorization', context.authToken); } }); }`
• buildUpgrade all subgraph and gateway packages to the same Apollo Federation version (e.g., 2.5.x)
• buildUse `@apollo/subgraph`'s `buildSubgraphSchema` to auto-generate `__resolveReference` from resolvers
• buildAdd a `@shareable` directive to fields that multiple subgraphs can resolve, avoiding conflict
• buildIncrease gateway timeout: set `requestTimeout: 10000` in the `ApolloGateway` constructor

A fix you cannot prove is a guess. Close the loop.

• verifiedRun a query that crosses subgraphs: `{ user(id: "1") { name reviews { body } } }` — if it returns data, the fix works
• verifiedCheck Apollo Studio for zero `RESOLUTION_FAILURE` errors in the last hour
• verifiedDeploy the fix to a canary subgraph and simulate load with k6: `k6 run --vus 10 --duration 30s script.js` — confirm no errors
• verifiedUnit test the `__resolveReference` function: mock the reference and assert it returns the entity
• verifiedIntegration test with the gateway: start subgraph and gateway locally, send a federated query using `gq` or `curl`
• verifiedVerify schema composition still passes: `rover subgraph check` returns `PASS`

Things that make this bug worse or harder to find.

• warningDon't ignore `@external` directives — they tell the subgraph which fields come from other subgraphs; missing them causes silent nulls
• warningDon't assume local gateway matches production: test with the exact same `@apollo/gateway` version
• warningDon't patch `__resolveReference` in a hotfix without updating the corresponding `@key` in the schema — they must be in sync
• warningDon't batch entity requests over HTTP/1.1 without connection pooling — it will cause socket exhaustion
• warningDon't set `requestTimeout` too high (e.g., 60s) — it masks underlying performance issues and degrades user experience

Frequently asked questions