JWT clock skew expiry: identify valid tokens rejected by time disagreement

The symptoms

• •API returns 401 with error codes containing `TokenExpiredError`, `jwt expired`, `ERR_JWT_EXPIRED`, or `invalid jwt: exp claim` shortly after a token has just been issued.
• •A token issued seconds in the past fails at a downstream verifier with the same payload but processed minutes later in a different environment.
• •Auth-related Set-Cookie headers carry short-lived JWT cookies that the browser immediately re-requests; the verifier log shows the cookie's token as expired within seconds of issuance.
• •Successful and failed token validations cluster around changes to host time sync, NTP correction, or container/VM resume after suspend, rather than around the token's real lifetime.
• •Tokens issued by an upstream identity provider silently fail after migrating a microservice onto a new node whose system clock diverges from the IdP.

Likely causes

• •Wall-clock time on the verifier host diverges from the issuer host beyond the standard validation skew tolerance, often after a sync daemon correction.
• •NTP, chrony, or `timesyncd` is disabled, masked by a frozen VM clock, or pointed at a source that disagrees with the identity provider's clock source.
• •`clockTolerance` / `leeway` / `clockSkewSeconds` on the verifier is set to 0 (or absent, with library default at 0), so any sub-second drift flips `exp` or `nbf` evaluation.
• •The verifier uses a separate monotonic-clock abstraction or a mocked/stubbed clock in test code that has leaked into production, so `Date.now()` and the verifier's claimed current time no longer match.
• •The issuer and verifier are in containers without a stable time source, or one side caches a token whose `iat` was computed from a previously-correct clock that has since drifted.

First ten minutes

1. 01Capture the failing token's unverified payload (header + claims) and record the verifier-side current time at the moment of rejection; compare `exp`, `nbf`, and `iat` against each other.
2. 02Read the system clock on the issuer host and the verifier host with `timedatectl status` (or platform equivalent) and record the offset, sync status, and last correction.
3. 03Inspect the verifier's JWT library configuration for `clockTolerance`, `leeway`, or equivalent skew knobs; note the numeric value and whether it is library default or explicitly set to 0.
4. 04Confirm the failure window by reproducing the rejection locally: take a freshly issued token and feed it to the verifier while watching its clock, rather than relying on the original request.
5. 05Decide whether the disagreement is fast (seconds, during request handling) or slow (minutes-hours, drift accumulation) before touching any configuration; this selects between tolerance tuning and clock-source repair.

Evidence to collect

• •Unverified JWT payload with `exp`, `nbf`, `iat`, and `iss` claims plus the standard expiration error message and stack trace from the verifier.
• •Verifier host clock reading, NTP/chrony/timesyncd status (`timedatectl`, `chronyc tracking`, `ntpq -p`), and a monotonic-clock-vs-wall-clock comparison if available.
• •Issuer host clock reading and its time-sync status, taken at the same moment as the verifier reading so the offset is meaningful.
• •JWT library version and the exact skew/latency setting the verifier used when deciding the token was expired.
• •Authentication log events showing the correlation between rejection timestamps and any NTP correction or container/VM time-jump events on the verifier host.

Where to look

• •At the issuer/verifier clock boundary: the system clock on each machine that creates or consumes tokens, including containers that inherit the host clock.
• •At the JWT verification middleware: the call site that evaluates `exp` and `nbf`, where the library's tolerance/clock argument is wired in.
• •At the time-sync boundary: NTP, chrony, `timesyncd`, or hypervisor time-sync settings on both the issuer and the verifier, plus any container runtime that injects a clock offset.
• •At the token transport boundary: the Set-Cookie header carrying a JWT cookie, and any proxy, gateway, or auth filter that rewrites `exp`/`iat` while forwarding requests.
• •At the test/production boundary: time-mocking utilities (`sinon.useFakeTimers`, `jest.useFakeTimers`, `clock` injection points) that can silently pin the verifier's notion of "now."

Diagnostic steps

1. 01Decode the failing token without verifying it; compute `exp - iat` to learn the token's nominal lifetime, and note `exp - (verifier now)` to see how far past its expiry the verifier thinks it is.
2. 02Compare the issuer's wall-clock to the verifier's wall-clock at the moment of failure; if the difference exceeds the configured skew tolerance, suspect clock disagreement rather than a long-lived token.
3. 03Verify the verifier's claim handler directly: feed the same token + payload to the verifier's own validation function with `clockTimestamp` set to known values bracketing the disagreement; map the exact transition point where the token flips from valid to invalid.
4. 04Replace any clock-mocking wrappers in the test path with real time and re-run the rejection locally; if the rejection disappears, the bug is a leaked clock stub, not real-world clock skew.
5. 05Step back along the request path: confirm the auth middleware is the layer emitting the expired-token error, not a downstream session store or cache reporting stale metadata as a JWT failure.

Common mistakes

• •Increasing token lifetime (e.g., bumping `exp` to `iat + 1h`) instead of fixing the time source, which papers over the disagreement and leaves other time-validated tokens exposed.
• •Setting `clockTolerance` to a large constant (such as several minutes) to mask intermittent drift, which weakens the meaning of `exp` and `nbf` for every request.
• •Trusting a container's reported uptime as evidence that the clock is correct, when the host clock may have jumped during suspend/resume and the container has not re-synced.
• •Assuming "issued seconds ago, fails seconds later" means the token is malformed, and re-issuing a new token without measuring the offset; the same failure will recur on the next issuance.
• •Checking only the verifier clock and not the issuer clock; symmetric time-sync health on both sides is required to make `exp`/`nbf` meaningful.

Safe fixes

• •If the verifier's skew tolerance is set to 0 and the measured offset between issuer and verifier is sub-second, set an explicit `clockTolerance` (commonly 30 seconds) on the JWT library's verifier configuration rather than relying on implicit defaults.
• •If NTP or its equivalent is disabled or unsynchronised on the verifier host, enable it and point it at a stable upstream; verify sync before re-enabling token validation in the request path.
• •If the verifier's clock has jumped (post-suspend VM or container resume), restart the verification service after the host clock has been corrected, or use a monotonic-clock helper that clamps large backward jumps so a stale JWT is not silently re-accepted.
• •If a test or local stub is supplying a fake `Date.now()` to the verifier in production, replace the abstraction with `systemClock.now()` and add a regression test that fails when mocking crosses the verification boundary.
• •If the issuer host is drifting while the verifier is correct, repair the issuer's time sync first; tokens minted with an inaccurate `iat` will produce `exp` claims that look valid in isolation but disagree at any other verifier.

Prove the fix

1. 01Replay the originally failing token through the verifier after the fix lands: the same payload that previously returned `TokenExpiredError` now returns a normal authenticated response within its nominal lifetime.
2. 02Run a synthetic skew probe: mint a token whose `exp` is exactly `verifier_now + tolerance - 1s` and confirm it is accepted, and a token whose `exp` is `verifier_now + tolerance + 1s` is rejected, pinning the new tolerance boundary.
3. 03Capture issuer and verifier wall-clock offsets over a 10-minute window with `timedatectl`/`chronyc tracking` style output; an acceptable fix keeps the absolute offset under the configured tolerance and the drift rate near zero.
4. 04Inspect the auth log for a sustained period (e.g., one full token-rotation cycle) and confirm no new `TokenExpiredError` or `jwt expired` entries appear for tokens whose `exp - iat` is shorter than the documented lifetime.
5. 05Force the verifier host clock backward by a small, safe amount (within the chosen tolerance) in a non-production environment and confirm legitimate tokens are still accepted, and tokens beyond `exp + tolerance` are still rejected.

Prevention and next steps

• •Run a periodic clock-offset check between every host that issues tokens and every host that verifies them; alert when the offset approaches the configured skew tolerance.
• •Make the JWT library's skew tolerance an explicit configuration value, documented and reviewed, rather than leaving it at the library default which may be 0.
• •Keep time-sync services managed (NTP/chrony/timesyncd) under infrastructure as code so they survive host rebuilds, container moves, and autoscaling events.
• •Add an integration test that issues a token, deliberately introduces a small clock offset on the verifier, and asserts the configured tolerance is honoured; this catches regressions in both the clock source and the verifier configuration.
• •Audit auth middleware for clock-injection points (test fakes, monotonic-clock helpers, custom `now()` functions) so a stray mock cannot reach production verification paths.

Safe commands and checks

timedatectl status
chronyc tracking
ntpq -p
date -u +%s.%N
printf '%s' <token> | cut -d. -f2 | base64 -d 2>/dev/null; echo
grep -E 'TokenExpiredError|jwt expired|ERR_JWT_EXPIRED' /var/log/<auth-service>.log