What this usually means
JWT validation compares the 'exp' claim against the current time of the server performing validation. When that server's clock is ahead of the issuer's clock by more than the token's remaining lifetime, valid tokens get rejected. This is not a code bug—it's an infrastructure time synchronization issue. The same token will validate on a server with a correct clock and fail on one that's drifted. The frustrating part: the error message says 'expired' so developers immediately look at token lifetime or generation code, wasting hours before checking the clock.
The first ten minutes — establish facts before touching code.
- 1Run `date` on the server that reports the error and compare with the token's 'iat' and 'exp' claims. If server time > exp, you have clock skew.
- 2Check NTP synchronization: run `timedatectl` (Linux) or `w32tm /query /status` (Windows). Look for 'System clock synchronized: yes' and NTP server reachable.
- 3Decode the failing JWT using `jq -R 'split(".") | .[1] | @base64d | fromjson'` and note the 'exp' (expiration) and 'iat' (issued at) times.
- 4Compare the server's current time (in epoch seconds) with the token's exp. Run `echo $(date +%s) - $(echo '<token>' | jq -R 'split(".") | .[1] | @base64d | fromjson | .exp') | bc` to see the drift.
- 5Check token generation server's clock too. If both servers show different times, the problem is a difference between them, not absolute accuracy.
The specific files, logs, configs, and dashboards that usually own this bug.
- search/var/log/syslog or /var/log/messages for NTP sync errors
- searchApplication logs for 'jwt expired' or 'token validation failed' entries
- searchSystemd journal: `journalctl -u ntp.service` or `journalctl -u chronyd.service`
- searchToken issuer's clock: run `date` on the issuing server and compare epoch with verifying server
- searchJWT library configuration: some libraries have a built-in 'clockTolerance' or 'leeway' setting (check your middleware config)
- searchNTP configuration file: /etc/ntp.conf or /etc/chrony.conf
- searchCloud provider's time sync service (e.g., AWS Time Sync, GCP metadata server)
Practical causes, not theory. These are the things you will actually find.
- warningOne server's clock drifted due to NTP service not running or misconfigured
- warningVirtual machine paused and resumed, causing clock to fall behind or jump ahead
- warningContainer host and guest clock mismatch where the container inherits an unsynchronized host clock
- warningDeveloper machine (used for testing) vs production server clock difference
- warningNewly provisioned server not yet synchronized with NTP (cold start drift)
- warningFirewall blocking NTP port 123, preventing clock sync
Concrete fix directions. Pick the one that matches your root cause.
- buildFix the clock: run `ntpdate -s time.google.com` (temporary) or ensure `chronyd`/`ntpd` is running and enabled: `systemctl enable --now chronyd`.
- buildAdd a small leeway (clock skew tolerance) in JWT validation: set `clockTolerance: 30` seconds (or a few seconds) in your JWT middleware. Do not set it to more than 60 seconds unless you understand the security trade-off.
- buildUse a relative time check: instead of comparing exp against system clock, compare against the token's iat and a maximum allowed lifetime. This removes dependency on absolute time.
- buildFor cloud environments, use the instance metadata service's time endpoint (e.g., AWS: `curl http://169.254.169.254/latest/meta-data/` for time sync).
- buildIf using Docker, mount the host's /etc/localtime as read-only and ensure the container uses the host's timezone: `-v /etc/localtime:/etc/localtime:ro`.
- buildIn Kubernetes, use a DaemonSet that runs chronyd on each node to ensure all nodes are synchronized.
A fix you cannot prove is a guess. Close the loop.
- verifiedAfter clock sync, compare `date` output on all servers; they should be within 1 second of each other.
- verifiedRedeploy the same JWT that previously failed; it should now validate successfully.
- verifiedMonitor NTP sync status: `timedatectl show -p NTPSynchronized` should return 'yes'.
- verifiedRun a continuous test: every minute, issue a JWT and validate it on the target server; log the delta between server time and exp. If the delta stays positive and stable, the fix holds.
- verifiedSimulate clock skew by temporarily setting a test server's clock ahead by 60 seconds, then verify the application rejects tokens with an appropriate error (and not a generic 'expired').
Things that make this bug worse or harder to find.
- warningSetting a large leeway (e.g., 300 seconds) to 'just make it work' — this widens the window for token replay attacks.
- warningOnly fixing one server without checking all servers behind a load balancer; the error will reappear on the next request to a skewed server.
- warningIgnoring the root cause (clock drift) and only adding leeway — the drift might grow over time and exceed your tolerance.
- warningUsing `ntpdate` as a cron job instead of running a proper NTP daemon; ntpdate is deprecated and can cause abrupt time jumps.
- warningAssuming cloud instances automatically sync time — many default AMIs do not have NTP installed. Always verify.
Production Auth Outage: JWT Expired for 10 Minutes Every Hour
Timeline
- 14:00PagerDuty alert: 'High rate of 401 errors' on auth service. ~5% of requests failing.
- 14:03Check logs: error is 'Signature has expired' for tokens issued within the last 30 seconds.
- 14:05Decode failing JWT: exp=1625234400, current server time=1625234420 (20 seconds ahead). Token is valid for another 10 minutes, but server thinks it expired.
- 14:07Check `timedatectl` on the reporting pod: NTP synchronized=no. Clock is 25 seconds ahead of actual time.
- 14:10Check other pods: one pod has correct time, another is 10 seconds behind. The drifting pod is in a different availability zone.
- 14:15Identify root cause: the EC2 instance hosting the skewed pod was launched from a custom AMI that had NTP disabled. The pod inherited the host's wrong time.
- 14:20Force NTP sync on the host: `sudo chronyd -q` and `sudo systemctl restart chronyd`. Error rate drops to zero.
- 14:30Add a DaemonSet to ensure chronyd runs on all nodes. Also add a 30-second leeway in JWT validation as a safety net.
The alert came in during a routine afternoon. Our auth service was throwing 401s for a small but growing percentage of requests. I decoded a few tokens and noticed something odd: the server's current time was consistently 20-30 seconds ahead of the token's expiration time, even though those tokens had been issued seconds earlier. That's when I suspected clock skew.
I hopped onto the affected pod and ran `timedatectl`. It showed 'NTP synchronized: no' and the clock was 25 seconds ahead of real time. I checked the other pods in the cluster — most were fine, but one other pod in a different AZ was actually behind by 10 seconds. The problem was clearly per-node clock drift, not a code issue.
Turns out, our EKS node group used a custom AMI that had NTP disabled. The node's clock had been drifting since boot. We forced a sync, and the errors vanished. Then we deployed a DaemonSet to run chronyd on every node and added a modest 30-second leeway in our JWT validation to handle brief drifts. The real fix was infrastructure, not code.
Root cause
EC2 instance running a Kubernetes node had NTP disabled, causing its clock to drift 25 seconds ahead of real time. The JWT library strictly compared 'exp' against the local clock, rejecting valid tokens.
The fix
Enabled chronyd on the host and added a DaemonSet to ensure NTP runs on all nodes. Also added a 30-second clock tolerance in the JWT validation middleware as a defense-in-depth measure.
The lesson
Always verify NTP synchronization on every node, especially when using custom AMIs. A small leeway (≤30s) is a reasonable safety net, but never treat it as the primary fix.
When a JWT is issued, the server sets the 'exp' (expiration) claim as a Unix timestamp (seconds since epoch). The verifying server reads the current system time and compares it to 'exp'. If current_time >= exp, the token is rejected. This is a simple integer comparison—no magic.
The problem arises because the system time is not absolute. If the verifying server's clock is ahead of the issuer's clock by even a few seconds, tokens that are still valid (based on the issuer's clock) will appear expired. Most JWT libraries do not account for clock differences by default. Some provide a 'leeway' parameter (e.g., PyJWT's `options={'verify_exp': True, 'leeway': 30}`) that adds a tolerance window. But the correct fix is to synchronize clocks.
To measure the skew, you need the difference between the verifying server's time and the token issuer's time. The simplest way: decode the token's 'iat' (issued at) and compare it with the server's current time. If the token was just issued but iat is 10 seconds behind server time, that's a 10-second skew (assuming issuer's clock is correct). But both clocks could be wrong.
For a more reliable measure, use an external time reference. Run `curl -I https://google.com 2>&1 | grep -i date` to get Google's server time, then compare with your local `date`. Or use a dedicated time API like `curl http://worldtimeapi.org/api/timezone/Etc/UTC`. The difference is your local clock's offset from UTC.
Containers inherit the host's clock. If the host's NTP is misconfigured, all containers on that host will have skewed time. A common issue: the host runs a minimal OS image without NTP daemon. Another: the host's NTP service uses a local NTP server that itself is skewed.
In Kubernetes, nodes should run chronyd or ntpd. But many managed Kubernetes services (EKS, GKE) do not install NTP by default. You must verify. Use a DaemonSet to run an NTP client container that ensures time sync. Alternatively, use the node's built-in NTP tool (e.g., `chrony` on Amazon Linux 2).
Adding a 30-second leeway means tokens can be used up to 30 seconds after they expire. This widens the replay window. For most apps, 30 seconds is acceptable, but for high-security environments (finance, military), even 1 second is too much. The better approach is to keep clocks within 1 second using PTP (Precision Time Protocol) or hardware clocks.
A middle ground: use a small leeway (like 5 seconds) to cover transient NTP sync delays, but fix the underlying clock drift. Monitor NTP sync status and alert if drift exceeds 1 second. This gives you both security and reliability.
Frequently asked questions
How do I check if a JWT token is expired due to clock skew or genuinely expired?
Decode the token and get the 'exp' value. Then run `date +%s` on the server validating the token. If server time is greater than exp, check the difference. If the difference is small (e.g., < 60 seconds) and the token was issued recently (check 'iat'), it's likely clock skew. Also check NTP sync status on the server.
What leeway value should I set for JWT validation?
Start with 0 and fix clock sync. If you must use leeway, set it no higher than 30 seconds. Most production systems can maintain sub-second NTP sync, so any leeway is a fallback. For high-security, aim for 0 leeway with hardware clock sync (e.g., PTP).
Does Docker Desktop cause clock skew?
Yes, especially on macOS and Windows where Docker runs in a VM. The VM's clock can drift significantly if the host sleeps. Restarting Docker or the VM resyncs the clock. In CI/CD pipelines, this can cause intermittent JWT failures. Always run `docker run --rm alpine date` to check container time.
Can I use the token's 'iat' instead of system time to validate expiration?
Technically yes, but you lose the ability to enforce absolute expiration. If you compare 'exp' against 'iat + max_lifetime', you're using the issuer's clock, which might also be skewed. A better approach: use a trusted time source like an NTP server and ensure all servers use it. Relative checks are only useful when both clocks are unreliable.
How do I fix clock skew in a Kubernetes pod without restarting?
You can't change the pod's clock directly; it inherits from the node. Either fix the node's NTP or delete the pod so it reschedules on a synced node. For a quick fix, you can run a privileged container with `chronyd` that shares the host's network and clock, but that's complex. Better to ensure all nodes have NTP configured via a DaemonSet.