Clock skew: diagnose timestamps that disagree across services

The symptoms

• •Certificate validation or TLS handshake failures that appear intermittently on otherwise healthy nodes, with logs citing "certificate not yet valid" or "certificate has expired" only on a subset of hosts.
• •Short-lived tokens (JWT, OAuth assertions, SAML, Kerberos TGT renewals) rejected as "expired" or "not yet valid" by a peer, even though the issuer and validator clocks appear correct in isolation.
• •Log events from different services that should be causally related arrive out of order, or that violate expected "happened-before" relationships, when timestamps are reconstructed from per-host wall clocks.
• •Distributed lock, lease, or consensus timeouts fire earlier than expected on some nodes and later on others, with the same TTL value producing different effective durations.
• •Database replication lag metrics reported as negative, or monotonically increasing timestamps on a follower that exceed the leader's commit time.
• •File integrity checksums, signature verification, or HMAC validation fail with messages that reference "timestamp out of window" or "message too old", concentrated on hosts that recently rebooted or resumed from suspend.

Likely causes

• •NTP or PTP daemon is not running, is misconfigured, or has been administratively disabled on one or more hosts, leaving the system clock to drift from the local oscillator and hardware RTC.
• •Asymmetric, congested, or firewalled network paths between hosts and upstream time sources so that NTP responses are repeatedly dropped or delayed, causing the local clock to fall out of sync.
• •Virtualization or containerization that does not expose a stable clock source, or that relies on a suspended host clock, so guest clocks diverge after live migration or hypervisor sleep.
• •Hardware real-time clock (RTC) battery failure, BIOS/UEFI clock misconfiguration, or dual-boot confusion between local time and UTC interpretation at boot.
• •Application code that reads time from a non-authoritative source (process start time, cached Date header, container start time) instead of the synchronized system clock.
• •Leap second handling differences between kernel versions, time daemons, or database engines, producing a one-second jump that consensus and timestamp-ordering code does not tolerate.

First ten minutes

1. 01Compare UTC time on each suspect host with `date -u` and `date +%s` (Unix epoch seconds) to obtain a coarse, single-shot offset between nodes; record the pair of hosts and the measured delta.
2. 02Ask timedatectl for the daemon-level synchronization status: `timedatectl status` reveals whether the system clock is unsynchronized, whether NTP is enabled, and the last successful sync.
3. 03Run `chronyc tracking` on each affected host to read the current offset, frequency, and stratum reported by the local NTP client, per the chrony-project.org documentation.
4. 04Run `chronyc sources -v` to see each upstream time source, its reachability, last measured offset, and stratum, which together indicate whether the local daemon has any healthy reference.
5. 05Run `chronyc activity` to count online versus offline sources, and `chronyc ntpdata <ip>` (if available) to inspect the most recent exchange with a specific peer.
6. 06Check the systemd unit or service manager for the time daemon: `systemctl status chronyd` (or the equivalent for chronyd, systemd-timesyncd, ntpd) to confirm it is active and not in a failed state.
7. 07Capture `hwclock --show` to compare the hardware RTC against the system clock; large divergence here is diagnostic even when software clocks appear sane.

Evidence to collect

• •Per-host snapshots of `date -u`, `date +%s`, `timedatectl status`, `chronyc tracking`, and `chronyc sources -v`, taken at the same moment across the suspect set.
• •Systemd journal entries for the time daemon (e.g., `journalctl -u chronyd --since "1 hour ago"`) covering reachability changes, source rejections, and stratum changes.
• •Application logs filtered for "certificate not yet valid", "JWT expired", "clock skew", "timestamp out of window", or equivalent, with surrounding host identifiers and timestamps.
• •Network-path evidence between hosts and their configured NTP sources: traceroute, ICMP RTT samples, and any firewall or security-group rules that selectively filter UDP port 123.
• •Hypervisor or container runtime metadata for affected hosts, including last live-migration time, suspend/resume events, and whether a paravirtualized clock is exposed.
• •Hardware health indicators from the host's BMC or IPMI, especially RTC battery status and recent CMOS reset events, where available.

Where to look

• •The local time daemon's configuration directory (e.g., `/etc/chrony/chrony.conf` for chrony per chrony-project.org) to confirm which servers, pools, and drift tolerance settings are in effect.
• •The systemd journal for the time daemon unit, where reachability flaps, source step adjustments, and stratum changes are logged with timestamp and reason.
• •Process and container runtime configuration, where CAP_SYS_TIME, seccomp, or AppArmor profiles may block clock adjustment syscalls even when the daemon appears to run.
• •Kernel logs (`dmesg`, `journalctl -k`) for "Clock unsynchronized", "TSC unstable", or "clocksource" messages that indicate a recent fallback in the kernel time source.
• •Boot logs and BIOS/UEFI settings, including the RTC time zone interpretation (local vs UTC) and any "clock was reset" indicators; per chrony-project.org, drift recovery assumes the initial system time is roughly correct.
• •Application-level caches and queues that store or compare timestamps, e.g., token stores, message broker time headers, and database commit timestamps, to localize where the disagreement is being detected.

Diagnostic steps

1. 01Establish a known-good reference: identify a small set of hosts whose clocks are independently verified (e.g., against an external trusted source) and use them as the baseline rather than trusting any single host in the cluster.
2. 02Compute the offset between each suspect host and the baseline using the daemon's own measurement (`chronyc tracking` "Last offset" or "System time") rather than `date` alone, since the daemon samples upstream sources with sub-second precision.
3. 03Correlate the offset with stratum: a high stratum value or a "Unreach" / "*" reachability flag in `chronyc sources -v` indicates the local daemon has lost contact with upstream time and is free-running.
4. 04Check whether the offset is steady or drifting: compare `chronyc tracking` snapshots taken minutes apart to determine whether the local oscillator is slewing, stepping, or running unconstrained.
5. 05Validate network reachability between the host and each configured NTP source using UDP/123-specific probes; a successful `ping` does not prove NTP packets are passing, so test the actual protocol.
6. 06Inspect virtualization layer: on virtualized hosts, confirm a paravirtualized clock (kvm-clock, vHPET, TSC scaling) is exposed and that the hypervisor host clock is itself synchronized, since guest clocks inherit the host's drift.
7. 07Verify application-side time source: instrument or trace the affected service to confirm it reads time via gettimeofday/clock_gettime from the system clock and not from a cached or process-start offset.
8. 08Quantify the failure boundary: identify the largest offset that still satisfies all observed symptoms (e.g., the smallest TTL or token lifetime that has been rejected) to define the tolerance you must prove out.

Common mistakes

• •Trusting a single host's wall clock as ground truth; if that host is itself drifting, every comparison derived from it will be wrong by the same amount.
• •Running `ntpdate` against a live `chronyd` or `ntpd` instance, which forces a step and can break in-flight TLS sessions, database transactions, and monotonic-clock assumptions.
• •Setting the clock manually with `date -s` instead of letting the NTP daemon slew it back into range, which masks the underlying drift cause and reintroduces the same offset after the next reboot.
• •Assuming "TLS works from one host but not another" is a certificate problem and replacing the certificate, when the actual issue is the validator's clock being ahead of the issuer's clock.
• •Disabling the time daemon because "it causes clock jumps", without investigating that the daemon was compensating for an underlying oscillator drift that would otherwise accumulate.
• •Adding more NTP sources without checking stratum and reachability, which can produce a false-positive "synchronized" status while the daemon selects a peer that is itself drifted.

Safe fixes

• •Enable and start the time daemon with the host's package manager and service manager so that the system clock is slewed continuously rather than left to free-run; refer to chrony-project.org for the daemon's documented drift-recovery model.
• •Configure chrony with multiple diverse upstream sources (mix of public pools and a local reference) and set a sensible `makestep` threshold so that large initial errors are corrected once, after which the daemon slews.
• •Open UDP port 123 egress and ingress on any firewall, security group, or network policy between hosts and their designated time sources, and verify reachability with a daemon-level query rather than a raw ping.
• •On virtualized hosts, ensure the hypervisor exposes a stable, paravirtualized clock and that the host itself is synchronized, so that guest clocks inherit a disciplined reference rather than a free-running TSC.
• •Set the BIOS/UEFI clock to UTC and confirm the OS interprets it that way, so that daylight-saving transitions and timezone conversions do not introduce spurious offsets; per chrony-project.org, the initial time must be roughly correct for the daemon to converge.
• •In application code, standardize on the system monotonic clock (`CLOCK_MONOTONIC`, `CLOCK_REALTIME` via a library that pins to the system clock) for elapsed-time checks and reserve the wall clock for absolute timestamps that participate in cross-host validation.

Prove the fix

1. 01After any change, re-run `chronyc tracking` on every affected host and confirm the reported offset is within the tolerance defined by the smallest token lifetime or lease TTL in the failing system, sustained over at least one full polling interval.
2. 02Re-run `chronyc sources -v` and confirm that at least the configured minimum number of sources are reachable, with no leading "?" or "x" reachability flags, and that the selected source has a lower or equal stratum to the previous baseline.
3. 03Reproduce the original failing scenario across the suspect set (token issuance, certificate validation, distributed lock acquisition) and confirm success without requiring a clock change, application retry, or certificate reissue.
4. 04Cross-check `date +%s` on a sample of hosts within the same second and confirm pairwise offsets are within the documented chrony slewing tolerance, with the daemon reporting a non-zero "System frequency" that is stable over time.
5. 05Inspect the systemd journal for the time daemon over the post-fix window and confirm absence of "step", "reach lost", or "unsynchronized" events, and absence of application log entries matching the pre-fix failure patterns.
6. 06Record the post-fix offset, daemon status, and scenario reproduction result alongside the original failure evidence, so that the fix is auditable and the tolerance threshold is documented for future regressions.

Prevention and next steps

• •Monitor the offset and reachability of each host's time daemon as a first-class production signal, with alerting on both absolute offset and sustained reachability loss, rather than discovering drift through downstream application failures.
• •Define and document a single cluster-wide clock tolerance, sized to the smallest token lifetime, lease TTL, or signature validity window in use, and treat any host exceeding it as out-of-policy.
• •Standardize on one time daemon and configuration across the fleet, version-controlled, so that divergence between hosts in how time is sourced is prevented by configuration rather than diagnosed after incident.
• •Audit hypervisor and container clock sources as part of host provisioning, so that virtualized workloads do not silently inherit a free-running clock from an unsynchronized hypervisor.
• •Prefer monotonic clocks for elapsed-time and lease-duration checks in application code, and reserve wall-clock comparisons for cross-host validation where all parties are known to be within tolerance.

Safe commands and checks

date -u && date +%s
timedatectl status
chronyc tracking
chronyc sources -v
chronyc activity
systemctl status chronyd
journalctl -u chronyd --since "1 hour ago"
hwclock --show