What this usually means
The textbook says TTL controls how long a DNS record is cached. In reality, many layers ignore or extend TTL: OS stub resolvers (systemd-resolved, macOS mDNSResponder), intermediate forwarders, network security appliances, and even application-level DNS caches (Java's InetAddress cache, Python's socket cache, Node.js's dns.lookup cache). Negative caching (NXDOMAIN) often has a separate, longer TTL. The most common culprit is a recursive resolver that does not respect the authoritative TTL or a client-side DNS cache that is not being flushed on TTL expiry. Also, some resolvers employ 'stale caching' where they serve expired records to avoid lookup delays.
The first ten minutes — establish facts before touching code.
- 1Run `dig +trace <yourdomain>` from the affected client to see which resolver returns the stale IP and compare with authoritative NS response
- 2Check the actual TTL returned by the authoritative nameserver: `dig <yourdomain> @<authoritative-ns>`
- 3Inspect local DNS cache: Linux `systemd-resolve --statistics` or `sudo journalctl -u systemd-resolved`; macOS `sudo killall -INFO mDNSResponder`; Windows `ipconfig /displaydns`
- 4For Java apps, check `-Dsun.net.inetaddr.ttl` default (usually 30s or -1 for infinite cache)
- 5Check if your network uses transparent DNS proxies or forwarders that may override TTL (e.g., corporate DNS, Pi-hole, Unbound with `serve-expired`)
- 6Simulate a TTL expiry by waiting the TTL plus a margin and re-querying with +norecurse to see if the record changes
The specific files, logs, configs, and dashboards that usually own this bug.
- search/etc/systemd/resolved.conf (systemd-resolved cache settings)
- search/etc/dnsmasq.conf (dnsmasq cache-size and local-ttl overrides)
- search/etc/unbound/unbound.conf (cache-min-ttl, serve-expired)
- searchApplication DNS cache: Java `$JAVA_HOME/lib/security/java.security` (networkaddress.cache.ttl), Python `socket.setdefaulttimeout` and `requests` session cache
- searchOS-level cache: Linux `/proc/net/ip_dynaddr`, macOS `scutil --dns`
- searchCDN/DNS provider logs (Cloudflare, AWS Route53) to check TTL propagation
Practical causes, not theory. These are the things you will actually find.
- warningStub resolver (systemd-resolved) has a positive cache that ignores TTL when upstream is unreachable (stale-serve)
- warningIntermediate forwarder (corporate DNS, ISP) has a minimum TTL setting that overrides shorter TTLs
- warningApplication-level DNS cache with infinite TTL (Java default `networkaddress.cache.ttl=-1`)
- warningOS-level negative caching: NXDOMAIN is cached for longer than the record's TTL (default 300s in systemd-resolved)
- warningDocker/container DNS: embedded DNS resolver (127.0.0.11) caches responses and does not respect TTL strictly (especially with `--dns` options)
- warningCloud load balancer DNS with CNAME flattening or ALIAS records that have hidden TTLs
Concrete fix directions. Pick the one that matches your root cause.
- buildFlush local DNS cache: Linux `sudo resolvectl flush-caches`; macOS `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`; Windows `ipconfig /flushdns`
- buildConfigure systemd-resolved to use a short `Cache=no` or set `DNSStubListener=no` and use a proper caching resolver
- buildFor Java apps, set JVM property `-Dsun.net.inetaddr.ttl=30` or modify `$JAVA_HOME/lib/security/java.security`
- buildFor Python, use `socket.setdefaulttimeout(5)` and disable `requests` session cache with `Cache-Control: no-cache` headers
- buildReduce TTL on DNS records to 60 seconds or lower during migration, then raise after stable
- buildEnsure authoritative TTL is honored by checking resolver logs for `max-ttl` or `cache-min-ttl` overrides
A fix you cannot prove is a guess. Close the loop.
- verifiedAfter flushing, run `dig <yourdomain>` from the client and confirm TTL matches authoritative value
- verifiedMonitor real traffic: check server access logs for new IPs appearing within expected TTL window
- verifiedUse `tcpdump -i any port 53` to watch DNS queries and verify no cached responses are sent
- verifiedFor application caches, add debug logging to confirm DNS lookup is performed on each request
- verifiedDeploy a canary and verify it resolves correctly while production still shows old records
- verifiedCheck resolver statistics: `resolvectl statistics` showing cache hits vs misses
Things that make this bug worse or harder to find.
- warningFlushing DNS cache on the wrong host (e.g., local dev machine instead of the production resolver)
- warningAssuming `dig` output reflects what the application sees — application may have its own cache
- warningSetting TTL too low (e.g., 1 second) can cause excessive queries and actually increase latency
- warningIgnoring negative caching: ensure NXDOMAIN TTL is also short or use DNSSEC to avoid spoofing
- warningNot checking if the DNS provider has a cache override (e.g., Cloudflare proxied records have a 300s minimum)
- warningKilling and restarting containers without clearing the Docker DNS cache (needs `systemctl restart docker` or node reboot)
The 15-Minute DNS TTL That Took Two Hours
Timeline
- 09:00DNS A record for api.example.com changed from 203.0.113.10 to 203.0.113.20 via Route53 with TTL=60
- 09:05Deploy new pods; some pods start but fail health checks against old IP
- 09:10Rollback triggered; old pods still failing because they resolve to new IP? No, they resolve to old IP!
- 09:15`dig api.example.com @8.8.8.8` returns new IP, but `dig` without @ returns old IP
- 09:20Check systemd-resolved cache: `resolvectl query api.example.com` shows old IP with TTL=45 remaining
- 09:25`sudo resolvectl flush-caches` — now `dig` returns new IP immediately
- 09:30But pods still failing! Discover that kube-dns (CoreDNS) has its own cache with default 30s TTL minimum
- 09:45Edit CoreDNS ConfigMap to set `cache 10` and restart CoreDNS pods
- 10:00Gradually pods start resolving new IP; traffic shifts to new endpoints
- 10:30All pods healthy; incident resolved after 90 minutes of unexpected delay
We were doing a routine DNS update to point api.example.com to a new load balancer. The TTL was set to 60 seconds — plenty of time for a quick cutover. I changed the record in Route53 and waited a minute. I ran `dig` and saw the old IP. I checked authoritative servers: correct new IP. I flushed my local cache and it worked. But my pods were still failing.
The first gotcha was systemd-resolved on the Kubernetes nodes. Even though I flushed the cache, the node's systemd-resolved still held stale records because it had a separate cache for stub queries. I had to flush that too. But even after that, pods inside the cluster were resolving the old IP. That's because CoreDNS, the cluster DNS, has its own cache. The default CoreDNS cache size is 128 responses with a 30-second minimum TTL. My 60-second TTL was being extended to effectively infinite because the cache wasn't evicting until it was full.
We ended up editing the CoreDNS ConfigMap to reduce the cache TTL minimum to 5 seconds and restarting CoreDNS. That finally allowed pods to pick up the new record within a minute. The lesson: you have to account for every caching layer. A 60-second TTL is meaningless if the resolver has a minimum TTL of 30 seconds and never evicts until full. Now we set our TTLs to 5 seconds during cutovers and rely on a script that flushes caches at each layer.
Root cause
Multiple layers of DNS caching (systemd-resolved on host, CoreDNS in Kubernetes) with minimum TTL overrides and cache eviction policies that ignored the authoritative 60-second TTL.
The fix
Flushed all caches; reduced CoreDNS cache minimum TTL from 30s to 5s; implemented a DNS migration runbook that flushes each layer in order.
The lesson
Always map the full DNS resolution chain. A short TTL is useless if any resolver enforces a longer minimum cache time. Test DNS changes by querying from each layer (host, container, application) before assuming propagation.
The OS stub resolver is often the first cache hit. systemd-resolved, for example, maintains a separate positive cache even when it forwards to an upstream resolver. It uses a 'stale' feature: if the upstream is unreachable, it serves cached records regardless of TTL. This is controlled by `Cache=yes` and `DNSStubListener=yes` in `/etc/systemd/resolved.conf`.
To check if systemd-resolved is caching, run `resolvectl statistics` and look at the cache hit count. To disable caching entirely, set `Cache=no` in resolved.conf. macOS's mDNSResponder also caches DNS responses and can be flushed with `sudo dscacheutil -flushcache`. On Windows, `ipconfig /displaydns` shows the cache. These caches are often overlooked because they are transparent.
Many runtimes have built-in DNS caches that ignore TTL. Java's default is to cache forever (`networkaddress.cache.ttl=-1`). Python's `socket.getaddrinfo` uses system DNS but the `requests` library may cache connections. Node.js's `dns.lookup` uses the OS cache but `dns.resolve` does not. The worst offender is Java: a JVM that has resolved a hostname will keep the IP until the JVM restarts unless you set `-Dsun.net.inetaddr.ttl=30`.
To debug application caches, enable trace logging. For Java, add `-Dsun.net.inetaddr.ttl=30` or set `networkaddress.cache.ttl=30` in `java.security`. For Go, the default DNS resolver does not cache, but libraries like `net/http` may reuse connections. For Python, use `socket.setdefaulttimeout` and avoid `requests.Session()` without clearing.
Intermediate DNS forwarders like BIND, Unbound, or corporate DNS servers often enforce a minimum TTL (e.g., `cache-min-ttl`). This is meant to reduce load but breaks short TTLs. Unbound's `cache-min-ttl: 300` will extend any TTL below 300 seconds to 300 seconds. Similarly, dnsmasq's `local-ttl` overrides TTL for local domains.
To check for this, query your resolver directly: `dig @<resolver-ip> yourdomain` and compare the TTL with the authoritative. If the TTL is higher than the authoritative, you've found a min-ttl override. Fix: request the network team to lower the min-ttl or use a different resolver. For your own Unbound, set `cache-min-ttl: 0`.
Docker's embedded DNS resolver (127.0.0.11) is a lightweight forwarder that caches responses. It does not always respect TTL strictly — it uses a LRU cache with a default size of 1024 entries. If you restart a container, it may still get cached responses from the host's Docker DNS. You can flush by restarting the Docker daemon (`systemctl restart docker`) or on each node.
In Kubernetes, CoreDNS (or kube-dns) is the cluster DNS. Its default cache plugin caches for 30 seconds minimum. You can change this in the Corefile with `cache 10` (10 seconds). Also, the node's systemd-resolved may interfere if the pod uses the host's DNS. The solution: use `dnsPolicy: Default` or `dnsPolicy: None` with explicit nameservers.
Frequently asked questions
Why does `dig` show the correct IP but my application still uses the old one?
`dig` bypasses most application-level caches and sometimes the OS stub resolver (if you use `@server`). Your application likely has its own DNS cache (e.g., Java, Python requests) or uses the OS stub resolver which may still hold a stale entry. Always test with the actual resolver your application uses.
How do I force a DNS cache flush in Kubernetes?
Flushing DNS cache in Kubernetes is not straightforward because each pod has its own cache (via CoreDNS or kube-dns). You can restart CoreDNS pods (`kubectl rollout restart -n kube-system deployment/coredns`), but that only clears CoreDNS's cache. Pods may still cache DNS in their own runtime. The best approach is to set low TTLs and rely on pod restarts or use `dnsPolicy: None` with custom resolvers.
What is stale caching and how do I disable it?
Stale caching is a feature where a resolver serves expired records to avoid lookup failures when the upstream is unreachable. For example, Unbound's `serve-expired: yes` and systemd-resolved's `Cache=yes` with stale serving. To disable, set `serve-expired: no` in Unbound, `Cache=no` in systemd-resolved, or use a resolver that strictly follows TTL.
Can a CDN like Cloudflare increase TTL beyond what I set?
Yes, if you use Cloudflare's proxy (orange cloud), the TTL you set is only for Cloudflare's edge. The actual TTL for the client is determined by Cloudflare's own caching policy, which has a minimum of 120 seconds for most plans. For DNS-only (gray cloud), the TTL is respected. Check your CDN documentation for 'DNS TTL' vs 'proxy TTL'.
My TTL is 1 second, but clients still see old IP for minutes. Why?
A 1-second TTL is often ignored by many resolvers that enforce a minimum TTL (e.g., 30 seconds). Also, clients may have TCP connection pools that reuse connections regardless of DNS. Additionally, some operating systems (like Windows) have a minimum DNS cache entry lifetime of 1 second but actually round up to 1 minute. Use a TTL of at least 5 seconds and ensure all resolvers have min-ttl set to 0.