What this usually means
The core issue is that caching is a contract between server and client, and any violation breaks it. The most common non-obvious causes are: (1) The Vary header lists dynamic values like User-Agent or Cookie, which creates separate cache entries per variation, effectively behaving like no caching. (2) Cache-Control directives conflict; for example, max-age=0 alongside ETag tells the client the response is stale immediately, so it always revalidates. (3) The response sets Cache-Control: private, which proxies (and sometimes browsers) interpret as non-cacheable. (4) The origin server sends no-cache or must-revalidate without a max-age, which forces revalidation but doesn't prevent caching — though many implementations wrongly assume it means 'don't cache'. (5) The ETag is weak (W/ prefix) and the request uses a strong validator requirement. (6) The server sends a Last-Modified that is in the future, which some clients treat as uncacheable.
The first ten minutes — establish facts before touching code.
- 1curl -I https://example.com/resource | grep -iE '(cache-control|etag|vary|last-modified)' — check all caching headers
- 2In Chrome DevTools Network tab, click a resource and inspect 'Size' column: 'from disk cache' vs 'from memory cache' vs actual size. If always 'from memory cache', it's session-only caching.
- 3Use curl with --etag-compare and --etag-save to simulate a conditional request: curl -I -H 'If-None-Match: "abc123"' https://example.com/resource — expect 304
- 4Check the Vary header: curl -sI https://example.com/resource | grep -i vary. If it includes User-Agent, Cookie, or Accept-Encoding, that fragments your cache.
- 5In Chrome, open chrome://net-export/ and capture a log. Look for CACHE events to see exactly why the cache decided not to store the response.
- 6Review server config (nginx: add_header Cache-Control ...; Apache: Header set Cache-Control ...). Many misconfigurations overwrite instead of append.
The specific files, logs, configs, and dashboards that usually own this bug.
- searchnginx config: /etc/nginx/conf.d/default.conf or site-specific — look for add_header Cache-Control directives
- searchApache config: .htaccess or httpd.conf — Header set Cache-Control directives
- searchCDN dashboard (CloudFront, Cloudflare, Fastly): caching behavior rules, edge TTL, and cache key settings
- searchApplication framework config: Rails config.static_cache_control, Django Cache-Control middleware, Express helmet/caching
- searchBrowser DevTools: Network tab → Response Headers, also 'Size' column shows cache source
- searchServer access logs: look for 304 vs 200 status codes per resource; high 200s with ETags present indicate caching failure
- searchcurl verbose output: curl -v https://example.com/resource 2>&1 | grep -E '(< |Cache)' — see all headers
Practical causes, not theory. These are the things you will actually find.
- warningVary header includes User-Agent or Cookie, creating too many cache variations — effectively no caching
- warningCache-Control: max-age=0 or no-cache is set alongside ETag, forcing revalidation on every request (no cached copy stored)
- warningCache-Control: private (or no-store) prevents intermediate proxies and sometimes browsers from caching
- warningServer sends Last-Modified in the future, causing cache to reject the response as uncacheable
- warningWeak ETag (W/"abc") used with a request that expects a strong validator (e.g., Range requests)
- warningProxy or CDN strips or overwrites ETag or Cache-Control headers (common with CloudFlare when cache rules are bypassed)
- warningResponse contains Set-Cookie header, which many caches treat as non-cacheable (unless configured otherwise)
Concrete fix directions. Pick the one that matches your root cause.
- buildRemove Vary header or restrict it to Accept-Encoding only. If you need Vary: User-Agent, consider using a cache key that includes only the first few chars of User-Agent or normalize it.
- buildSet Cache-Control: public, max-age=31536000, immutable for static assets. Ensure no conflicting directives like no-cache or private.
- buildFor dynamic content, use ETag + Cache-Control: no-cache (not no-store) to allow conditional requests (304) while preventing stale cache.
- buildEnsure Last-Modified is in the past and consistent. Use a fixed date or derive from file modification time.
- buildUse strong ETags (without W/ prefix) for byte-range serving and conditional GETs. Weak ETags are acceptable for most browsers but can cause issues with some proxies.
- buildCheck CDN settings: for CloudFront, ensure 'Cache Based on Selected Request Headers' is not set to 'All' or 'User-Agent'. Use 'Whitelist' with only Accept-Encoding.
- buildIf Set-Cookie is needed, consider using a separate domain for static assets (e.g., static.example.com), which won't receive cookies.
A fix you cannot prove is a guess. Close the loop.
- verifiedAfter fix, curl -I returns expected Cache-Control and no conflicting headers. Then curl with If-None-Match returns 304.
- verifiedChrome DevTools: load resource twice, see '200 (from disk cache)' or '304' on second load.
- verifiedIncognito window: load resource, close window, reopen, load again — should see 'from disk cache'.
- verifiedUse curl -w '%{http_code}' -o /dev/null -H 'If-None-Match: "<etag>"' — expect 304.
- verifiedCheck CDN hit ratio metrics: should increase from 0% to >90%.
- verifiedRun a WebPageTest test and review 'Caching' grade; it should pass for static assets.
Things that make this bug worse or harder to find.
- warningDon't blindly set Cache-Control: max-age=0 as a catch-all; it forces revalidation but still stores the response in cache (with ETag). Use no-cache for that.
- warningAvoid setting both no-cache and max-age in the same response — max-age wins but can cause confusion.
- warningDon't combine Vary: * with caching — Vary: * means vary on everything, effectively disabling cache.
- warningDon't rely on browser 'disable cache' checkbox being unchecked — it only affects DevTools, not real users.
- warningNever set Last-Modified to a future date or dynamic value like current time on every request — it breaks caching.
- warningDon't forget that HTTPS and cookies can affect proxy caching; if you need caching behind HTTPS, ensure your CDN supports it and is configured to cache.
CDN Caching Failure: ETag Present but Zero Hit Rate
Timeline
- 09:15PagerDuty alert: CDN cache hit rate dropped to 0% for /api/v1/products endpoint
- 09:20Check CloudFront metrics: origin request count spiked 10x, all requests go to origin
- 09:25curl -I api.example.com/api/v1/products returns ETag and Cache-Control: public, max-age=3600
- 09:30Conditional GET test: curl -H 'If-None-Match: "xxx"' returns 200 (not 304) — no caching!
- 09:35Inspect response headers: Vary: Cookie, User-Agent, Accept-Encoding found
- 09:40Check Nginx config: add_header 'Vary: Cookie, User-Agent, Accept-Encoding' was added for API versioning
- 09:45Remove Vary header from Nginx config, restart, re-test: conditional GET returns 304
- 09:50Monitor CloudFront metrics: cache hit rate recovers to 95% within 10 minutes
At 09:15, I got paged that the CDN cache hit rate for our API had dropped to zero. We had recently deployed a new version with API versioning headers. I immediately checked CloudFront metrics and saw that every request was going to origin — no cache hits. I thought maybe the Cache-Control headers were missing.
I curled the endpoint and saw ETag and Cache-Control: public, max-age=3600 were present. That looked fine. But then I tested a conditional GET with If-None-Match and got 200 instead of 304. That meant CloudFront wasn't even storing the response. I started thinking maybe the origin was setting something that prevented caching.
I used curl -v to see all response headers and spotted the Vary header: it included Cookie and User-Agent. CloudFront by default uses the entire Vary header to build the cache key. That means each unique combination of Cookie and User-Agent creates a different cache entry. Since most users have unique cookies, the cache was effectively disabled. I removed the Vary header from our Nginx config (it was a mistake from the versioning feature) and within minutes the hit rate recovered to 95%.
The root cause was the Vary header. It's a common misconfiguration — teams add Vary for legitimate reasons but don't realize how it fragments caching at CDN level. The fix was to remove Vary or use only Accept-Encoding. For API versioning, we moved version info into the URL path instead.
Root cause
Nginx config included `add_header 'Vary: Cookie, User-Agent, Accept-Encoding'` for API versioning, which caused CloudFront to cache each unique combination separately, rendering caching useless.
The fix
Removed the custom Vary header from Nginx config. For API versioning, switched to URL path versioning (e.g., /api/v2/products).
The lesson
Always check Vary header when debugging caching failures. CDNs often respect Vary strictly, and including dynamic values like Cookie or User-Agent fragments the cache to zero.
The Vary header tells intermediate caches that the response can vary based on request headers listed. If Vary: User-Agent is set, the cache must store a separate entry for each unique User-Agent string. Since there are thousands of distinct User-Agent values, the effective cache hit rate approaches zero.
Commonly, frameworks add Vary: Cookie to prevent caching of personalized content. But if your public API doesn't vary by cookie, including Cookie in Vary is catastrophic. The fix: only add Vary for headers that actually affect the response. For static assets, omit Vary entirely or use only Accept-Encoding.
At CDN level, some providers (like CloudFront) allow you to control which headers are included in the cache key. You can override the Vary behavior by adding a custom cache key policy that ignores certain headers. But the simplest fix is to not send Vary for headers that don't matter.
Many engineers confuse max-age=0 with no-cache. max-age=0 means the response is stale immediately, so the client must revalidate with a conditional request (If-None-Match) before each use. However, the response is still stored in cache. So if you set max-age=0 plus ETag, you'll get 304 on revalidation — not ideal for performance, but it works.
no-cache, on the other hand, means the response must not be used to satisfy a subsequent request without successful revalidation. It does NOT mean 'do not cache' — the response can still be stored. Some implementations incorrectly treat no-cache as no-store. To truly prevent caching, use no-store.
If you see both max-age=0 and no-cache, the behavior is effectively the same as no-cache. The important thing: if you want caching, ensure max-age is a positive number (e.g., 3600) and avoid no-cache or no-store. If you want conditional caching (always revalidate), use no-cache with ETag — not max-age=0.
ETags come in two flavors: strong (e.g., "abc123") and weak (W/"abc123"). Strong ETags guarantee that the resource is byte-for-byte identical. Weak ETags only guarantee semantic equivalence — the response may differ slightly (e.g., whitespace or encoding).
For most browser caching and CDN conditional requests, weak ETags are fine. However, certain scenarios require strong ETags: Range requests (partial content) require strong validation because a weak ETag doesn't guarantee byte identity. Also, some CDNs (like Akamai) may reject weak ETags for caching. If you see browsers sending Range headers and never getting 304, check if your ETag is weak.
The fix: generate strong ETags by hashing the complete response body (e.g., MD5 or SHA1). Avoid using last-modified or file size alone. In Rails, you can use `fresh_when` with `strong_etag: true`. In nginx, you can disable weak ETag generation with `etag on;` (but it uses a weak ETag by default — check version).
curl is your first line of defense. Use `curl -I` to see headers. Use `curl -H 'If-None-Match: "etag"' -w '%{http_code}' -o /dev/null` to test conditional GET. Also use `curl -H 'Cache-Control: max-age=0'` to simulate a fresh request.
For browser-specific issues, Chrome's net-export tool is invaluable. Go to chrome://net-export, start logging, reproduce the issue, stop logging, and open the log at chrome://net-internals/#import. Filter for 'CACHE' events. You'll see entries like 'CACHE_ENTRY_NOT_CACHEABLE' with a reason (e.g., 'Vary header mismatch', 'Set-Cookie present', 'Response is stale').
Also use the DevTools Network tab: click a resource and check the 'Headers' tab for the 'Cache-Control' and 'ETag'. The 'Size' column tells you the cache source: 'from disk cache' (persistent) vs 'from memory cache' (session-only). If you never see 'from disk cache', your response is not being stored persistently — check for no-store or private.
Frequently asked questions
Why does my browser show 'from memory cache' but never 'from disk cache'?
This is a sign that the response is cached only for the current session. Common causes: Cache-Control: no-store or private is set, or the response includes a Set-Cookie header. Also, HTTPS resources with certain Cache-Control directives may be memory-cached only. Check your Cache-Control header and ensure it's 'public' with a max-age > 0.
I have ETag and Cache-Control: public, max-age=3600, but CDN still shows 0% hit rate. What could be wrong?
Check the Vary header. If Vary includes User-Agent, Cookie, or other varied headers, the CDN will create separate cache entries per variation. Also check if the CDN is configured to ignore query strings or headers that cause cache key fragmentation. Another possibility: the origin is returning a Set-Cookie header that the CDN interprets as non-cacheable.
What is the difference between 'no-cache' and 'no-store'?
'no-cache' means the response can be stored in cache, but it must be revalidated with the origin before every use (using conditional headers). 'no-store' means the response must not be stored in any cache at all. If you want to use ETags to validate freshness but still allow caching, use 'no-cache'. If you want to prevent any caching, use 'no-store' (and possibly 'private' for user-specific data).
My server returns 304 on conditional GET, but the browser still makes a full request. Why?
A 304 response should be cached. But if the browser receives a 304 with different caching headers (e.g., Cache-Control: no-cache), it may invalidate the cache. Also, if the original response had a Vary header that the 304 response lacks, the browser may treat the cache entry as invalid. Ensure the 304 response mirrors the original caching headers.
Can I use both ETag and Last-Modified together?
Yes, they are complementary. Last-Modified provides a fallback for clients that don't support ETag. However, some servers send both, and browsers may use either. If you send both, ensure they are consistent: if the resource changes, both should change. Otherwise, you risk confusing the cache. In practice, ETag is more precise, so prefer it over Last-Modified.