What this usually means
Cassandra handles deletes by writing tombstones — markers that indicate data is deleted. During a read, Cassandra must reconcile all replicas and merge tombstones with live data. If a partition accumulates many tombstones (e.g., from frequent deletes or TTL expirations without compaction), the coordinator can spend excessive time merging, eventually exceeding the read timeout (default 5s or 10s). The root cause is often a mismatch between the workload pattern (frequent deletes/updates) and the compaction strategy (SizeTiered vs Leveled) or compaction settings (tombstone threshold, gc_grace_seconds).
The first ten minutes — establish facts before touching code.
- 1Run `nodetool tablestats <keyspace>.<table>` and look for 'SSTable count' and 'Estimated tombstone drop time'. If SSTable count > 50 and tombstones are high, proceed.
- 2Run `nodetool cfhistograms <keyspace>.<table>` and check if read latency histogram shows a long tail (p99 > 5000ms).
- 3Enable tracing on a single problematic query: `tracing on; SELECT * FROM <table> WHERE pk = X;` and examine the trace for 'Scanned xxx tombstones'.
- 4Check compaction logs (system.log) for lines like 'tombstone ratio 0.2', 'Compacted 100 sstables to 1' with high tombstone counts.
- 5View GC logs (gclog.log) for long pauses: `grep 'Pause Young' gclog.log | tail -20` — if young gen pauses exceed 300ms under load, tombstones are likely bloating the heap.
The specific files, logs, configs, and dashboards that usually own this bug.
- searchsystem.log — search for 'Tombstone' or 'tombstone ratio'
- searchnodetool tablestats output — focus on 'SSTable count', 'Space used (live)', 'Estimated tombstone drop time'
- searchnodetool cfhistograms output — read latency histogram
- searchQuery tracing results via cqlsh or DataStax Studio
- searchGC log (usually /var/log/cassandra/gc.log) — look for young-gen and old-gen pause duration
- searchnodetool compactionstats — see if compaction is falling behind
- searchJMX metrics (via jconsole or prometheus) — org.apache.cassandra.metrics.Table.TombstoneScannedHistogram
Practical causes, not theory. These are the things you will actually find.
- warningCompaction strategy set to SizeTieredCompactionStrategy (STCS) with high write/delete volume — STCS does not aggressively compact tombstones, allowing them to accumulate in many overlapping SSTables
- warninggc_grace_seconds set too high (default 864000 = 10 days) giving tombstones more time to accumulate before being eligible for dropping
- warningFrequent UPDATE operations that rewrite whole rows — each update creates a tombstone for the old version if the row is wide
- warningTTL-based expiry on a high-write table where the TTL is long (e.g., days) but writes are frequent — tombstones from TTL expirations pile up
- warningLack of manual compaction or repair: if compaction is disabled or falling behind, tombstones remain in SSTables indefinitely
- warningApplication pattern: batch deletes on a single partition key (e.g., deleting all records for a user in a time-series table) creates massive tombstone per-partition
Concrete fix directions. Pick the one that matches your root cause.
- buildSwitch to LeveledCompactionStrategy (LCS) for tables with high tombstone generation — LCS compacts smaller sets of SSTables and drops tombstones faster
- buildIncrease tombstone compaction thresholds: set tombstone_compaction_interval (default 86400s) lower and tombstone_threshold (default 0.2) to 0.5 or higher to force compaction when tombstone ratio is high
- buildReduce gc_grace_seconds to match your application's eventual consistency requirements (e.g., 2 days instead of 10) after ensuring repairs complete within that window
- buildRedesign data model: avoid wide partitions; use separate tables for frequently deleted data; use TTL for automatic expiry instead of explicit deletes where possible
- buildRun an aggressive compaction cycle: `nodetool compact <keyspace> <table>` to force SSTable merge and drop tombstones (careful with production load)
- buildIncrease read timeout (read_request_timeout_in_ms in cassandra.yaml) as a temporary band-aid, but only while addressing root cause
A fix you cannot prove is a guess. Close the loop.
- verifiedAfter compaction, run `nodetool tablestats <keyspace>.<table>` and verify 'SSTable count' decreased and 'Estimated tombstone drop time' is now negative (meaning all tombstones dropped)
- verifiedRun the same traced query and verify 'Scanned tombstones' dropped to near zero
- verifiedCheck read latency histograms via nodetool cfhistograms — p99 should be under 100ms
- verifiedMonitor client-side timeouts: they should drop to zero after fix
- verifiedCheck compaction logs for reduced tombstone ratio (e.g., < 0.05) after compactions complete
Things that make this bug worse or harder to find.
- warningSetting gc_grace_seconds to 0 — this will cause data resurrection if a replica misses a repair
- warningRunning nodetool compact on all keyspaces simultaneously during peak traffic — it can overwhelm disk I/O and cause cascading timeouts
- warningSwitching to LCS without adjusting sstable_size_in_mb — default 160MB may cause too many SSTables per level if writes are bursty
- warningIgnoring repair schedules: if you lower gc_grace_seconds, you must run repair more often to avoid data resurrection
- warningBlindly increasing read timeout without addressing tombstone accumulation — this only masks the problem and can lead to heap exhaustion
- warningNot monitoring tombstone ratio over time — a one-time fix without alerting will let the problem recur
The 5-Second Read Timeout That Wasn't a Network Issue
Timeline
- 09:15PagerDuty alert: read timeout rate > 5% for keyspace 'metrics' table 'events'.
- 09:18Check nodetool tpstats: read stage tasks pending, but no exception in system.log.
- 09:20Run nodetool cfhistograms events: read latency p99 = 8700ms, p999 = 15000ms.
- 09:22Enable tracing on a sample read: found 'Scanned 1,200,000 tombstones' for a single partition key.
- 09:25Check GC logs: young-gen pauses averaging 350ms, old-gen GC running every 10 minutes.
- 09:30Run nodetool tablestats events: 120 SSTables with 'Estimated tombstone drop time' in 7 days.
- 09:35Decision: force compaction on the events table during maintenance window.
- 10:00Run 'nodetool compact metrics events' — took 15 minutes, reduced SSTables from 120 to 4.
- 10:20Verify: cfhistograms shows p99 read latency = 45ms, timeouts drop to 0.
- 10:30Root cause identified: application sends frequent DELETE on old rows without TTL, STCS doesn't compact aggressively.
- 11:00Changed compaction to LCS, reduced gc_grace_seconds to 172800, added TTL to rows.
The alert came in at 9:15 AM: read timeout rate exceeded 5% for the 'events' table. My first thought was network or disk I/O, but nodetool tpstats showed no backpressure. I ran cfhistograms and saw a massive latency tail — p99 at 8.7 seconds. That's not a slow disk; that's something blocking reads.
I enabled tracing on a query that was timing out. The trace showed the coordinator scanning 1.2 million tombstones for a single partition key. That's when it clicked: tombstones. The events table had a pattern of batch-deleting old records per user, and STCS was not compacting fast enough. GC logs confirmed young-gen pauses of 350ms from tombstone overhead.
I forced a compaction during a maintenance window. It merged 120 SSTables into 4, dropping millions of tombstones. Read latency dropped from 8.7s to 45ms. The permanent fix was switching to LCS and reducing gc_grace_seconds to match our repair window. We also added TTL to rows so they'd expire naturally instead of being explicitly deleted.
Root cause
SizeTieredCompactionStrategy (STCS) with high-frequency deletes leads to tombstone accumulation across many overlapping SSTables, causing reads to scan millions of tombstones per partition.
The fix
Switched to LeveledCompactionStrategy (LCS), reduced gc_grace_seconds from 10 days to 2 days, added TTL to rows, and forced a compaction to clear existing tombstones.
The lesson
Monitor tombstone ratio proactively with nodetool tablestats and set alerts on tombstone_scanned_histogram. Compaction strategy choice must match write/delete pattern — STCS is not suitable for heavy deletes.
Cassandra uses tombstones as a delete marker. When a read hits a partition, the coordinator must merge all replicas' responses, including tombstones. The merging logic walks through all SSTables that contain the partition. If there are many SSTables (e.g., 100+ from STCS) and each has many tombstones, the coordinator may need to merge millions of tombstone entries. This is CPU and memory intensive.
The read timeout (default 5s in Cassandra 2.x, 10s in 3.x) is the maximum time the coordinator waits for a replica to respond. If the merging takes longer, the coordinator throws a ReadTimeoutException. The client sees a timeout, even though the data is available. The actual bottleneck is not the disk but the CPU doing reconciliation.
SizeTieredCompactionStrategy (STCS) compacts SSTables of similar size. This means many overlapping SSTables can exist for the same partition, each with its own tombstones. STCS does not prioritize tombstone removal — it only merges when size tiers match. Over time, tombstone density increases.
LeveledCompactionStrategy (LCS) divides SSTables into levels (L0, L1, L2...). It ensures that within a level, SSTables are non-overlapping. LCS compacts more frequently and promotes SSTables to higher levels, dropping tombstones earlier. For write-heavy workloads with deletes, LCS is almost always better. However, LCS uses more disk I/O for compaction, so monitor that.
gc_grace_seconds controls how long a tombstone lives before being eligible for garbage collection during compaction. Default is 864000 (10 days). If you repair regularly (daily), you can safely reduce this to 172800 (2 days) or less. Setting it too low risks data resurrection if a node misses a repair.
You can also set tombstone_compaction_interval (default 86400s) to run compaction more frequently when tombstone ratio is high. The tombstone_threshold (default 0.2) triggers compaction if the ratio of tombstones to live data exceeds 20%. Lowering this threshold forces compaction sooner. But be careful: too aggressive compaction can cause write amplification.
The fastest way to confirm tombstone issues is query tracing. In cqlsh, run 'tracing on;' then execute the slow query. Look for lines like 'Scanned 50000 tombstones' or 'Merged 1200 tombstones'. If the number exceeds 100,000 per partition, you have a problem.
JMX metrics: org.apache.cassandra.metrics.Table.TombstoneScannedHistogram gives distribution of tombstones scanned per read. Set alerts on the 99th percentile exceeding 10,000. Also monitor Compaction.TotalCompactionsCompleted — if it's low compared to flush rate, compaction is falling behind.
Emergency: Force compaction with 'nodetool compact <keyspace> <table>'. This is blocking — it will merge all SSTables for that table into one, dropping all tombstones. Expect high disk I/O. Run during low traffic. Alternatively, if you can't force compaction, increase read timeout temporarily (e.g., to 30s) and reduce concurrency.
Long-term: Switch to LCS, tune gc_grace_seconds, add TTL to rows, and redesign partition keys to avoid wide partitions. Also consider using TWCS (TimeWindowCompactionStrategy) for time-series data — it naturally segregates SSTables by time window, reducing tombstone overlap. Finally, implement a tombstone budget per partition in your application logic.
Frequently asked questions
What is the difference between a tombstone read timeout and a normal read timeout?
A normal read timeout can be caused by network issues, slow disks, or overloaded nodes. A tombstone read timeout is specifically caused by the coordinator spending too much time merging tombstones during read repair. The key indicator is seeing high tombstone counts in traces and high tombstone ratio in compaction logs.
Can I prevent tombstones entirely?
No, tombstones are inherent to Cassandra's eventual consistency model. But you can minimize their impact by using TTL for automatic deletion (which also creates tombstones, but they are batch-removed sooner), avoiding frequent updates on wide rows, and using a compaction strategy that drops tombstones aggressively (LCS or TWCS).
How often should I run nodetool compact to clear tombstones?
You should not rely on manual compaction. Instead, configure compaction strategy and thresholds so that compactions happen automatically. If you must run manual compaction, do it only during maintenance windows and only on the affected table. Running it too often causes write amplification.
What does 'Estimated tombstone drop time' mean in nodetool tablestats?
This is the earliest time when all tombstones in that SSTable will be eligible for dropping (based on gc_grace_seconds). If the value is in the future, tombstones are still present. If it's negative, the time has passed and tombstones can be dropped during next compaction. A far-future value indicates a long gc_grace_seconds or recent deletes.
Should I use LCS for all tables?
No. LCS is best for write-heavy workloads with many updates/deletes. For read-heavy or append-only workloads, STCS may be more efficient because it produces fewer SSTables overall. Always test with your workload. Also, LCS requires more disk space for temporary compaction and more I/O, so ensure your disk can handle it.