What this usually means
DynamoDB partitions data based on partition key hash. When a single partition key value (or a small set) generates a disproportionate amount of traffic, that partition's throughput limit is hit while other partitions have spare capacity. This is not a capacity planning failure at the table level—your total provisioned throughput may be fine—but the data distribution is uneven. Common causes include a 'hot' user, a popular event, a timestamp-based key with high write volume on a single time unit, or a GSI with low cardinality. The fix involves either redistributing the key space, using write sharding, or implementing caching.
The first ten minutes — establish facts before touching code.
- 1Check CloudWatch metric 'ThrottledWriteEvents' for the table, split by partition (enable per-partition metrics). Look for one partition with >80% of throttling.
- 2In CloudWatch Logs Insights, query for ProvisionedThroughputExceededException in your application logs with the specific table name and partition key value.
- 3Run a script to count items per partition key value using DynamoDB Scan with a projection of the partition key (beware of large tables—use parallel scan or export to S3).
- 4If using DAX, check DAX metrics for cache hit ratio—low hit ratio may indicate hot keys bypassing cache.
- 5Check DynamoDB API call rate by partition key using AWS CloudTrail—filter by eventName like 'PutItem' and 'UpdateItem' and group by requestParameters.partitionKey.
The specific files, logs, configs, and dashboards that usually own this bug.
- searchCloudWatch Metrics: DynamoDB > Per-Table Metrics > ThrottledWriteEvents, ThrottledReadEvents, ConsumedWriteCapacityUnits, ConsumedReadCapacityUnits (with per-partition dimension)
- searchCloudWatch Logs: application logs filtered by ProvisionedThroughputExceededException or HTTP 400 status codes
- searchAWS CloudTrail: Look for DynamoDB API calls with errorCode 'ProvisionedThroughputExceeded'
- searchDynamoDB Console: Table 'Metrics' tab shows throttling events, but per-partition requires CloudWatch
- searchApplication tracing (X-Ray): segments showing DynamoDB calls with 'throttle' flag or retry count >3
- searchS3 Export of DynamoDB table (if large) to analyze partition key distribution using Athena
Practical causes, not theory. These are the things you will actually find.
- warningSingle partition key value (e.g., user_id='12345') receives >80% of all writes due to viral activity or batch job.
- warningTime-based partition key (e.g., YYYY-MM-DD) where all writes go to today's partition.
- warningGSI with low cardinality (e.g., status='active') causing hot partition on the index.
- warningInsufficient write sharding: not appending random suffix or hash to high-volume keys.
- warningBurst capacity depletion: all partitions exhausted burst even though table WCU not exceeded, due to uneven traffic.
Concrete fix directions. Pick the one that matches your root cause.
- buildImplement write sharding by appending a random number (e.g., 0-9) to the partition key to spread writes across partitions.
- buildIf the hot key is known and limited, use a cache (ElastiCache or DAX) for read-heavy workloads to reduce direct reads.
- buildIncrease provisioned WCU/RCU on the table—but this only helps if other partitions also need capacity (often not effective alone).
- buildUse DynamoDB auto-scaling with a higher target utilization to absorb spikes, but still limited by partition-level limits.
- buildRedesign the primary key to use a composite key with a high-cardinality attribute (e.g., user_id + timestamp) to avoid hot keys.
A fix you cannot prove is a guess. Close the loop.
- verifiedAfter fix, monitor CloudWatch metric 'ThrottledWriteEvents' per partition—should show even distribution across partitions.
- verifiedSet up a CloudWatch alarm on 'ThrottledWriteEvents' with threshold 0 for 5 minutes to catch regressions.
- verifiedUse DynamoDB 'Contributor Insights' to see which partition keys are causing throttling—ensure the hot key no longer appears.
- verifiedRun a load test that simulates the previous hot traffic pattern and verify no throttling errors.
- verifiedCheck application logs for absence of ProvisionedThroughputExceededException over a 24-hour period.
Things that make this bug worse or harder to find.
- warningJust increasing table capacity without addressing the hot key—you'll still throttle on that partition.
- warningUsing write sharding without proper read logic: reads must know the shard to query all shards in parallel.
- warningIgnoring GSI hot partitions: even if main table is fine, a GSI can be the bottleneck.
- warningNot cleaning up old shards: over time, write sharding creates many small items if you add a random suffix without a time-based component.
- warningApplying a uniform random shard to all writes: some keys may still be hot; consider adaptive sharding based on traffic.
The Viral Tweet That Brought Down Our Leaderboard
Timeline
- 09:15PagerDuty alert: ProvisionedThroughputExceeded exceptions on 'user_scores' table.
- 09:17Check CloudWatch: ThrottledWriteEvents at 45% of total writes, but table consumed WCU only at 60%.
- 09:22Enable per-partition metrics in CloudWatch; see one partition with 99% throttling.
- 09:25Identify partition key: 'game_session_12345' is the hot key—a viral tweet linked to that session.
- 09:30Quick fix: temporarily increase table WCU from 1000 to 5000 to absorb the spike.
- 09:35Monitor: throttling drops but not eliminated; still throttling on that partition because per-partition limit is 1000 WCU.
- 09:40Implement write sharding: append random suffix 0-9 to partition key in Lambda handler.
- 09:50Deploy hotfix; throttling errors drop to zero within 2 minutes.
We run a gaming service where each game session writes scores to a 'user_scores' table. The partition key was 'game_session_id', and we had a viral tweet promoting a specific session—'game_session_12345'. Suddenly, thousands of users wrote scores to that one partition. Our table had 1000 WCU, which meant each partition could handle 1000 WCU. But that single partition was receiving 5000 WCU, causing massive throttling.
Our first instinct was to increase table capacity, but we soon realized that didn't help because the per-partition limit is 1/1000 of total capacity (with a minimum of 1000). So even with 5000 WCU, that partition only got 5000/ (number of partitions) but still couldn't handle the skew. We needed to spread the writes across more partitions.
We implemented write sharding by appending a random number 0-9 to the partition key. That way, writes for 'game_session_12345' were distributed across 10 partitions, each receiving ~500 WCU—well within limits. Reads became more complex: we had to query all 10 shards and merge results. We also added a cache for the hot session to reduce read pressure. The fix took 20 minutes to code and deploy. After that, no more throttling.
Root cause
Single partition key 'game_session_12345' received disproportionate write traffic due to viral event, exceeding per-partition throughput limit.
The fix
Write sharding by appending random suffix 0-9 to partition key, plus caching hot reads.
The lesson
Always design partition keys with high cardinality and consider traffic spikes. Use write sharding for known hot keys. Don't rely solely on table-level capacity.
Each DynamoDB partition can handle up to 3000 RCU or 1000 WCU, and up to 10 GB of data. When you provision 1000 WCU on a table, DynamoDB creates at least one partition. If you later increase capacity, it splits partitions, but each partition's throughput limit is fixed. The key insight: your table's total throughput is the sum of all partition throughputs, but traffic must be evenly distributed across partitions to use it all.
A hot partition occurs when one partition key's traffic exceeds 1000 WCU (or 3000 RCU) even if the table's total usage is low. For example, a table with 5000 WCU might have 10 partitions (each 500 WCU average), but if one key gets 2000 writes/sec, that partition throttles. The rest of the capacity is wasted. Always look at per-partition metrics, not just table-level.
To see per-partition metrics, you must enable 'Per-Partition CloudWatch Metrics' in the table settings. Then navigate to CloudWatch > Metrics > DynamoDB > Per-Table Metrics. You'll see dimensions like 'TableName' and 'PartitionId'. Filter by 'ThrottledWriteEvents' and group by PartitionId. A single partition with high throttling indicates a hot key.
If you cannot enable per-partition metrics (cost or time), use CloudTrail to find the specific partition key. Filter DynamoDB PutItem events with errorCode 'ProvisionedThroughputExceededException' and look at the requestParameters item for the partition key value. This gives you the exact hot key.
Write sharding spreads writes for a hot key across multiple logical partitions. The simplest approach: append a random suffix (e.g., 0-9) to the partition key. For example, instead of 'session_123', use 'session_123_0', 'session_123_1', etc. This distributes writes across 10 partitions. Choose the shard count based on expected write volume: for 10,000 writes/sec, use 10 shards (each gets 1000).
The trade-off: reads must now query all shards and merge results. For point reads, you need to know which shard the item is in (e.g., by hashing the original key). For queries that scan many items, you may need to query all shards in parallel. Consider using a global secondary index with a different key schema to avoid this complexity.
Contributor Insights is a managed feature that automatically identifies hot keys and throttled requests. Enable it on your table (costs extra). It provides a dashboard showing the top partition keys by throttling count. This is the fastest way to identify the offending key without manual log analysis.
Once enabled, go to DynamoDB Console > Table > Contributor Insights. You'll see 'Throttled requests' and 'Top partition keys'. Click on a key to see its traffic pattern. Use this information to apply write sharding or caching to that specific key.
DynamoDB provides burst capacity: if you don't use throughput for a period, you can burst above your provisioned limit for up to 5 minutes (300 seconds). However, burst capacity is per-partition. A hot partition quickly depletes its burst and starts throttling while other partitions have unused burst.
Adaptive capacity (now default) allows DynamoDB to reallocate unused throughput from idle partitions to busy ones, but only within the same partition's limit? Actually, adaptive capacity increases the partition's throughput limit temporarily if other partitions are underutilized. But it has limits—cannot exceed 2x the per-partition limit. Still, the best solution is to fix the key distribution.
Frequently asked questions
How do I know if my table has hot partitions without per-partition metrics enabled?
You can infer hot partitions by looking at CloudWatch metric 'ThrottledWriteEvents' vs 'ConsumedWriteCapacityUnits'. If throttling occurs while consumed capacity is below provisioned, it's likely a hot partition. Also, use CloudTrail to find the specific partition key from error events, or enable Contributor Insights.
Will increasing provisioned capacity fix a hot partition?
Only if the hot partition's traffic is below the per-partition limit (1000 WCU or 3000 RCU). If the traffic exceeds that, increasing table capacity may add more partitions, but the hot key still lands on one partition. You need to redistribute the key space, not just add capacity.
What is the difference between write sharding and using a GSI?
Write sharding adds a random suffix to the partition key to spread writes across multiple partitions. A GSI might have a different key schema that better distributes traffic. However, GSIs also have their own partition limits and can become hot. Write sharding is more granular but requires application-level changes.
How do I handle reads after write sharding?
For point reads, you need to compute the shard from the original key (e.g., by hashing). For queries that need all items, you must query all shards and merge results in your application. Consider using a secondary index with the original key as sort key to allow efficient querying.
Can DynamoDB auto-scaling prevent hot partitions?
Auto-scaling adjusts table-level capacity, not partition-level. It can help if the traffic is distributed evenly, but it cannot fix a single hot partition. You still need to address the key distribution.