What this usually means
You're sending more read/write requests per second than your DynamoDB table's provisioned capacity allows, or—more insidiously—your workload has a hot partition. DynamoDB distributes items across partitions by partition key; a single partition only gets 1/(partition count) of total throughput. Even if the table's total capacity looks fine, one partition can be overwhelmed. The error surfaces when the request rate exceeds the provisioned throughput for that partition or when burst capacity (5 minutes of unused capacity) is exhausted. Auto Scaling can lag behind spikes, and GSIs have their own separate capacity pools.
The first ten minutes — establish facts before touching code.
- 1Check CloudWatch metric ConsumedWriteCapacityUnits vs ProvisionedWriteCapacityUnits averaged over 1 minute; look for sustained consumption above 100%.
- 2Run: aws dynamodb describe-table --table-name your-table — examine TableStatus, ProvisionedThroughput, and GlobalSecondaryIndexes for their throughput settings.
- 3Query CloudWatch Logs Insight for ProvisionedThroughputExceededException in your application logs; grep for the exact error string.
- 4Use CloudWatch Contributor Insights for DynamoDB to identify the most throttled partition keys (requires Contributor Insights enabled).
- 5Run a test: aws dynamodb get-item --table-name your-table --key '{"pk":{"S":"test"}}' with --return-consumed-capacity TOTAL to see consumed units per request.
The specific files, logs, configs, and dashboards that usually own this bug.
- searchCloudWatch Metrics: DynamoDB → Per-Table Metrics → ConsumedReadCapacityUnits, ConsumedWriteCapacityUnits, ThrottledRequests, SystemErrors.
- searchCloudWatch Logs: application logs (e.g., /var/log/app/error.log) for ProvisionedThroughputExceededException stack traces.
- searchDynamoDB Console → Tables → Monitor tab: Throttling events, Top Partition Keys by Throttled Requests.
- searchAWS CloudTrail: Look for DynamoDB.PutItem or DynamoDB.Query events with errorCode ProvisionedThroughputExceeded.
- searchAuto Scaling configuration: Check DynamoDB Application Auto Scaling target tracking policies and CloudWatch alarms for scaling.
- searchSource code: Look for retry logic (exponential backoff) and partition key selection logic.
Practical causes, not theory. These are the things you will actually find.
- warningHot partition: a few partition keys absorb the majority of traffic, exceeding partition throughput (1/table capacity per partition).
- warningAuto Scaling lag: target tracking policy has large step adjustments or too high a cooldown period; traffic spikes faster than scaling can respond.
- warningGSI throttling: GSI has insufficient provisioned capacity and no auto scaling; writes to base table consume GSI capacity, causing throttling on GSI.
- warningBurst capacity exhaustion: sustained traffic above provisioned for more than 300 seconds depletes burst buffer, leading to immediate throttling.
- warningUneven read distribution: scan operations that read entire table instead of querying with keys; or read-heavy queries without pagination.
- warningClient-side retry misconfiguration: no exponential backoff or too aggressive retries causing thundering herd on retry.
Concrete fix directions. Pick the one that matches your root cause.
- buildImplement exponential backoff with jitter in your SDK client (SDK does it by default; verify with maxRetries=3, base=50ms).
- buildRedesign partition key: add a suffix (e.g., user ID mod 10) to distribute writes across more partitions; or use a composite key.
- buildEnable DynamoDB Auto Scaling with conservative settings: target utilization 70%, scale-in cooldown 60s, scale-out cooldown 60s.
- buildIncrease provisioned throughput temporarily: either manual update or pre-warm for known spikes (e.g., flash sales).
- buildFor GSIs: either increase GSI throughput or move to table-level capacity with same partition key; consider using DAX for read-heavy GSIs.
- buildUse DynamoDB Accelerator (DAX) to offload read requests and reduce read capacity consumption for eventual consistency reads.
A fix you cannot prove is a guess. Close the loop.
- verifiedAfter fix, monitor CloudWatch ThrottledRequests metric for the table and any GSIs; expect zero or near-zero.
- verifiedRun load test with same traffic pattern; verify ConsumedWriteCapacityUnits stays below 80% of provisioned for sustained periods.
- verifiedCheck CloudWatch Contributor Insights: throttled partition keys count should drop to zero.
- verifiedVerify auto scaling alarms in CloudWatch: OK state, and no throttle-related alarms firing.
- verifiedIn application logs, grep for ProvisionedThroughputExceededException; count should be 0 after fix.
- verifiedUse AWS CLI describe-table: check ProvisionedThroughput.NumberOfDecreasesToday to ensure you didn't exceed daily decrease quota (4 per day).
Things that make this bug worse or harder to find.
- warningDon't just increase table capacity without checking partition key distribution — you may mask a hot partition and waste money.
- warningDon't disable auto scaling completely; instead adjust target utilization or cooldowns.
- warningDon't ignore GSI capacity: writes to base table consume GSI capacity; check both.
- warningDon't use scan operations in production without pagination and limit — they consume huge read capacity.
- warningDon't set retry strategy with no backoff or fixed interval; this will make throttling worse.
- warningDon't forget to monitor burst capacity depletion; if you rely on burst, plan for sustained traffic.
E-commerce Checkout Throttling During Flash Sale
Timeline
- 09:00Flash sale starts; 10x traffic spike to checkout service.
- 09:02CloudWatch alarm fires: ProvisionedThroughputExceeded on Orders table.
- 09:05PagerDuty alerts team; I begin investigation.
- 09:10Check CloudWatch metrics: ConsumedWriteCapacityUnits = 1500, Provisioned = 1000; ThrottledRequests = 200/min.
- 09:15Run Contributor Insights: top throttled partition key = 'productId#flash-item-123' (the featured item).
- 09:20Identify root cause: partition key was productId alone; all orders for the flash item hit same partition.
- 09:25Emergency fix: increase table write capacity to 5000 (temporary). Also implement exponential backoff in Lambda (but SDK already does).
- 09:30Throttling drops to zero; orders flow again.
- 10:00Permanent fix: redesign partition key to composite (userId + productId) and enable auto scaling.
I was on-call when the flash sale went live. At 09:02 my pager screamed. The checkout Lambda was throwing ProvisionedThroughputExceededException for every order attempt. The first thing I did was open CloudWatch — I saw ConsumedWriteCapacityUnits at 1500 against a provisioned 1000. ThrottledRequests was climbing. I knew we had burst capacity but it only lasts 300 seconds; we were past that.
I checked Contributor Insights and found that a single partition key — the product ID of the flash sale item — was being throttled. That explained everything: our partition key was just productId, so all orders for that one item went to the same partition, limited to ~1000 WCU / (number of partitions). The table had 10 partitions, so that partition only got 100 WCU. We were hammering it with 1500.
I temporarily bumped the table's write capacity to 5000 to spread across partitions (each partition gets 500 now). That stopped throttling immediately. After the sale, I redesigned the partition key to userId#productId so writes distribute across many partitions. I also enabled auto scaling with a target of 70%. No throttling since.
Root cause
Hot partition: single productId as partition key caused all writes for the flash sale item to hit one partition, exceeding its allocated throughput.
The fix
Temporary: increase table write capacity to 5000. Permanent: composite partition key (userId#productId) + auto scaling at 70% target.
The lesson
Always stress-test partition key distribution with expected traffic patterns. Auto scaling alone can't fix a hot partition; key design is fundamental.
DynamoDB stores data across partitions; each partition gets an equal share of the table's provisioned throughput. For example, a table with 1000 WCU and 10 partitions gives each partition 100 WCU. If one partition key receives 200 WCU/s, that partition throttles even if total table consumption is only 200 WCU. This is the most common non-obvious cause.
The number of partitions is determined by table size and throughput; you cannot control it directly. As you increase throughput, DynamoDB adds partitions. But partition count doesn't change instantly — it can take minutes. So a sudden traffic spike on a few keys will throttle before partitions split.
Each table has a burst capacity pool equal to 5 minutes of unused provisioned throughput. When traffic exceeds provisioned, DynamoDB first drains burst capacity. Once burst is exhausted (after ~300 seconds of sustained overuse), every request above provisioned is throttled. This explains why throttling often starts suddenly after a few minutes of high traffic.
Monitoring BurstCapacityBalance in CloudWatch tells you how much burst is left. If it drops to zero, you're about to see throttling. Many engineers mistake burst capacity for unlimited headroom; it's a finite buffer.
DynamoDB Auto Scaling uses Application Auto Scaling with target tracking policies. It works, but it's not instantaneous. Scale-out actions can take 5-15 minutes, and scale-in cooldowns (default 15 minutes) prevent rapid reductions. During fast spikes, auto scaling lags behind, leading to throttling until capacity catches up.
Best practices: Set target utilization to 70% (not 50% or 90%) to leave headroom. Reduce scale-out cooldown to 60 seconds. Use CloudWatch synthetic alarms to pre-warm capacity for known events. Also, consider using on-demand mode for unpredictable workloads — it's more expensive but eliminates throttling from throughput planning.
AWS SDKs (v2 and v3) implement exponential backoff by default with jitter. But if you use a custom HTTP client or older SDK, you might not have it. Without backoff, retries resend requests immediately, worsening congestion. Always verify your SDK configuration: set maxRetries to at least 3, and ensure base delay is around 50ms with exponential factor 2.
Additionally, consider using circuit breaker patterns for higher-level resilience. If throttling persists, don't keep retrying; instead, fail fast and return a 503 to the client. This prevents resource exhaustion in your own application threads.
Frequently asked questions
What is the difference between ProvisionedThroughputExceededException and ThrottlingException?
ProvisionedThroughputExceededException is specific to DynamoDB and indicates you've exceeded the table's provisioned capacity (or a GSI's). ThrottlingException is a more generic HTTP 400 error that can come from other AWS services; in DynamoDB context they are often the same. The DynamoDB API returns ProvisionedThroughputExceededException, while the SDK may wrap it as a ThrottlingException in some languages.
How can I see which partition key is being throttled?
Enable CloudWatch Contributor Insights for DynamoDB. It automatically collects top partition keys by throttled requests. You can also query DynamoDB Streams or application logs if you log the key on each request. For quick triage, use the DynamoDB console's 'Monitor' tab → 'Top Partition Keys by Throttled Requests'.
Should I use DynamoDB on-demand instead of provisioned to avoid throttling?
On-demand mode eliminates throttling due to capacity planning; it scales instantly. However, it costs more (up to 2.5x for sustained workloads). It's ideal for unpredictable traffic, but for steady, predictable workloads, provisioned with auto scaling is more cost-effective. On-demand still has limits per account (default 40K RCU/WCU) but you can request increases.
Why did my write throttle even though total consumed capacity was below provisioned?
That's a classic hot partition symptom. Total capacity is evenly distributed across partitions; one partition can exceed its share while others are underutilized. Check per-partition metrics (available via CloudWatch Contributor Insights or partition-level metrics in older AWS CLI). Redesign your partition key to distribute load more evenly.
How do I increase DynamoDB throughput without downtime?
DynamoDB allows updating provisioned throughput on the fly with no downtime. Use the AWS CLI: aws dynamodb update-table --table-name mytable --provisioned-throughput ReadCapacityUnits=2000,WriteCapacityUnits=2000. The change propagates in a few minutes. However, you are limited to 4 decreases per day per table; increases have no limit.