What this usually means
Step Functions execution failures typically fall into one of three categories: IAM permissions misconfiguration, state machine definition errors (invalid JSON paths, missing fields), or service integration failures (Lambda timeouts, API throttling). The Step Functions execution event history is the most reliable source of truth—it records every state transition and any errors encountered. Many developers ignore the 'Causes' field in the event details, which contains the actual error message from the failed service. Non-obvious causes include payload size limits (262 KB for input/output), Lambda function concurrency limits, and VPC networking issues when using Lambda in a VPC.
The first ten minutes — establish facts before touching code.
- 1Open the Step Functions console, select the failed execution, and click the 'Execution event history' tab. Look for events with type 'TaskFailed', 'ExecutionFailed', or 'ChoiceStateEntered' with a 'cause' field.
- 2Run `aws stepfunctions describe-execution --execution-arn <arn>` to get the execution status and error details. Parse the 'cause' field for the actual error message.
- 3Check the CloudWatch Logs group for the state machine (usually /aws/stepfunctions/<state-machine-name>). Search for the execution ID to find correlated logs.
- 4If a Lambda task failed, check the Lambda function's CloudWatch Logs for the specific invocation. Look for 'Task timed out after X seconds' or 'ResourceNotFoundException'.
- 5Validate the state machine definition using `aws stepfunctions validate-state-machine-definition --definition file://definition.json`. This catches syntax errors and invalid JSON paths.
- 6Check IAM roles: ensure the state machine's execution role has permissions for all integrated services. Use IAM Access Analyzer to validate the policy.
The specific files, logs, configs, and dashboards that usually own this bug.
- searchStep Functions Console → Execution Details → Execution Event History (the 'cause' field in each event)
- searchCloudWatch Log Group: /aws/stepfunctions/<state-machine-name>
- searchCloudWatch Logs for each Lambda function integrated with the state machine
- searchAWS CloudTrail events for the state machine's IAM role to see denied API calls
- searchIAM Role: <state-machine-name>-execution-role (check trust policy and attached policies)
- searchState machine definition JSON (especially 'Resource' ARNs, 'Parameters', and 'ResultPath')
- searchLambda function configuration: timeout, memory, VPC settings, reserved concurrency
Practical causes, not theory. These are the things you will actually find.
- warningIAM execution role missing permissions for a downstream service (e.g., DynamoDB PutItem, SQS SendMessage)
- warningLambda function timeout shorter than the state machine task timeout, causing 'TaskTimedOut'
- warningState machine input or output exceeds 262144 bytes (256 KB) after state transformation
- warningInvalid JSONPath in 'InputPath', 'ResultPath', or 'OutputPath' causing the state to fail silently
- warningLambda function in a VPC without a VPC endpoint for Step Functions (or missing NAT gateway)
- warningService API throttling (e.g., DynamoDB write capacity exceeded) causing retries to fail
- warningChoice state conditions with conflicting rules or missing default state causing unexpected transitions
Concrete fix directions. Pick the one that matches your root cause.
- buildAdd the missing IAM permissions to the execution role. Use AWS managed policies or create a custom policy with minimal required actions.
- buildIncrease Lambda timeout or reduce the state machine task timeout to match. Ensure 'HeartbeatSeconds' and 'TimeoutSeconds' are set appropriately.
- buildReduce payload size by trimming input/output or using 'ResultPath' to exclude unnecessary fields. Consider using S3 for large payloads.
- buildFix JSONPath expressions by testing them with a local JSON parser. Use '$$.Execution.Input' for context object access.
- buildConfigure Lambda VPC settings: either remove VPC, add VPC endpoints for Step Functions and other services, or use a NAT gateway.
- buildImplement retry logic with exponential backoff in the state machine definition for throttling errors (e.g., 'States.ALL' with interval and max attempts).
- buildReorganize Choice state rules: ensure conditions are mutually exclusive or order them correctly, and always include a 'Default' state.
A fix you cannot prove is a guess. Close the loop.
- verifiedRe-run the execution using the same input and confirm the status changes to 'Succeeded'
- verifiedCheck the execution event history for any 'TaskFailed' or 'ExecutionFailed' events after the fix
- verifiedMonitor CloudWatch metrics for the state machine: 'ExecutionThrottled', 'ExecutionsFailed', 'ExecutionsTimedOut' should drop to zero
- verifiedTest edge cases: large input, missing fields in JSON, or high concurrency to ensure the fix holds
- verifiedUse `aws stepfunctions start-execution` with the same input and verify the final output matches expectations
Things that make this bug worse or harder to find.
- warningDo not ignore the 'cause' field in execution events—it contains the actual error message from the failed service
- warningDo not modify the state machine definition without testing the changes via a dry run or in a non-production environment
- warningDo not assume Lambda logs are always accessible; if Lambda is in a VPC, ensure CloudWatch Logs VPC endpoint or NAT is configured
- warningDo not set retry intervals too short—this can cause immediate retries that still fail and waste execution time
- warningDo not forget to check for 'ResultPath' conflicts: if you set 'ResultPath' to a path that already exists in the state output, it will overwrite the data
- warningDo not rely solely on the console UI; use the AWS CLI for detailed error messages and automation
The Silent Timeout: A Lambda Task Fails with No Logs
Timeline
- 09:15Deploy new state machine 'OrderProcessor' to production
- 09:22First customer order fails; step function execution shows 'Failed' status
- 09:25Check execution event history: 'TaskFailed' on 'ValidateOrder' Lambda, cause: 'Task timed out after 30 seconds'
- 09:28Check Lambda CloudWatch Logs: no log entries for that execution ID
- 09:35Notice Lambda function is configured with a 5-second timeout, but state machine task timeout is 30 seconds
- 09:40Realize Lambda is in a VPC without a NAT gateway; it can't reach DynamoDB, causing the function to hang until its own timeout
- 09:45Add DynamoDB VPC endpoint to the VPC and increase Lambda timeout to 30 seconds
- 09:50Re-run the execution with same input; it succeeds
The morning started smoothly until the first customer order hit production. The Step Functions execution failed immediately. I opened the console and saw the red 'Failed' status. The execution event history showed 'TaskFailed' on the 'ValidateOrder' Lambda with cause 'Task timed out after 30 seconds'. That was odd—the Lambda function had a 5-second timeout, but the state machine was waiting 30 seconds. The mismatch should have caused the Lambda to fail after 5 seconds, not 30. Something else was going on.
I clicked into the Lambda CloudWatch Logs group—nothing. No log entries for that execution ID. That was the red flag. The Lambda function wasn't even starting. I checked the function configuration: VPC enabled, no subnet route to the internet. The function needed to call DynamoDB to validate the order, but DynamoDB is a public endpoint. Without a NAT gateway or VPC endpoint, the Lambda function couldn't reach DynamoDB. The SDK client hung trying to connect until the Lambda's own timeout (5 seconds) kicked in, but the state machine saw a 'TaskTimedOut' after its configured 30 seconds because it didn't receive a response before its timeout.
The fix was two-fold: add a DynamoDB VPC endpoint to the Lambda's VPC so it could reach DynamoDB privately, and increase the Lambda timeout to match the state machine's task timeout (30 seconds) to avoid confusion. After deploying the changes and re-running the execution with the same input, it succeeded. The lesson: when a Lambda task fails with a timeout but has no logs, it's almost always a networking issue—VPC without endpoints or NAT. Also, align timeouts between Lambda and Step Functions to avoid misleading error messages.
Root cause
Lambda function in a VPC with no DynamoDB VPC endpoint or NAT gateway caused the SDK client to hang, leading to a timeout that was misreported by Step Functions.
The fix
Added a DynamoDB VPC endpoint to the Lambda's VPC and increased Lambda timeout from 5s to 30s to match the state machine task timeout.
The lesson
Always test Lambda functions in the same VPC configuration as production. When a Lambda task times out with no logs, suspect networking issues first. Align timeouts between Step Functions and Lambda to avoid cascading failures.
The execution event history is a chronological list of every state transition and action taken during an execution. Each event has a 'type' (e.g., 'TaskStateEntered', 'TaskScheduled', 'TaskFailed') and a 'cause' field that contains the error message from the failed service. The most common mistake is only looking at the final 'ExecutionFailed' event, which often says 'An error occurred while executing the state' without details. Instead, scroll up to the 'TaskFailed' or 'ChoiceStateEntered' event immediately before the failure—that's where the real error lives.
Use the CLI to get a machine-readable version: `aws stepfunctions get-execution-history --execution-arn <arn> --query 'events[?type==`TaskFailed`].{cause: cause, timestamp: timestamp}'`. This filters only failure events and their causes. For choice state failures, look for 'ChoiceStateEntered' events with a 'cause' that says 'No choice matches the input' or similar.
Step Functions assumes an IAM role to execute tasks. If that role doesn't have permissions to invoke a Lambda function, write to DynamoDB, or send to SQS, the execution fails with an access denied error. However, the error might be buried. For example, a 'TaskFailed' event might have cause: 'Lambda.Unknown' when the role can't invoke the function. The fix is to add the 'lambda:InvokeFunction' permission (or 'states:InvokeFunction' for Step Functions).
Use IAM Access Analyzer to validate the execution role's policy. Generate a policy based on CloudTrail access to see what actions are being denied. The command `aws iam simulate-principal-policy --policy-source-arn <role-arn> --action-names lambda:InvokeFunction` can test specific permissions. For multi-service workflows, ensure the trust policy allows Step Functions to assume the role (Service: states.amazonaws.com).
Step Functions has a 262144 byte (256 KB) limit for the combined input and output of each state. If a state's output exceeds this, the execution fails with 'States.DataLimitExceeded' or 'OutputTooLarge'. This often happens when a Lambda function returns a large dataset. The fix is to trim the payload using 'ResultPath' to select only necessary fields, or use S3 for large data (e.g., store large results in S3 and pass the bucket/key in the state output).
Another common issue is 'ResultPath' causing data loss. If you set 'ResultPath': '$.myResult' but the state's input already has a 'myResult' field, it will be overwritten. Always initialize output paths carefully. Use 'ResultPath': null to discard the result if not needed, or use 'ResultSelector' to reshape the output before assigning.
When a Lambda function is attached to a VPC, it loses internet access by default. If the function needs to call AWS services (like DynamoDB, SQS) that are not in the same VPC, it will hang until the Lambda timeout. The state machine may report a different timeout because it has its own 'TimeoutSeconds' setting. Always ensure Lambda has a route to the required services: either use VPC endpoints for each service, or place the Lambda in a private subnet with a NAT gateway.
Align timeouts: set the Lambda function timeout to be slightly less than the state machine's 'TimeoutSeconds' for that task. For example, if the state machine task has 'TimeoutSeconds': 30, set Lambda timeout to 25 seconds. This ensures that if Lambda times out, it returns a 'TaskTimedOut' error to Step Functions before the state machine's own timeout, giving a clearer error.
Choice states evaluate conditions using JSONPath expressions. A common failure is when the input doesn't contain the expected field, causing the choice to fall to the default state or fail. For example, `Variable: '$.status'` when 'status' is missing from the input will cause an error unless a default state is defined. Always include a 'Default' state that handles unexpected inputs.
Another issue is ordering of rules: Choice rules are evaluated in order, and the first match wins. If you have overlapping conditions (e.g., '$.value >= 10' and '$.value > 5'), the more specific rule must come first. Use the 'And' operator to combine conditions. Validate JSONPath expressions using `aws stepfunctions validate-state-machine-definition` or a JSONPath tester.
Frequently asked questions
How do I get the exact error message from a failed Step Functions execution?
Use the CLI: `aws stepfunctions describe-execution --execution-arn <arn> --query 'status, error, cause'`. The 'cause' field contains the detailed error message. For more granularity, get the execution history and filter for 'TaskFailed' events: `aws stepfunctions get-execution-history --execution-arn <arn> --query 'events[?type==`TaskFailed`].{cause: cause, timestamp: timestamp}'`.
Why does my Lambda task fail with 'TaskTimedOut' even though the Lambda function returns quickly?
This usually means the Lambda function is not responding within the state machine's 'TimeoutSeconds'. Check if the Lambda is in a VPC without internet access—it may hang while trying to reach an external endpoint. Also, ensure the Lambda function timeout is set to a value less than or equal to the state machine task timeout. If the Lambda times out before returning, Step Functions will see a 'TaskTimedOut'.
Can I see the output of each state for debugging?
Yes, in the execution event history, look for events of type 'TaskStateExited' or 'PassStateExited'. These contain the 'output' field which is the state's output after processing. However, if the output is too large (>256 KB), it will be truncated. You can also use CloudWatch Logs for the state machine to see all state inputs and outputs if logging is enabled.
How do I handle large payloads in Step Functions?
Step Functions has a 256 KB limit for input/output. For larger payloads, store the data in S3 and pass only the S3 key between states. Use the 's3:PutObject' and 's3:GetObject' actions in your state machine definition. Alternatively, use 'ResultPath' to trim the payload by selecting only necessary fields.
What is the difference between 'States.ALL' and 'States.Timeout' in retry policies?
'States.ALL' catches all errors, including timeouts, while 'States.Timeout' catches only timeout errors. If you want to retry on timeouts but not on other errors (like access denied), use 'States.Timeout'. For a general retry policy, use 'States.ALL' with a limited number of retries. Always define retry intervals with exponential backoff to avoid overwhelming downstream services.