What this usually means
The AWS IAM authorization engine evaluates multiple policy types in a deny-by-default model. A permission denied error means that during evaluation, either there was no explicit allow statement that matched the request, or there was an explicit deny from an identity-based policy, resource-based policy, service control policy (SCP), permissions boundary, or session policy. The tricky part is that the error message rarely tells you which policy caused the deny. You must systematically check each layer: identity policies, resource policies, SCPs, permissions boundaries, and session policies. Additionally, conditions like IP address, time, or tags can cause a deny even when the action is allowed.
The first ten minutes — establish facts before touching code.
- 1Run `aws sts decode-authorization-message --encoded-message <encoded>` with the encoded message from the error to get a human-readable JSON explaining the denied result.
- 2Use the IAM Policy Simulator in the AWS console or CLI with the exact principal, action, resource, and context keys to simulate the request.
- 3Check CloudTrail logs for the event: `aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=* --query 'Events[?ErrorCode==`AccessDenied`]'`.
- 4Review the principal's attached policies: `aws iam list-attached-user-policies --user-name <user>` and `aws iam list-user-policies --user-name <user>`, plus group policies.
- 5Check for any SCPs affecting the account: `aws organizations list-policies-for-target --target-id <account-id> --filter SERVICE_CONTROL_POLICY`.
- 6If using roles, check the trust policy and any permissions boundary: `aws iam get-role --role-name <role>` and look at `PermissionsBoundary`.
The specific files, logs, configs, and dashboards that usually own this bug.
- searchCloudTrail event history for the specific error event (look for `errorCode: AccessDenied`)
- searchIAM user/role policy details: `aws iam get-user-policy --user-name <user> --policy-name <policy>` or `aws iam get-role-policy`
- searchAWS Organizations SCPs for the account: `aws organizations list-policies-for-target`
- searchResource-based policies (e.g., S3 bucket policy, KMS key policy): check the resource console or CLI
- searchIAM Policy Simulator output (console or CLI via `iam simulate-custom-policy`)
- searchAWS Config advanced queries for IAM policies (if enabled)
- searchAWS Support Center case logs (if you opened a case)
Practical causes, not theory. These are the things you will actually find.
- warningMissing explicit Allow for the specific action/resource in identity-based policy
- warningImplicit Deny from a Service Control Policy (SCP) that applies at the OU or account level
- warningExplicit Deny in a resource-based policy (e.g., S3 bucket policy Denies a principal)
- warningPermissions boundary attached to the role that does not include the action
- warningSession policy (e.g., from STS AssumeRole) that limits permissions further
- warningCondition key mismatch: the request does not satisfy a condition (e.g., IP address range, MFA, tags)
- warningCross-account access: trust policy allows AssumeRole but resource-based policy does not grant access to the assumed role
Concrete fix directions. Pick the one that matches your root cause.
- buildAdd an explicit Allow statement for the missing action/resource to the appropriate identity-based policy.
- buildModify the SCP to allow the action if it's not critical to restrict, or create an exception using a condition.
- buildUpdate the resource-based policy to grant the principal (or role) the necessary actions (e.g., S3:GetObject).
- buildRemove or update the permissions boundary to include the required actions.
- buildAdjust the trust policy or session policy (if using temporary credentials) to include the needed permissions.
- buildRevise conditions in policies to match the request context (e.g., update IP range, remove MFA requirement).
A fix you cannot prove is a guess. Close the loop.
- verifiedRe-run the original command and confirm no AccessDenied error.
- verifiedUse the IAM Policy Simulator with the same principal, action, resource, and context – expect 'allowed'.
- verifiedCheck CloudTrail for a successful event (no errorCode) after the fix.
- verifiedReview the effective permissions via `aws iam simulate-principal-policy` for the principal.
- verifiedIf cross-account, test the assume role and then the resource access end-to-end.
Things that make this bug worse or harder to find.
- warningOnly checking the user's inline policies and ignoring group or role policies.
- warningForgetting to check SCPs when in an AWS Organization – SCPs affect all principals.
- warningAssuming that a Deny in the error message is from an explicit deny – it could be an implicit deny (no Allow).
- warningNot simulating conditions exactly – use the same source IP, MFA status, etc.
- warningOverlooking resource-based policies that are separate from the principal's identity policies.
- warningFailing to consider that the request might be using temporary credentials with a session policy.
EC2 Cannot Describe Instances After Role Switch
Timeline
- 14:00PagerDuty alert: prod monitoring script fails with 'AccessDenied' when calling ec2:DescribeInstances
- 14:03Engineer logs into AWS console as admin, runs `aws ec2 describe-instances` from the monitoring server – gets AccessDenied
- 14:08Checks the instance's IAM role: 'monitoring-role' has policy 'EC2ReadOnly' which allows DescribeInstances
- 14:12Runs `aws iam simulate-principal-policy --policy-source-arn arn:aws:iam::123456789012:role/monitoring-role --action-names ec2:DescribeInstances --resource-arns '*'` – result: allowed
- 14:18Checks CloudTrail for the AccessDenied event – sees the caller is 'arn:aws:sts::123456789012:assumed-role/monitoring-role/session' but the error message includes 'source identity' condition
- 14:22Decodes the authorization message: `aws sts decode-authorization-message --encoded-message <encoded>` – shows explicit deny from a service control policy
- 14:25Checks AWS Organizations SCPs – finds an SCP attached to the OU that denies ec2:DescribeInstances if the request does not come from a specific VPC endpoint
- 14:30The monitoring server is not using the VPC endpoint – fixes by modifying the SCP to add an exception for the monitoring role or by routing traffic through the VPC endpoint
We got paged because our production monitoring script suddenly couldn't list EC2 instances. The error was just 'AccessDenied' with no further detail. I assumed it was a policy issue on the IAM role, but the role clearly had EC2ReadOnly policy attached. The policy simulator said it should work. That's when I realized the problem might be elsewhere.
I decoded the authorization message from the CloudTrail event and it pointed to a Service Control Policy. We had recently reorganized our AWS Organizations and applied a restrictive SCP to the prod OU that required all EC2 API calls to come from a VPC endpoint. The monitoring server was using the public endpoint.
We updated the SCP to include an exception for the monitoring role's ARN. After the change, the script ran successfully. The root cause was an SCP we didn't realize was affecting the OU. Now we always check SCPs when debugging permission issues in an organization.
Root cause
A Service Control Policy attached to the production OU denied ec2:DescribeInstances unless the request originated from a VPC endpoint. The monitoring role's calls came from the public internet, triggering the deny.
The fix
Modified the SCP to add a condition exception that allowed the specific monitoring role ARN to call ec2:DescribeInstances from any endpoint.
The lesson
Always include SCPs in the debugging checklist when working in an AWS Organization. The policy simulator does not automatically include SCP effects – you must simulate with the SCP context.
The IAM Policy Simulator is more powerful than just checking policies manually. You can simulate with the exact principal, action, resource, and context keys. Use the AWS CLI: `aws iam simulate-principal-policy --policy-source-arn <arn> --action-names <action> --resource-arns <arn> --context-entries <context>`. The context entries are crucial – include source IP, MFA status, and any tags.
One common mistake: the simulator does not automatically include SCPs or permissions boundaries unless you explicitly add them via the `--policy-input-list` parameter. To simulate SCPs, you need to create a file with the SCP policy JSON and pass it as an additional policy. Always simulate with all applicable policies to get an accurate result.
Cross-account access involves two separate policies: the trust policy on the role (allows the external account to assume the role) and the resource-based policy on the target resource (e.g., S3 bucket policy) that grants the assumed role access. A common cause of 'AccessDenied' is that the trust policy allows the assume, but the resource-based policy denies or does not allow the action.
When debugging cross-account, check both: `aws iam get-role --role-name <role>` for trust policy, and the resource policy (e.g., `aws s3api get-bucket-policy --bucket <bucket>`). Also, ensure the resource policy uses the correct principal ARN – often the role's ARN, not the user's.
When you assume a role, you can optionally pass a session policy (inline or managed) that further restricts permissions. The effective permissions are the intersection of the role's identity policy and the session policy. Similarly, a permissions boundary sets the maximum permissions for the role. If the boundary does not include an action, it's denied even if the identity policy allows it.
To check for a permissions boundary, look at the role's `PermissionsBoundary` field. For session policies, check the `--policy-arns` or `--policy-document` used during `sts:AssumeRole`. Use `aws sts get-caller-identity` to see the effective principal ARN and check if it includes a session policy.
CloudTrail logs every API call. For permission denied errors, filter by `errorCode: AccessDenied`. The event record includes the `userIdentity`, `requestParameters`, and `errorMessage`. The `encodedAuthorizationMessage` field is present for some services – decode it as described earlier.
AWS Config can track IAM policy changes. If you have Config enabled, you can query historical policy changes to see if a policy was modified right before the error started. Use the Config advanced query: `SELECT * WHERE resourceType = 'AWS::IAM::Policy' AND configuration.policyDocument LIKE '%ec2:DescribeInstances%'`. This helps identify when a policy was added or changed.
Frequently asked questions
Why does the IAM Policy Simulator show 'allowed' but I still get AccessDenied?
The simulator might not include all policies that apply to the request. Common missing policies: SCPs, permissions boundaries, session policies, and resource-based policies. Ensure you add these as additional policy inputs in the simulation. Also, conditions like source IP or MFA must match exactly – use the context entries in the simulation.
How do I find which policy caused the deny?
Use the `decode-authorization-message` API with the encoded message from the error. It returns a JSON with the 'matched entries' showing which policy and statement caused the deny. If the encoded message is not available in the error, check CloudTrail for the event and get the encoded message from there.
Can an SCP cause an AccessDenied even if the IAM policy allows it?
Yes. SCPs apply at the account or OU level and can deny actions even if the IAM policy allows them. The SCP is evaluated after identity policies but before resource-based policies. An explicit deny in an SCP overrides any allow. Always check SCPs when debugging in an AWS Organization.
What is a permissions boundary and how does it affect access?
A permissions boundary is a managed policy that sets the maximum permissions for an IAM role or user. The effective permissions are the intersection of the identity policy and the boundary. If an action is not allowed by the boundary, it will be denied even if the identity policy allows it. Check the role or user's 'PermissionsBoundary' attribute.
Why does cross-account access fail even though the trust policy allows it?
The trust policy allows the external account to assume the role, but the resource-based policy (e.g., S3 bucket policy) on the target resource must also grant the assumed role access. Additionally, the role's identity policy must allow the action. Check all three: trust policy, role policy, and resource policy.