What this usually means
The Kubernetes RBAC system denied a request because the user or service account lacks the necessary permissions. This typically occurs when a Role or ClusterRole is missing, the binding is not attached to the correct subject, or the resource/verb combination is incorrect. It can also happen if the API group is wrong or if the resource does not exist in the expected API version.
The first ten minutes — establish facts before touching code.
- 1Run kubectl auth can-i --list --as=system:serviceaccount:namespace:sa to see effective permissions for the service account
- 2Check the username in the error message from kubectl or audit logs to identify the subject
- 3Use kubectl describe clusterrolebinding <binding> to verify the role binding includes the correct subject
- 4Inspect the Role or ClusterRole with kubectl describe clusterrole <role> to see allowed resources and verbs
- 5Enable audit logs and grep for 'Forbidden' to get the exact resource and verb denied
- 6Check if the resource is in a different API group (e.g., apps/v1 vs batch/v1) by running kubectl api-resources
The specific files, logs, configs, and dashboards that usually own this bug.
- search/var/log/kubernetes/audit.log or kube-apiserver audit log stream
- searchkubectl auth can-i output for the user/service account
- searchkubectl describe clusterrolebinding and rolebinding in the namespace
- searchkubectl get clusterrole -A and kubectl get role -A for existing roles
- searchKubernetes API server flags (--authorization-mode, --authorization-webhook-config-file) if custom authz is used
- searchServiceAccount YAML definition and its associated secrets/tokens
Practical causes, not theory. These are the things you will actually find.
- warningMissing RoleBinding or ClusterRoleBinding for the service account
- warningWrong namespace in the RoleBinding (subject namespace mismatch)
- warningIncorrect API group specified in the role rules (e.g., using core group vs apps group)
- warningTypo in resource name or verb in the Role (e.g., 'pods' vs 'pod')
- warningSubject name mismatch: the username in the error does not match the subject name in the binding
- warningRole is bound but the role itself has no rules or empty rules
Concrete fix directions. Pick the one that matches your root cause.
- buildCreate a RoleBinding or ClusterRoleBinding that binds the correct ClusterRole to the service account
- buildEnsure the RoleBinding's subject namespace matches the service account's namespace
- buildUse the correct API group: check kubectl api-resources for the resource's group
- buildUse kubectl create clusterrole <name> --verb=get,list --resource=pods to generate correct YAML
- buildIf using a custom controller, ensure it uses the service account token mounted at /var/run/secrets/kubernetes.io/serviceaccount
A fix you cannot prove is a guess. Close the loop.
- verifiedRun kubectl auth can-i get pods --as=system:serviceaccount:namespace:sa and expect 'yes'
- verifiedExecute the previously failing kubectl command (e.g., kubectl get pods) with the service account and confirm no error
- verifiedCheck audit logs for 'Allowed' instead of 'Denied' for the same user and action
- verifiedUse kubectl run --image=busybox --restart=Never test-pod -- sh -c 'curl -k https://kubernetes.default/api/v1/namespaces/default/pods' with appropriate token
- verifiedVerify the role binding exists with kubectl get rolebinding -n namespace -o wide
Things that make this bug worse or harder to find.
- warningBinding a ClusterRole with a RoleBinding in the wrong namespace (ClusterRoleBindings are cluster-scoped)
- warningUsing 'pods' when the resource is 'pods' (correct) but misspelling verbs like 'create' as 'creat'
- warningForgetting that some resources are subresources (e.g., pods/log) and require separate rules
- warningAssuming the default service account has permissions (it does not in RBAC-enabled clusters)
- warningNot restarting pods after updating service account permissions (token is cached, but re-read on pod restart)
Jenkins Pipeline Fails with 403 Forbidden When Creating Deployments
Timeline
- 09:15Jenkins job triggers Helm upgrade on a new microservice
- 09:16Helm fails with: Error from server (Forbidden): deployments.apps "my-app" is forbidden: User "system:serviceaccount:jenkins:jenkins-sa" cannot create resource "deployments" in API group "apps" in namespace "default"
- 09:18Engineer checks Jenkins pod's service account: kubectl get pod jenkins-agent -n jenkins -o yaml | grep serviceAccount
- 09:20Runs kubectl auth can-i create deployments --as=system:serviceaccount:jenkins:jenkins-sa -n default -> no
- 09:22Lists existing role bindings: kubectl get rolebinding -n default
- 09:24Finds a ClusterRoleBinding named 'jenkins-admin' that binds cluster-admin to jenkins-sa but in namespace 'jenkins' (subject namespace)
- 09:26Realizes the ClusterRoleBinding subject has namespace 'jenkins' but the binding itself is cluster-scoped; however, the role is cluster-admin which should work? But the error says 'cannot create deployments' in namespace 'default'
- 09:28Discovers that the ClusterRoleBinding actually binds to a different service account name: 'jenkins' vs 'jenkins-sa'. Typo in the binding YAML.
- 09:30Corrects the subject name to 'jenkins-sa' and reapplies the ClusterRoleBinding
- 09:32Re-runs helic upgrade, success
I was called in because a Jenkins pipeline that had been running for months suddenly started failing with a 403 Forbidden error when trying to create a deployment. The error message pointed to the service account 'jenkins-sa' in namespace 'jenkins' not having permission to create deployments in the 'default' namespace.
My first instinct was to check the RBAC bindings. I ran kubectl get clusterrolebinding and found one named 'jenkins-admin'. I described it and saw that it bound the 'cluster-admin' ClusterRole to a service account named 'jenkins' (not 'jenkins-sa') in the 'jenkins' namespace. There was the typo! The binding referenced 'jenkins' instead of 'jenkins-sa'. Someone had copied the name incorrectly when setting up the CI/CD pipeline.
After fixing the subject name to 'jenkins-sa' and reapplying the ClusterRoleBinding, the pipeline succeeded. The lesson: always double-check the exact service account name used by your pods, and use kubectl auth can-i to verify before assuming permissions are correct.
Root cause
Typo in the subject name of a ClusterRoleBinding: 'jenkins' instead of 'jenkins-sa'.
The fix
Corrected the subject name in the ClusterRoleBinding YAML to 'jenkins-sa' and applied with kubectl apply -f clusterrolebinding.yaml.
The lesson
Always verify the subject name and namespace in role bindings. Use kubectl auth can-i --as to test permissions before deployment.
RBAC in Kubernetes consists of four key objects: Role, ClusterRole, RoleBinding, and ClusterRoleBinding. A Role defines permissions within a namespace, while a ClusterRole is cluster-scoped (can also be used in namespaces via RoleBinding). A RoleBinding attaches a Role to a subject (user, group, or service account) within a namespace; ClusterRoleBinding attaches a ClusterRole cluster-wide.
The 403 Forbidden error occurs when the authorization module evaluates the request and finds no rule that allows the action. The request includes the user (or service account), the action (verb), the resource, and optionally the subresource and API group. The system checks all applicable RoleBindings and ClusterRoleBindings. If no rule matches, the request is denied.
The kubectl auth can-i command is the fastest way to test permissions. For a service account, use --as=system:serviceaccount:<namespace>:<name>. For example: kubectl auth can-i create deployments --as=system:serviceaccount:default:mysa -n default. If it returns 'no', you need to check the bindings.
Audit logs provide a definitive record. Enable audit logging in the API server with --audit-log-path=/var/log/kubernetes/audit.log and --audit-policy-file. Then grep for 'Forbidden' to see the exact request. The log entry includes the user, verb, resource, and the decision. Use jq to parse: grep 'Forbidden' /var/log/kubernetes/audit.log | jq '.user.username, .objectRef.resource, .verb'.
Resources belong to API groups. For example, pods are in the core group (empty string), deployments are in apps/v1, and cronjobs are in batch/v1. If your Role rule specifies 'deployments' without an API group, it will not match the deployments in apps/v1. Always check the group with kubectl api-resources | grep deployment.
Subresources like pods/log require a separate rule with the subresource field. The rule must have resources: ['pods/log'] and verbs: ['get', 'list']. Many miss this and get 403 when trying to access logs.
Sometimes the 403 is not due to missing RBAC but because the service account token is invalid or not mounted. Check that the pod has a service account token mounted at /var/run/secrets/kubernetes.io/serviceaccount/token. Use kubectl exec into the pod and try curl with the token: TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token) && curl -k -H "Authorization: Bearer $TOKEN" https://kubernetes.default.svc/api/v1/namespaces/default/pods.
If the token is missing, ensure the pod spec has automountServiceAccountToken: true (default) and that the service account exists. Also check if the service account has an associated secret (older Kubernetes versions) or uses projected tokens (1.24+).
If your cluster uses multiple authorization modes (Node, RBAC, Webhook), the first mode that allows the request short-circuits. But if all modes deny, the error is 403. Check the API server flags: --authorization-mode=Node,RBAC,Webhook. If a webhook returns 'Denied', RBAC rules won't help. Inspect webhook logs to see if it's blocking the request.
Also, some managed Kubernetes services (like GKE) have additional IAM policies that can cause 403s. For example, a GKE cluster with Workload Identity requires IAM bindings on the GCP service account. The error might appear as a 403 from the Kubernetes API but the root cause is missing IAM permissions.
Frequently asked questions
What is the difference between a Role and a ClusterRole?
A Role is namespaced and only grants permissions within that namespace. A ClusterRole is cluster-scoped and can be used to grant permissions across all namespaces, or to cluster-scoped resources (like nodes). When bound with a RoleBinding, a ClusterRole only grants permissions within the binding's namespace.
How do I check if a service account has permissions without actually making a request?
Use kubectl auth can-i <verb> <resource> --as=system:serviceaccount:<namespace>:<name>. For example: kubectl auth can-i get pods --as=system:serviceaccount:default:my-sa. This simulates the authorization check without actually executing the request.
Why am I getting 403 Forbidden even though I have a ClusterRoleBinding with cluster-admin?
Check that the subject (user or service account) in the ClusterRoleBinding matches exactly. Common mistakes: typo in the name, wrong namespace for the service account, or the ClusterRoleBinding is not applied. Also verify that the ClusterRole 'cluster-admin' exists (it does by default).
Can I grant permissions to a specific pod instead of a service account?
No, permissions are granted to service accounts, not directly to pods. You assign a service account to a pod via the serviceAccountName field in the pod spec. The pod's token belongs to that service account, so the pod inherits its permissions.
How do I debug RBAC for a user authenticating via OIDC?
The username will be in the format of the OIDC claim (e.g., user@domain.com). Use kubectl auth can-i --as=<username> to test. Check the OIDC provider's groups claim and ensure ClusterRoleBindings reference the correct group names. Also verify the API server's OIDC configuration (--oidc-username-claim, --oidc-groups-claim).