What this usually means
At its core, this failure means the Ingress controller cannot route incoming requests to the correct backend service. The controller relies on a chain: the Ingress resource must have correct host and path rules, the referenced service must have endpoints (pods must be running and ready), and any TLS certificates must be valid and properly referenced. Non-obvious causes include missing ingress class, misconfigured annotations, network policies blocking traffic, or the controller itself being misconfigured (e.g., wrong --publish-service flag). The problem is rarely the Ingress spec alone—it's usually a disconnected link in the routing chain.
The first ten minutes — establish facts before touching code.
- 1kubectl get ingress -n <namespace> -o wide — check ADDRESS field is populated
- 2kubectl describe ingress <name> -n <namespace> — look for events and backend details
- 3kubectl get svc <serviceName> -n <namespace> — verify service type (ClusterIP) and ports
- 4kubectl get endpoints <serviceName> -n <namespace> — ensure endpoints list pod IPs, not empty
- 5kubectl logs -l app.kubernetes.io/name=ingress-nginx -n ingress-nginx --tail=50 — check controller logs for errors about backend or TLS
- 6curl -v http://<host>/path — observe response headers (X-Ingress-Controller, etc.) and error codes
The specific files, logs, configs, and dashboards that usually own this bug.
- searchIngress resource: kubectl describe ingress <name> -n <namespace>
- searchIngress controller logs: kubectl logs -n ingress-nginx deploy/ingress-nginx-controller
- searchService endpoints: kubectl get endpoints <serviceName> -n <namespace>
- searchPod readiness: kubectl get pods -n <namespace> -o wide | grep <service>
- searchIngress controller configmap: kubectl get cm -n ingress-nginx ingress-nginx-controller -o yaml
- searchTLS secret: kubectl describe secret <tls-name> -n <namespace>
- searchNetwork policy: kubectl get networkpolicy -n <namespace> -o yaml
Practical causes, not theory. These are the things you will actually find.
- warningIngress controller not deployed or not matching ingressClassName
- warningService name or port mismatch between Ingress and actual Service spec
- warningBackend pods not running or failing readiness probes
- warningTLS secret missing or containing invalid certificate/key
- warningMissing or conflicting Ingress annotations (e.g., rewrite-target, ssl-redirect)
- warningNetwork policies blocking ingress traffic to pods
- warningIngress controller missing --publish-service flag causing no external IP
- warningDefault backend 404 if no rules match (host/path not specified)
Concrete fix directions. Pick the one that matches your root cause.
- buildAdd or correct ingressClassName in Ingress spec to match controller (e.g., 'nginx')
- buildUpdate service port in Ingress backend to match Service spec port number
- buildRestart or scale up pods, check readiness probe configuration
- buildRecreate TLS secret with correct base64-encoded certificate and key
- buildRemove conflicting annotations or set correct values for rewrite-target
- buildAdd network policy allowing ingress from controller namespace
- buildSet --publish-service in controller args or annotate ingress with kubernetes.io/ingress.class
A fix you cannot prove is a guess. Close the loop.
- verifiedkubectl get ingress -n <namespace> -o wide shows an IP/hostname in ADDRESS
- verifiedcurl -v http://<host>/path returns 200 and expected response body
- verifiedkubectl describe ingress shows correct backend service and port in Rules section
- verifiedIngress controller logs show no errors related to service or TLS
- verifiedkubectl get endpoints <serviceName> shows at least one pod IP
- verifiedNetwork policy audit: kubectl can exec into a pod and curl the service cluster IP
Things that make this bug worse or harder to find.
- warningAssuming the Ingress controller is the default (Nginx) — check actual deployment
- warningForgetting that Ingress rules are host-based: curl must match the host header
- warningUsing wrong TLS secret name or namespace (secret must be in same namespace as ingress)
- warningSetting ssl-redirect annotation globally when only some routes need HTTPS
- warningNot checking ingress controller logs for 'Configuration won't work' warnings
- warningOverlooking that Ingress resources are cluster-scoped but controller runs in a namespace
The Case of the Silent 404: Ingress Not Routing to Pods
Timeline
- 09:15Deploy new microservice version. Ingress updated to point to new service name.
- 09:18Test endpoint returns 404. curl -v shows nginx 404 page.
- 09:20Check ingress: kubectl get ingress -n prod -o wide shows ADDRESS populated.
- 09:22Describe ingress: shows backend service 'my-svc-v2:8080'.
- 09:25kubectl get svc my-svc-v2 -n prod: shows ClusterIP, port 8080.
- 09:27kubectl get endpoints my-svc-v2 -n prod: returns empty list.
- 09:30kubectl get pods -n prod -l app=my-svc-v2: pods are running but '0/1 Ready'.
- 09:32Describe one pod: Readiness probe fails — HTTP GET /health returns 503.
- 09:35Check deployment: readiness probe path is '/healthz' but code uses '/health'.
- 09:40Fix deployment probe path, pods become Ready. Ingress starts routing traffic.
I had just rolled out a new version of our internal API. The deployment went fine, pods started, but when I hit the endpoint I got a plain nginx 404. My first instinct was 'Ingress rules are wrong.' I checked the ingress address — populated. Described the ingress — backend pointed to 'my-svc-v2:8080', which matched the service. But curl kept returning 404.
I spent 15 minutes double-checking the ingress spec, host headers, and annotations. Nothing. Then I remembered: check endpoints. kubectl get endpoints my-svc-v2 -n prod returned nothing. The service had no endpoints. Pods were listed as running, but '0/1 Ready'. The readiness probe was failing.
I described a pod and saw the probe: HTTP GET /healthz — but the service's health endpoint is /health. The deployment YAML had a typo in the probe path. I corrected it, the pods became ready, endpoints appeared, and traffic flowed. The lesson: never skip checking endpoints. The Ingress chain is only as strong as its weakest link, and that link is often pod readiness.
Root cause
Readiness probe path mismatch in deployment spec caused pods to be marked NotReady, resulting in empty service endpoints and ingress returning 404.
The fix
Updated deployment spec livenessProbe.httpGet.path from '/healthz' to '/health' and applied the change. Pods restarted, readiness passed, endpoints populated.
The lesson
Always verify service endpoints when ingress fails. The Ingress controller routes to endpoints, not pods directly. Empty endpoints mean no routing.
Kubernetes clusters often run multiple ingress controllers (nginx, Traefik, AWS ALB, etc.). The Ingress resource must specify the correct ingressClassName, or the controller must be the default. If you omit ingressClassName and no default is configured, the Ingress is ignored. Check: kubectl get ingress -n <ns> -o jsonpath='{.spec.ingressClassName}'.
The ingress controller logs will show 'Ignoring ingress because of class' if mismatched. Also, some controllers (like nginx) have a --watch-ingress-without-class flag; if not set, they ignore classless ingresses. Always explicitly set ingressClassName.
A common pitfall: the ingress terminates TLS but then forwards HTTP to the backend. If your pod expects HTTPS (e.g., because it has its own cert), you must set the annotation nginx.ingress.kubernetes.io/backend-protocol: HTTPS. Without it, the controller sends plain HTTP, causing connection errors or 502.
Also, TLS secret must be in the same namespace as the Ingress. If the secret is missing or has invalid data, the ingress controller will fail to start listeners for that host. Check: kubectl describe secret -n <ns> shows certificate and key data fields. Use openssl x509 -in <(kubectl get secret -n <ns> <name> -o jsonpath='{.data.tls\.crt}' | base64 -d) -text -noout to validate the cert.
When using path-based routing, the nginx ingress controller's default behavior is to pass the entire URL path to the backend. If your service expects a stripped path (e.g., /api to /), you need the annotation nginx.ingress.kubernetes.io/rewrite-target: /. Missing this causes 404 from the backend because it doesn't recognize the full path.
Also, path type matters: Prefix vs Exact. If you use Exact, only the exact path matches; trailing slashes matter. Verify with kubectl describe ingress and look at the Rules section. Add nginx.ingress.kubernetes.io/use-regex: true if you need regex matching.
Even if Ingress and Service are correct, network policies may block traffic from the ingress controller to your pods. The controller runs in its own namespace (e.g., ingress-nginx). Check if a NetworkPolicy in your app's namespace denies ingress from other namespaces. A quick test: exec into a controller pod and curl the service cluster IP. If it fails, network policy is likely blocking.
Solution: add a NetworkPolicy allowing ingress from the ingress controller namespace with the appropriate pod selector and ports. Also check cloud firewall rules (AWS security groups, GCP firewall) if using external load balancers.
Frequently asked questions
Why does my Ingress show an IP address but curl returns 404?
The IP address means the ingress controller has assigned an external IP (usually from a cloud load balancer). A 404 can mean: no Ingress rules match the requested host/path, the backend service has no endpoints, or the default backend is being hit. Check kubectl describe ingress for events. Also ensure your curl uses the correct Host header matching the ingress spec.
What does 'upstream prematurely closed connection' in ingress controller logs mean?
This usually indicates the backend pod is closing the connection before the response is sent. Common causes: pod crash looping, readiness probe failing (pod not ready), or the backend timeout is shorter than the proxy timeout. Check pod logs and readiness probe configuration. Increase proxy-read-timeout in the ingress configmap if the backend takes a long time to respond.
My ingress works for HTTP but HTTPS gives 502. What's wrong?
Check the TLS secret: it must contain a valid certificate and key. Also ensure the backend protocol is correct. If the backend expects HTTPS, add annotation nginx.ingress.kubernetes.io/backend-protocol: HTTPS. If the backend expects HTTP, but the ingress forces HTTPS redirection (ssl-redirect), the backend might not handle the redirect. Disable ssl-redirect with annotation nginx.ingress.kubernetes.io/ssl-redirect: false for that path.
How do I debug if the ingress controller is not updating its configuration?
Force reload by sending SIGHUP to the controller process: kubectl exec deploy/ingress-nginx-controller -n ingress-nginx -- kill -HUP 1. Or check the controller's configmap for changes. Use kubectl logs -n ingress-nginx -f to watch for 'Configuration changed' messages. If no update, check if the ingress resource has the correct ingressClassName and the controller watches that class.
Can multiple Ingress resources conflict?
Yes. If two Ingresses define the same host and path, the behavior is undefined. The controller may merge rules or only apply one. Check for duplicate host entries across namespaces. Use kubectl get ingress --all-namespaces -o wide | grep <host> to find conflicts. Also, if you have a wildcard host and a specific host, the more specific one should take precedence, but verify in controller logs.