What this usually means
A sidecar container that fails to start typically indicates a dependency on an init container that never completed, a resource request that cannot be satisfied alongside the main container, or a startup/readiness probe that kills it before it can fully initialize. In many cases, the sidecar's entrypoint script or binary is missing in the image, or the sidecar expects a configuration file that is only mounted after the main container's init completes. The pod scheduler may also reject the combined resource requirements, leading to the sidecar being OOM-killed or evicted silently.
The first ten minutes — establish facts before touching code.
- 1Run 'kubectl describe pod <pod-name>' and inspect Events for Init, BackOff, or OOMKill
- 2Check sidecar logs with 'kubectl logs <pod-name> -c <sidecar-name>' (add --previous if restarted)
- 3Verify init container status: 'kubectl get pod <pod-name> -o jsonpath={.status.initContainerStatuses}'
- 4Compare resource requests/limits: 'kubectl get pod <pod-name> -o yaml | grep -A 10 resources'
- 5Check node resource usage: 'kubectl describe node <node-name>' (look for Allocated resources)
The specific files, logs, configs, and dashboards that usually own this bug.
- search/var/log/pods/<namespace>_<pod-name>_<uid>/<sidecar-name>/0.log on the node
- searchPod YAML: 'kubectl get pod <pod-name> -o yaml' (check containerStatuses, initContainerStatuses)
- searchEvent stream: 'kubectl get events --field-selector involvedObject.name=<pod-name>'
- searchCluster autoscaler logs if node resource issues (in cloud provider's managed control plane)
- searchkubelet logs on the node: 'journalctl -u kubelet -n 100' (look for pod admission failures)
- searchImage pull secrets: 'kubectl get secret <secret-name> -o yaml' if sidecar image is from private registry
Practical causes, not theory. These are the things you will actually find.
- warningInit container for sidecar dependencies (e.g., config generator) crashes
- warningSidecar image missing entrypoint or binary (e.g., typo in image tag)
- warningResource limits too low for sidecar to start alongside main container
- warningReadiness/liveness probe configured with too-short initialDelaySeconds
- warningVolume mount conflict: sidecar expects a volume that is only created by main container
- warningNode resource pressure causing sidecar to be evicted before it can start
- warningService account token mount fails due to invalid automountServiceAccountToken setting
Concrete fix directions. Pick the one that matches your root cause.
- buildAdd an init container that pre-checks sidecar dependencies and exits 0
- buildAdjust resource requests/limits: increase sidecar memory request or set resource quotas per namespace
- buildSet 'initialDelaySeconds' on probes to allow sidecar time to initialize (at least 10-30 seconds)
- buildSwitch sidecar to a separate deployment if resource contention is persistent
- buildUse shared volume mounts with subPath to avoid conflicts
- buildEnsure sidecar image uses a valid entrypoint (e.g., 'CMD' vs 'ENTRYPOINT' mismatch)
A fix you cannot prove is a guess. Close the loop.
- verifiedMonitor pod status: 'kubectl get pod -w' and ensure sidecar transitions to Running/Ready
- verifiedCheck sidecar logs for successful startup message: 'kubectl logs <pod-name> -c <sidecar-name>'
- verifiedVerify readiness probe passes: 'kubectl describe pod <pod-name>' shows Ready condition true
- verifiedSimulate node pressure by running 'stress' and ensure sidecar is not OOM-killed
- verifiedRun e2e test: deploy a test pod with same sidecar config and verify it starts
- verifiedCheck kubelet events for no more OOMKill or BackOff events for the sidecar
Things that make this bug worse or harder to find.
- warningDon't assume the sidecar image is valid; always pull and run it locally first
- warningDon't set readiness probes with initialDelaySeconds of 0 for sidecars that need time
- warningDon't use 'restartPolicy: Never' on pods with sidecars; use 'OnFailure' at least
- warningDon't forget to update resource quotas when adding a sidecar to existing namespaces
- warningDon't ignore init container failures because they are not visible in standard pod status
- warningDon't use the same service account for sidecar and main container if sidecar needs different permissions
Logging sidecar stuck in CrashLoopBackOff after config init failure
Timeline
- 09:15Alert: Pod 'api-v2-7f4b9c' in CrashLoopBackOff for sidecar 'fluentd'
- 09:18kubectl describe pod shows init container 'config-init' completed successfully, but fluentd container OOMKilled
- 09:22Checked fluentd logs (kubectl logs -c fluentd --previous): 'cannot load config file /etc/fluentd/fluent.conf'
- 09:25Inspected init container logs: config-init placed config in /tmp/fluentd, but sidecar expects it in /etc/fluentd
- 09:30Volume mount mismatch: init container mounted config volume at /tmp, sidecar at /etc
- 09:35Fixed by aligning mount paths to /etc/fluentd for both containers
- 09:40Pod restarted; sidecar now Running and Ready. No further alerts.
The alert came in around 9:15 AM: the new api-v2 pod was in CrashLoopBackOff. I saw the main container (a Go API) was running fine, but the Fluentd sidecar kept dying. kubectl describe showed an init container called 'config-init' had completed, but the sidecar was OOMKilled. That was odd because Fluentd usually uses little memory.
I checked the sidecar logs with --previous and saw 'cannot load config file /etc/fluentd/fluent.conf'. The init container logs showed it had generated the config and placed it in /tmp/fluentd/fluent.conf. Aha—the init container mounted the config volume at /tmp, but the sidecar mounted it at /etc/fluentd. The file wasn't there, so Fluentd crashed, and Kubernetes kept restarting it until OOM limits kicked in.
I fixed the volume mount path in the deployment YAML so both containers used /etc/fluentd. After redeploying, the pod started cleanly. The lesson: always double-check that shared volumes are mounted at consistent paths across all containers in the pod.
Root cause
Mismatched volume mount paths between init container and sidecar container caused missing configuration file.
The fix
Updated the Deployment YAML to mount the config volume at /etc/fluentd for both init container and sidecar.
The lesson
When sharing volumes between init containers and sidecars, ensure mount paths are identical. Use environment variables or ConfigMaps to enforce consistency.
Init containers run sequentially and must complete before any sidecar starts. If an init container crashes or produces an incorrect output, the sidecar will never start. This is a common hidden failure: the pod status shows 'Init:CrashLoopBackOff' for the init container, but engineers look at the sidecar logs first.
Check init container status explicitly: 'kubectl get pod <pod> -o jsonpath="{.status.initContainerStatuses[*].state}"'. If init containers show 'terminated' with non-zero exit code, inspect their logs. A typical fix is to add retries or health checks inside the init container script.
Sidecars often share the same CPU/memory pool as the main container. If the pod's total resource requests exceed the node's allocatable resources, the scheduler will not place the pod, or the sidecar may be OOM-killed if it spikes during startup. The symptom: sidecar shows OOMKilled in container status, but main container never experiences issues.
Use 'kubectl describe node' to see actual resource usage. Increase the sidecar's memory request/limit or lower the main container's limits. Alternatively, set resource quotas per namespace to prevent overcommit. Another fix: use a separate deployment for the sidecar if resource isolation is critical.
Readiness and liveness probes are the top cause of sidecar failures. If initialDelaySeconds is too short, the probe kills the sidecar before it can start. For sidecars that need to download plugins or connect to external services, set initialDelaySeconds to at least 30 seconds and failureThreshold to 3 or more.
Test probes by temporarily disabling them and observing if the sidecar starts. Use 'kubectl exec' to manually check the probe endpoint. For HTTP probes, verify the endpoint returns 200: 'kubectl exec <pod> -c <sidecar> -- curl -f http://localhost:8080/health'.
Sidecar images are often based on minimal distroless images, which may lack a shell or common binaries. If the sidecar's entrypoint script uses '#!/bin/bash' but the image only has 'sh', the container will fail silently. Always test the image locally: 'docker run --rm <image> <command>'.
Another mistake: using the same image for main container and sidecar but with different commands. Ensure the sidecar's command is executable and does not rely on environment variables that are only set in the main container's context.
Sidecars often share volumes with the main container (e.g., log directories). If the main container creates the directory with restrictive permissions, the sidecar may not be able to read/write. This manifests as 'Permission denied' errors in sidecar logs.
Solution: use a shared volume with a subPath that both containers can access, or run the sidecar as the same user as the main container (set securityContext.fsGroup). Also, avoid mounting the same volume in different modes (e.g., read-write vs read-only) unless explicitly supported.
Frequently asked questions
How is a sidecar different from an init container?
An init container runs to completion before the main containers start, while a sidecar runs alongside the main container for the entire pod lifetime. Sidecars are used for auxiliary services like logging, monitoring, or proxying.
Can a sidecar be in CrashLoopBackOff while the main container is healthy?
Yes. Since containers in a pod are independent processes, one can fail while others run. The pod overall will be considered not Ready if a sidecar fails its readiness probe.
How do I debug a sidecar that fails during startup?
First, check the sidecar logs with 'kubectl logs <pod> -c <sidecar>'. If the container has restarted, add '--previous' to see logs from the terminated instance. Also, run 'kubectl describe pod' and look for events and container state details.
What resource limits should I set for a sidecar?
Start with requests equal to the sidecar's baseline memory usage (e.g., 50Mi for Fluentd) and limits at 2x that. Monitor with 'kubectl top pod' and adjust. Avoid setting limits too low, as sidecars often need extra memory during startup.
Can I use a sidecar as an init container?
No, they serve different purposes. An init container runs once and exits; a sidecar runs continuously. If you need setup before the sidecar starts, use an init container to prepare resources (e.g., generate config files) and then have the sidecar use those resources.