ECS Task Definition Error: Why Your Container Won't Start

When ECS launches a task, the ECS agent on the container instance (or the Fargate agent) needs to pull the container image. For ECR repositories, this requires a multi-step authentication process. First, the agent calls `ecr:GetAuthorizationToken` to get a base64-encoded token. It then uses that token to log in to the ECR registry. After login, it calls `ecr:BatchCheckLayerAvailability` and `ecr:BatchGetImage` to fetch the image manifest and layers.

The execution role must have permissions for all these actions. The managed policy `AmazonECSTaskExecutionRolePolicy` includes them all. A custom role that only includes `ecr:BatchGetImage` will fail at the first step—`GetAuthorizationToken`—because the agent has no credentials to authenticate. The error message in `describe-tasks` will be `CannotPullContainerError`, which is generic. You must inspect the ECS agent logs on the instance to see the specific HTTP 401 or 403 response from ECR.

For public images from Docker Hub, no ECR permissions are needed, but the instance must have outbound internet access (via NAT gateway or internet gateway). For Fargate tasks, the VPC must have a NAT gateway or VPC endpoints for ECR API and DKR. Missing network connectivity is another common sub-cause that manifests as the same `CannotPullContainerError`.

ECS task definitions have both a `memory` hard limit (container can't exceed this) and a `memoryReservation` soft limit (minimum). If the hard limit is set too low, the container will be killed with an `OutOfMemoryError` stop code. This often happens with Java applications where the JVM heap alone can exceed 256 MB, but the task definition only allocates 128 MB.

The tricky part is that ECS doesn't validate memory against the image's actual requirements at registration time. The task will start, run for a few seconds, then stop. To debug, look for `OOMKilled` in the stop reason or check CloudWatch logs for abrupt termination. Use `docker inspect` on the image to see its default memory usage, or run locally with `docker stats` to measure peak consumption. Then set `memory` to at least that peak plus a buffer (e.g., 512 MB for a typical Java microservice).

CPU limits work similarly. If you set `cpu` to 256 (0.25 vCPU) but the image is CPU-bound, the container may become unresponsive but not necessarily stop. The task stays RUNNING but health checks fail. Always ensure the CPU allocation is realistic for your workload.

ECS task definitions can reference secrets from AWS Secrets Manager or Parameter Store using the `secrets` array in the container definition. If the execution role lacks `secretsmanager:GetSecretValue` (or `ssm:GetParameter`), the container will fail to start with `ResourceInitializationError`.

I've seen cases where the task definition references a secret by ARN, but the execution role policy is scoped to a specific secret ARN that doesn't match (e.g., a wildcard in the wrong place). The error message will say something like 'cannot retrieve secret' but won't tell you which secret. The fix is to review the IAM policy's resource block and ensure it covers all referenced secrets. Use the `secretsmanager:ResourceTag` condition if you tag secrets with environment names.

Another gotcha: if the secret is in a different region, the task definition must include the full ARN with region. ECS doesn't cross-region resolve secrets. The error will be a timeout trying to reach the secret endpoint.

Fargate tasks use `awsvpc` network mode, meaning each task gets its own elastic network interface (ENI). The task must be able to reach ECR and CloudWatch Logs (if logging is configured). If the VPC subnet has no route to the internet (no NAT gateway or internet gateway), the task will fail to pull images or send logs. The error will be a timeout in the ECS agent logs.

The proper fix is to either (1) create VPC endpoints for `com.amazonaws.region.ecr.api`, `com.amazonaws.region.ecr.dkr`, and `com.amazonaws.region.logs` in the same VPC, or (2) ensure the subnet routes to a NAT gateway. VPC endpoints are preferred for security and cost. Without them, tasks in private subnets cannot pull images.

A common misconfiguration is creating VPC endpoints but forgetting to add the route table association. The endpoint will appear active but the subnet won't use it. Always verify the route table has a route to the endpoint (e.g., for `com.amazonaws.us-east-1.ecr.dkr`).

AWS ECS validates the task definition schema at registration time—it checks that required fields exist and types are correct. However, it does NOT validate that the image URI exists, that referenced secrets exist, or that the execution role has sufficient permissions. This is by design: ECS doesn't make network calls to external services during registration.

This means a task definition can be registered successfully but fail every time it's run. The only way to catch these issues is by actually running a task and examining the stop reason. This is a critical point: always do a test `run-task` after registering a new task definition revision, especially when changing the image, secrets, or roles.

CI/CD pipelines should include a step that runs a short-lived ECS task (with a command that exits immediately) and waits for it to reach STOPPED with a zero exit code. This will catch most definition errors before they hit production.