What this usually means
When a Lambda function is attached to a VPC, AWS creates an Elastic Network Interface (ENI) in your VPC to enable connectivity. This ENI must be created and attached to the execution environment during a cold start. The process involves provisioning an ENI, attaching it to the Lambda sandbox, and assigning a private IP. This can take 5–15 seconds, dominating cold start time. The delay is not from your code but from the AWS control plane orchestrating network plumbing. Warm starts reuse the existing ENI, so latency drops. The problem worsens with functions that are invoked infrequently or have very bursty traffic patterns.
The first ten minutes — establish facts before touching code.
- 1Check Lambda cold start duration in CloudWatch Metrics: `aws cloudwatch get-metric-statistics --namespace AWS/Lambda --metric-name Duration --dimensions Name=FunctionName,Value=YOUR_FUNCTION --statistics Average --period 60`
- 2Enable Lambda Insights and look for `init_duration` > 2000 ms
- 3Examine CloudWatch Logs for `INIT_START` log line and parse the duration from the timestamp difference
- 4Use X-Ray tracing on the Lambda function; look for a long gap before the first subsegment
- 5Check VPC flow logs for ENI creation events (`ACCEPT OK` for ENI traffic) around cold start time
- 6Invoke the function after a 15-minute idle period and measure total execution time using `aws lambda invoke --function-name YOUR_FUNCTION out.txt`
The specific files, logs, configs, and dashboards that usually own this bug.
- searchCloudWatch Logs for Lambda function, specifically the `INIT_START` line and any `Task timed out` messages
- searchAWS Lambda console: Configuration → VPC section to see subnets and security groups
- searchEC2 Console → Network Interfaces: filter by description containing `AWS Lambda VPC ENI`
- searchCloudTrail events for `CreateNetworkInterface` and `AttachNetworkInterface` API calls from Lambda service
- searchVPC Flow Logs for the ENI's traffic patterns (idle vs. active)
- searchAWS X-Ray traces for the function: look for `Initialization` subsegment duration
- searchLambda Insights dashboard: `init_duration` metric and `provisioned_concurrency` usage
Practical causes, not theory. These are the things you will actually find.
- warningFunction is VPC-enabled but doesn't actually need VPC access (e.g., it calls only public APIs)
- warningSubnets with insufficient available IP addresses, causing delays in ENI allocation
- warningSecurity group rules that are too restrictive, causing ENI creation to retry
- warningVPC with no NAT gateway/instance, forcing Lambda to use a public subnet without internet access (still requires ENI)
- warningFunction configured with multiple subnets across AZs, increasing ENI attachment complexity
- warningUsing a shared VPC with limited ENI creation rate limits (e.g., 5 per second per account per region)
Concrete fix directions. Pick the one that matches your root cause.
- buildRemove VPC configuration if function does not need private network access; use IAM roles for AWS service calls
- buildUse AWS Lambda Hyperplane ENI for VPC functions that only need access to private resources (still has cold start but improved)
- buildEnable Provisioned Concurrency to keep execution environments warm and avoid cold starts entirely
- buildUse VPC endpoints (AWS PrivateLink) for services like S3, DynamoDB, etc., to allow functions outside VPC to access them securely
- buildOptimize subnet selection: use subnets with large IP pools and spread across at least two AZs for high availability
- buildConfigure Lambda to reuse ENIs by setting `ReservedConcurrency` to a minimum level to keep environments warm
- buildMove to AWS Lambda SnapStart for Java functions to reduce initialization time
A fix you cannot prove is a guess. Close the loop.
- verifiedAfter removing VPC, invoke function and measure cold start time; should drop to <1 second
- verifiedWith Provisioned Concurrency, invoke function after idle period and confirm `Init duration` is 0 ms in logs
- verifiedCheck CloudWatch Metric `Invocations` and `ConcurrentExecutions` to see if warm pool is maintained
- verifiedReview VPC flow logs: ENI attachment events should reduce significantly
- verifiedRun a load test with gradual ramp-up; cold start times should remain consistent if fix works
- verifiedUse `aws lambda get-function-configuration` to verify VpcConfig is removed or updated
Things that make this bug worse or harder to find.
- warningAdding more subnets without checking IP availability – can make cold start worse
- warningAssuming VPC is required because of security concerns – often functions can use IAM policies and VPC endpoints instead
- warningSetting Provisioned Concurrency to a very high value without cost analysis; it incurs costs even when idle
- warningUsing a single subnet in one AZ – creates a single point of failure and can cause ENI exhaustion
- warningIgnoring ENI deletion delays: when scaling down, Lambda doesn't immediately delete ENIs, leading to IP exhaustion if you reuse subnets
- warningTuning Lambda memory/timeout settings thinking it will reduce cold start – it has no effect on ENI attachment time
The 12-Second Cold Start That Took Down Our API
Timeline
- 09:15PagerDuty alert: API latency P95 > 10s for /users endpoint
- 09:18Check CloudWatch – Lambda Duration shows spikes up to 12,000 ms
- 09:20Look at X-Ray trace: huge gap before any subsegment, suggests initialization issue
- 09:22Examine CloudWatch Logs: 'INIT_START' line shows 8.5s duration
- 09:25Review VPC flow logs: new ENI creation events at same time as cold starts
- 09:28Check function VPC config: 2 private subnets, no NAT – but function only calls DynamoDB via VPC endpoint
- 09:32Cross-reference with CloudTrail: 'CreateNetworkInterface' taking ~7 seconds
- 09:35Enable Provisioned Concurrency for 2 instances
- 09:38Cold start latency drops to <500 ms, P95 normalizes
- 09:40Investigation: function didn't actually need VPC – it used DynamoDB via public endpoint but was mistakenly placed in VPC
The alert came in at 9:15 AM. Our API latency had spiked from 200 ms to over 10 seconds. I immediately checked the Lambda metrics and saw Duration spikes correlating with cold starts. The function was a critical user lookup endpoint that ran infrequently – about once every 20 minutes. The first call after idle would take 12 seconds.
I dove into X-Ray traces and saw a 9-second gap before any subsegment. The CloudWatch logs confirmed `INIT_START` taking 8.5 seconds. That's not code – that's the runtime environment setup. I checked the VPC configuration: two private subnets, no NAT. The function only called DynamoDB, and we had a VPC endpoint for that. Why was it even in a VPC? A previous developer had added VPC for 'security' without understanding the cost.
The fix was two-fold: first, I enabled Provisioned Concurrency for 2 instances to stop the bleeding – latency dropped immediately. Then, I removed the VPC configuration entirely and updated IAM roles. The function now uses DynamoDB via the public endpoint with IAM authentication. Cold start times went from 12 seconds to 300 ms. The root cause was unnecessary VPC attachment causing ENI creation delays.
Root cause
Lambda function was unnecessarily attached to a VPC, causing ENI creation and attachment during every cold start, adding 8–12 seconds of latency.
The fix
Removed VPC configuration and enabled Provisioned Concurrency as immediate mitigation. Long-term: use IAM roles and VPC endpoints for DynamoDB access without VPC.
The lesson
Never attach a Lambda to a VPC unless it absolutely needs private network access. Always measure cold start times and use X-Ray to distinguish initialization from execution latency.
When a Lambda function is configured with a VPC, the Lambda service creates an Elastic Network Interface (ENI) in your VPC to allow the function to access resources within that VPC. This ENI is provisioned by the Lambda control plane, which must allocate an IP address from the specified subnet, create the ENI object, and attach it to the execution environment. This entire process is synchronous for the first invocation after the function is idle or scaled from zero.
The delay is typically 5–15 seconds, depending on subnet IP availability, security group rules, and regional API latency. The ENI is then cached for the lifetime of the execution environment (usually 5–15 minutes of inactivity). After that, the ENI is deleted, and the next cold start repeats the process. This is not a bug; it's inherent to the Lambda VPC architecture.
Use CloudWatch Logs to find the `INIT_START` timestamp. The duration from the start of the invocation to the first log line is the initialization time. Compare this to the actual code execution time. If the initialization time dominates (>2 seconds), ENI attachment is likely the cause. X-Ray traces also show an `Initialization` subsegment if enabled.
Another technique: temporarily remove the VPC configuration and deploy a test function. If cold start time drops dramatically, the VPC is the culprit. Also, check the number of ENIs in your account: `aws ec2 describe-network-interfaces --filters Name=description,Values="*Lambda*"` – a high count indicates many cold starts.
Provisioned Concurrency (PC) keeps a specified number of execution environments warm, eliminating cold starts entirely. However, it incurs costs even when idle. It's a blunt instrument: good for critical functions with predictable traffic, but expensive for spiky workloads.
PC works by pre-creating environments and keeping them alive. This avoids ENI creation because the environment already has an attached ENI. However, if you have a burst of traffic beyond the provisioned count, the new environments will still incur cold starts. PC is not a fix for the root cause; it's a mitigation.
Many teams attach Lambda to a VPC solely for accessing services like DynamoDB, S3, or RDS. AWS offers VPC endpoints (PrivateLink) that allow access to these services without leaving the AWS network, and IAM roles provide fine-grained permissions. This removes the need for VPC attachment entirely.
For RDS, consider using IAM database authentication with a public endpoint over TLS, or use a proxy like RDS Proxy. For on-premises resources, use AWS Direct Connect or VPN, but still avoid attaching Lambda to a VPC if possible. The trade-off is network latency vs. cold start latency; in most cases, avoiding VPC wins.
Frequently asked questions
Can I reduce ENI creation time by using larger subnets?
Larger subnets with many available IPs can reduce the time to allocate an IP, but the bottleneck is usually the ENI creation API call and attachment process, which takes several seconds regardless. Using subnets with plenty of free IPs helps avoid IP exhaustion but won't eliminate the 5-15 second delay.
Does using a NAT gateway speed up cold starts?
No. NAT gateway affects outbound internet access, not ENI creation. The cold start delay is from ENI provisioning, not from network routing. In fact, adding a NAT gateway increases complexity and cost without improving cold start latency.
Will increasing Lambda memory reduce cold start time?
Increasing memory can improve CPU speed for code execution but has no effect on ENI attachment latency. The ENI creation is a control plane operation outside the execution environment. However, more memory can reduce initialization time for language runtimes (e.g., JVM warm-up), but for Node.js or Python, the effect is minimal.
How do I know if my function is suffering from ENI cold start vs. other initialization?
Use X-Ray tracing: if you see a long gap before any subsegment, it's initialization. Check CloudWatch Logs for `INIT_START` and calculate duration. If the function has no VPC, initialization is usually <1 second. If it's VPC-enabled and initialization >3 seconds, ENI is the likely cause.
Is there a way to reuse ENIs across function versions or aliases?
ENIs are tied to a specific execution environment (a combination of function version and resources). They are not shared across versions or aliases. However, using a single function with aliases and Provisioned Concurrency can keep environments warm, avoiding new ENI creation.