Terraform State Drift: Resource Changed Outside Terraform

Terraform maintains a state file that maps your configuration to real-world resources. When a resource is modified outside of Terraform (via console, CLI, or API), the state file becomes stale—it no longer reflects the actual resource attributes. On the next 'terraform plan', Terraform compares its state (the old values) against the configuration (your desired values) and sees a diff: it thinks the resource needs to be updated to match config, but actually the resource has already been changed externally. This is drift.

Drift can be intentional (e.g., emergency fix) or accidental (e.g., auto-scaling group modifies launch config). The critical point is that Terraform's default behavior is to overwrite external changes, which can cause service disruption if not handled carefully. Understanding the refresh phase is key: when you run 'terraform plan', Terraform optionally refreshes state (unless -refresh=false) by querying the provider API. This refresh updates the state to match reality, but only for the duration of that command—it doesn't persist unless you explicitly run 'terraform apply -refresh-only'.

The first step is to isolate which resources have drifted. Use 'terraform plan -out=tfplan' and then 'terraform show -json tfplan' to get a machine-readable diff. Look for resources with actions like 'update in-place' or 'destroy and recreate'. For each drifted resource, run 'terraform state show <resource_address>' to see the current state values, and compare with your configuration file. For example, if an AWS security group rule is missing in state, you'll see it in the plan as an addition.

Cloud provider audit logs are gold. In AWS, use CloudTrail via CLI: 'aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue=<resource-id>'. This shows who made the change, from where, and when. In GCP, use 'gcloud logging read "resource.type=aws_..."'. Cross-reference the timestamp with incident reports or deployment windows. Also check if any automated scripts (e.g., Lambda functions) have permissions to modify the resource—often drift comes from automation that isn't tracked in IaC.

Once drift is confirmed, the safest approach is to update the Terraform configuration to match the desired state (which includes the external changes if they are valid) and then run 'terraform apply -refresh-only' to sync state. This ensures the state file correctly represents reality without making any resource changes. After that, a normal 'terraform plan' should show no diff. If the external change is undesirable, you need to reverse it: first, revert the manual change via the console or API, then run 'terraform apply' to enforce the correct state.

For resources that must not be disrupted (e.g., production databases), use 'terraform plan -target=<resource>' to scope operations. If a resource has been deleted outside Terraform, 'terraform apply' will try to recreate it—which may be desired or not. Use 'terraform state rm' to remove the deleted resource from state if you don't want Terraform to recreate it. Always test in a non-production environment first. For complex drifts, consider using 'terraform import' to bring existing resources under management with the correct config.

Prevention starts with governance. Use IAM policies to restrict write access to resources outside of Terraform. In AWS, you can use Service Control Policies (SCPs) to deny modifications unless they come from a specific IAM role used by Terraform. For example, add a condition that checks 'aws:ViaAWSService' or 'aws:PrincipalArn' matches your CI/CD role. This forces all changes through Terraform.

Implement automated drift detection in your CI/CD pipeline. After every apply, run a follow-up 'terraform plan' and fail the pipeline if there are any diffs. Use tools like Terraform Cloud's drift detection, Atlantis, or a simple cron job that runs 'terraform plan' and alerts on changes. Also, educate your team: any emergency fix must be followed by a ticket to update Terraform config and state. The 'lifecycle' meta-argument can be used sparingly—for example, 'ignore_changes' on tags that are modified by external systems, but avoid overuse as it hides drift.

When multiple team members or automation processes modify the same resource, state conflicts can arise. Terraform supports state locking via backends like S3 (with DynamoDB) or Azure Storage. Ensure locking is enabled and your CI/CD system uses it. If a lock is not released (e.g., due to a failed apply), you may need to force unlock: 'terraform force-unlock <lock_id>'. Be cautious—forcing unlock can lead to state corruption.

Another advanced scenario is when external changes happen between a plan and an apply. For example, you run 'terraform plan', see no issues, but by the time you apply, someone manually changes the resource. To mitigate, use 'terraform plan -out=tfplan' and then 'terraform apply tfplan'—this replans based on the saved plan, but the apply still does a final refresh. For critical resources, consider using 'precondition' and 'postcondition' checks to validate state before and after apply. In Terraform 1.2+, you can use 'check' blocks for custom assertions.