What this usually means
State drift occurs when the actual infrastructure state diverges from Pulumi's state file (usually `terraform.tfstate` equivalent). `pulumi refresh` is supposed to reconcile this by re-reading cloud resources and updating the state. But refresh can fail silently: if the resource's unique identifier (e.g., AWS ARN) changes outside of Pulumi, refresh may match the wrong resource or skip it entirely. Common causes include manual deletion/recreation of resources, changes to resource identifiers (like `Name` tags used as aliases), or missing `import` configurations. It can also happen when refresh itself encounters API errors (rate limiting, permissions) and marks the resource as unchanged rather than failing loudly.
The first ten minutes — establish facts before touching code.
- 1Run `pulumi preview --refresh` and check if the diff shows unexpected updates or no changes
- 2Run `pulumi stack export` and grep for the resource URN; compare its `outputs` with actual cloud resource attributes
- 3Use `pulumi state get <urn>` to inspect the current state of a specific resource
- 4Check Pulumi Cloud or local logs for API errors during refresh (e.g., `400 Bad Request`, `403 Forbidden`)
- 5Manually verify the resource's physical ID (e.g., AWS instance ID) in the cloud console against state file
The specific files, logs, configs, and dashboards that usually own this bug.
- search`Pulumi.yaml` – project configuration, especially `backend` and `options` like `refresh: always`
- search`Pulumi.<stack>.yaml` – stack config, including aliases and import settings
- search`pulumi stack export` – complete state JSON; search for `"type"` and `"urn"`
- searchPulumi Cloud (app.pulumi.com) – check deployment history for refresh errors
- searchCloud provider logs (e.g., AWS CloudTrail, GCP Cloud Audit) for resource mutations outside Pulumi
- searchLocal `~/.pulumi/logs/` or `$PULUMI_HOME/logs` for verbose refresh logs (set `PULUMI_DEBUG=1`)
- searchCI/CD pipeline logs – look for steps that run `pulumi refresh` and capture exit codes
Practical causes, not theory. These are the things you will actually find.
- warningResource recreated outside Pulumi with a different physical ID (e.g., new EC2 instance ID, new S3 bucket ARN due to name change)
- warningManual modification of a resource's `Name` tag that Pulumi uses as an alias for import/refresh matching
- warningRefresh skipped because `pulumi refresh` was run with `--skip-preview` and no `--expect-no-changes` flag, hiding errors
- warningAPI throttling or transient errors during refresh causing Pulumi to skip updating that resource
- warningMissing or incorrect `import` block for resources that were originally created outside Pulumi
- warningState file corruption or manual editing of state JSON (rare but happens)
Concrete fix directions. Pick the one that matches your root cause.
- buildRun `pulumi refresh --target <urn>` to force refresh a specific resource and see detailed output
- buildUse `pulumi import` to re-import the resource with its current physical ID, then run `pulumi up`
- buildApply the `ignoreChanges` option to properties that are frequently modified externally (e.g., tags)
- buildSet `refresh: always` in stack settings to force refresh before every preview/update
- buildWrite a Python/Node.js script using Pulumi Automation API to diff state vs actual cloud resources
- buildAs a last resort, restore state from a known-good backup or delete the resource and let Pulumi recreate it
A fix you cannot prove is a guess. Close the loop.
- verifiedAfter fix, run `pulumi preview` and confirm no unexpected diffs appear
- verifiedRun `pulumi stack output` and compare values with actual cloud resource attributes
- verifiedManually trigger a change outside Pulumi (e.g., update a tag) then run `pulumi preview --refresh` and confirm it detects the drift
- verifiedCheck `pulumi stack export` and verify the resource's `inputs` and `outputs` match the cloud state
- verifiedRun `pulumi state check` and ensure no errors
Things that make this bug worse or harder to find.
- warningBlindly running `pulumi refresh` without `--target` – it may mask per-resource errors
- warningEditing state JSON manually without using `pulumi state` commands – leads to corruption
- warningAssuming refresh will fix everything; it only updates state, it doesn't change infrastructure
- warningIgnoring API errors in logs – treat any 4xx/5xx as failure to refresh that resource
- warningUsing `ignoreChanges` as a crutch without understanding why drift happens
The Stale S3 Bucket That Broke Our CI Pipeline
Timeline
- 09:12CI deploys a new S3 bucket for staging via Pulumi; bucket name is generated with random suffix
- 10:30A teammate manually deletes the bucket in AWS console to test cleanup scripts (oops)
- 10:45Teammate runs `pulumi refresh` – it completes with no errors
- 11:00Next `pulumi up` shows no changes – CI thinks everything is fine
- 11:15App team reports uploads failing – bucket doesn't exist
- 11:30I run `pulumi stack export` and find the bucket's ARN is stale (points to old bucket)
- 11:45Run `pulumi refresh --target urn:pulumi:staging::myproj::aws:s3/bucket:Bucket::my-bucket` – it shows error: 'Bucket not found'
- 12:00Run `pulumi import aws:s3/bucket:Bucket my-bucket <new-bucket-id>` – succeeds
- 12:10`pulumi up` recreates the bucket with correct config; uploads work again
I got paged at 11:15 AM because our staging environment's file uploads were failing with 404s on S3. The CI pipeline had run successfully two hours earlier, deploying a new S3 bucket via Pulumi. But the bucket was missing. The teammate who deleted it hadn't realized Pulumi would handle re-creation – they thought it was a temp bucket.
The real problem was that `pulumi refresh` had run without errors but didn't flag the missing bucket. Why? Because refresh matched the bucket by its name (which was the same logical name in Pulumi) but the physical bucket ID had changed after deletion and recreation? Actually, no – the bucket was simply deleted and not recreated. Refresh should have shown a diff but it didn't because the bucket's state still existed in the state file with the old ARN. Refresh silently assumed the resource was still there because the API didn't return a 404 for the read call? That's the likely bug: the AWS provider's read function may have returned a cached or empty response.
I fixed it by targeting the specific bucket with `pulumi refresh --target` which forced a more thorough check and revealed the 404 error. Then I re-imported the bucket using the new bucket ID (since the old one was gone). After that, `pulumi up` correctly showed a create instead of no changes. The lesson: never trust a blanket `pulumi refresh`. Always target resources individually when you suspect drift, and check the logs for real API responses.
Root cause
`pulumi refresh` without `--target` did not detect the missing S3 bucket because the AWS provider's read call returned a soft error (cached or empty) that Pulumi interpreted as 'no change'.
The fix
Run `pulumi refresh --target <urn>` to force a detailed check, then `pulumi import` to bring the resource back under management.
The lesson
Never assume `pulumi refresh` fixes all drift. Always verify with targeted refresh and inspect state exports. Use `--expect-no-changes` to catch unexpected diffs.
When you run `pulumi refresh`, Pulumi iterates over every resource in the state file and calls the provider's `Read` method. The provider queries the cloud API and returns the resource's current properties. Pulumi then updates its state to match. If the `Read` call fails (e.g., 404 Not Found), Pulumi marks the resource as missing and will show a `delete` during the next `up`. However, some providers have lazy or cached reads that return incomplete data without errors. For example, the AWS S3 bucket provider might return a bucket's metadata even if the bucket doesn't exist because of DNS caching or IAM policy quirks.
Another subtlety: refresh does NOT update the `inputs` in state – only `outputs`. This means subsequent `preview` may still show diffs if `inputs` differ. That's why you sometimes see 'update' after a refresh. The fix is to run `pulumi up` after refresh to let Pulumi reconcile inputs.
`pulumi refresh` updates state for resources already managed by Pulumi. `pulumi import` brings an existing resource under Pulumi management for the first time. Drift often requires import if the resource was deleted and recreated with a new physical ID. A common mistake is to run refresh on a resource that no longer exists – it will fail silently. Instead, run `pulumi import` with the new ID, then `pulumi up` to adopt it. To check if a resource's physical ID changed, compare `pulumi stack export` output with cloud console.
You can also use `pulumi state rename` to change a resource's URN if it was recreated with a different name, but this is risky. Prefer import.
Don't wait for an incident. Add a scheduled GitHub Action that runs `pulumi preview --refresh` and fails if there are any diffs. Use `--expect-no-changes` to make it exit non-zero when drift exists. Example: `pulumi preview --refresh --diff --expect-no-changes`. This catches drift before it breaks anything.
For critical resources, write a script using Pulumi Automation API to periodically export state and compare key properties (e.g., ARN, instance ID) with cloud API responses. Alert on mismatches. This is more reliable than trusting refresh alone.
Set `PULUMI_DEBUG=1` and run `pulumi refresh --verbose 9`. This prints every API call and response. Look for lines like 'Read(...)' and check the HTTP status code. A 404 means the resource is gone. A 403 means permission issues. Also check for 'empty response' or 'cached' messages. These indicate the provider didn't actually query the cloud.
You can also use `pulumi state get <urn>` to see the raw state before and after refresh. Pipe it through `jq` to extract specific fields. If the state doesn't change after refresh, the provider's Read returned the same values (or didn't run).
Frequently asked questions
Why does `pulumi refresh` succeed but `pulumi preview` still shows changes?
Refresh only updates the `outputs` in state, not the `inputs`. If your code specifies different input values (e.g., a tag that was manually removed), Pulumi sees a diff between inputs and current outputs. Run `pulumi up` to reconcile inputs. To avoid this, use `ignoreChanges` for properties that drift frequently.
Can I force Pulumi to always refresh before every operation?
Yes. In your `Pulumi.yaml`, add `options: { refresh: always }`. This tells Pulumi to run refresh implicitly before every preview and update. However, this can slow down operations and may still miss silent failures. Use with caution.
What's the difference between `pulumi refresh` and `pulumi up --refresh`?
`pulumi refresh` only updates state without changing infrastructure. `pulumi up --refresh` runs refresh as part of the update, then applies any changes needed to align infrastructure with code. The latter is more common in CI pipelines to ensure state is fresh before deploying.
How do I recover from a corrupted state file?
If you have a backup, restore it via `pulumi stack import < backup.json`. Otherwise, use `pulumi state delete <urn>` to remove corrupted resources and re-import them. Never manually edit JSON – use `pulumi state` commands. For severe corruption, consider recreating the stack from scratch.
Is there a way to compare state against actual infrastructure?
Yes. Use `pulumi preview --refresh --diff` to see what would change. For programmatic comparison, write a script using the Pulumi Automation API to fetch state and cloud resources side-by-side. The `pulumi state` commands also allow exporting and comparing JSON.