Deployment rollback checklist

The symptoms

• •Error rate, p99 latency, or saturation in production dashboards climbs within minutes to hours of the new version becoming the live traffic target.
• •Health endpoints on instances of the new version start returning non-2xx responses or stop responding within the configured timeout.
• •Upstream proxies or load balancers begin reporting HTTP 502 Bad Gateway responses when routing to the new version, indicating the upstream pool cannot produce a valid response.
• •Application or platform logs from the new version show repeating startup, dependency-resolution, or readiness-check failures that did not appear in the previous version.
• •Synthetic or canary checks tied to the release tag diverge from the previous version's baseline by more than the agreed error budget.

Likely causes

• •A configuration, secret, or feature-flag value shipped with the new version is incompatible with the running environment, so the process starts but cannot serve valid responses.
• •A dependency or schema change in the new version breaks compatibility with downstream consumers, causing upstream proxies to return 502 when backends close or hang connections.
• •Resource limits, connection pools, or worker counts in the new version are mis-sized for current traffic, producing timeouts that surface as 502 at the edge.
• •A previously healthy prior version has been removed or marked unhealthy by the deploy system, leaving no safe target to roll back to unless the prior artifact is still available.
• •The rollback path itself is unverified: the prior version's artifact, configuration set, or database migration state cannot be reinstated without data loss or extended downtime.

First ten minutes

1. 01Stop further automated rollout steps so the bad version does not spread across more regions, shards, or instance groups.
2. 02Capture the exact release identifier, build SHA, configuration revision, and timestamp of the deployment so every later action references the same artifact.
3. 03Confirm the failure is in the application tier, not in shared infrastructure, by checking edge and proxy logs for HTTP 502 responses and identifying which upstream pool they reference.
4. 04Verify that the previous version's artifact, configuration, and database schema are still available and that down-migration is safe, before any rollback command is executed.
5. 05Decide rollback versus forward-fix using a written threshold: if user-facing error rate or 502 share exceeds the agreed budget and forward-fix ETA exceeds the agreed service-level objective, proceed to rollback.
6. 06Communicate the decision in the incident channel with the rollback target version, expected duration, and the verification signal that will declare success.

Evidence to collect

• •Edge or load balancer access logs showing HTTP 502 counts before and during the incident, scoped to the release identifier.
• •Per-version health and readiness endpoint response codes and latencies for both the current and prior versions, sampled over the same time window.
• •Deploy-system record linking the running artifact SHA to configuration revisions, feature flags, and database migrations in effect.
• •Application logs from instances of the current version showing the first failure timestamp and any recurring error class.
• •A diff between the previous and current version's external response on a fixed request, captured from synthetic probes or replayed traffic.

Where to look

• •Edge and reverse-proxy logs at the boundary between clients and the application, where HTTP 502 responses and their upstream pool are first visible.
• •The deploy system's release history showing which artifact SHA and configuration revision are pinned to each environment.
• •Artifact or image registry containing the previous version's build, and the configuration store containing the previous version's pinned values.
• •Application health, readiness, and metrics endpoints on instances of both versions, used to compare behavior on identical input.
• •Database migration ledger showing which schema versions are forward-compatible and whether the previous application version can read current schema state.

Diagnostic steps

1. 01Compare edge HTTP 502 share and upstream error mix before versus after the release timestamp to confirm the regression is tied to the deploy boundary.
2. 02Replay a known-good synthetic request against both the current and prior version's health endpoint, recording response code, headers, and body for a controlled diff.
3. 03Inspect the deploy system to confirm the prior version's artifact is still present, the prior configuration revision is restorable, and no irreversible data migration was applied.
4. 04Check whether the failure reproduces with the current version under reduced traffic, which can isolate capacity and timeout issues from code or configuration defects.
5. 05Verify the rollback mechanism on a non-production or shadow environment using the exact artifact and configuration revision you intend to restore, so the production action is rehearsed.
6. 06Decide the rollback scope: full regional revert, canary-only revert, or feature-flag-only revert, based on which blast radius restores the agreed service-level objective fastest.

Common mistakes

• •Rolling back without confirming that the previous version's artifact, configuration, and database schema are still compatible, which can replace one outage with a worse one.
• •Declaring success because error dashboards calm briefly, without checking that the prior version is actually serving the affected traffic and that 502 share has returned to baseline.
• •Mutating production data to "fix" the new version before deciding to roll back, which can leave the previous version unable to read what was written.
• •Relying on internal deploy-system "success" indicators instead of user-facing HTTP responses, which can report a green rollout while edge proxies still return 502.
• •Skipping the rehearsal step, so the first time the rollback procedure executes is during the incident, with untested ordering of traffic shift, config revert, and schema steps.

Safe fixes

• •Shift traffic away from the current version back to the previous version through the deploy system's traffic controller, keeping the prior artifact and configuration pinned throughout.
• •Revert the configuration revision to the one recorded for the previous version, then confirm via the deploy system that the live configuration matches the recorded prior state.
• •If the failure is gated by a feature flag, disable only the offending flag at the edge and re-measure before attempting a full version rollback, provided the flag's disable path was rehearsed.
• •If a database migration is forward-only and the prior version cannot read the new schema, do not roll back the application; instead, mitigate at the application layer or in a forward-compatible hotfix.
• •After traffic is fully on the prior version, drain and terminate instances of the new version so they do not re-enter the upstream pool and re-trigger 502 responses.

Prove the fix

1. 01Edge and proxy logs show HTTP 502 share for the affected service returning to the pre-incident baseline within the agreed observation window after traffic is shifted.
2. 02Synthetic probes replayed against the now-live prior version produce response codes, headers, and bodies that match the recorded pre-incident baseline within tolerance.
3. 03The deploy system record shows the prior artifact SHA and prior configuration revision pinned as the live target, with no remaining instances of the new version serving traffic.
4. 04Per-version metrics for the prior version match its pre-incident values for error rate, p99 latency, and saturation, sampled over at least one full traffic cycle.
5. 05A post-incident note records the rollback decision threshold, the artifacts and configuration involved, and the verification evidence, so the next rollback can be rehearsed against the same criteria.

Prevention and next steps

• •Keep at least one prior version's artifact, configuration revision, and database schema state available and explicitly listed as a rollback target in the deploy system.
• •Rehearse the rollback path in a non-production environment on a fixed cadence, using the same traffic-shift, config-revert, and schema steps planned for production.
• •Treat forward-only database migrations and irreversible data changes as a separate gate that must be cleared before any release that may need to be rolled back.
• •Wire HTTP 502 share, health endpoint status, and synthetic probe results into the same dashboard as deploy events so regressions are visible at the release boundary, not only inside the application.
• •Write the rollback decision threshold, scope, and verification signal into the runbook so each incident applies the same criteria rather than re-deciding under pressure.

Safe commands and checks

echo "Inspect deploy history for the affected service: list release identifiers, build SHAs, and configuration revisions so the rollback target is named explicitly."
echo "Filter edge or reverse-proxy logs for status 502 over the incident window, grouped by upstream pool and release identifier, to confirm the regression is tied to the deploy boundary."
echo "Capture a synthetic probe against the affected health endpoint on both versions, saving response code, latency, and body for later diffing against baseline."
echo "Query the artifact registry for the prior version's build SHA and verify it is still present and pullable from the deploy runners that will execute the rollback."
echo "Read the configuration store for the previous version's pinned revision and export a checksum so the live configuration can be compared after revert."
echo "Inspect the database migration ledger to confirm whether the schema is forward-compatible with the prior application version before any rollback is executed."
echo "After traffic shift, sample edge logs and per-version metrics over at least one full traffic cycle to verify HTTP 502 share and latency match the pre-incident baseline."
echo "Record the rollback target, decision threshold, and verification evidence in the incident timeline so the same criteria can be reused on the next rollback."