PostgreSQL · intermediate
PostgreSQL query became slow after deploy: compare plan inputs
When a PostgreSQL query slows down right after a deploy, the most productive first move is to compare the new plan against the old one at the level of inputs the planner actually consumes — statement text, parameter values, table statistics, and index inventory — rather than chasing the symptom in the application.
The symptoms
- •One or a small set of queries suddenly take longer or use more I/O after a deploy, while neighboring queries on the same database remain unaffected.
- •Application latency or timeout errors appear in the same window as a code release, schema migration, or ANALYZE-adjacent change, and disappear when the deploy is reverted.
- •EXPLAIN (ANALYZE, BUFFERS) output for the regressed query shows a different node shape than before, even though the underlying tables and indexes are unchanged.
- •Row estimates from the planner diverge sharply from actual row counts in the new EXPLAIN output, signaling a stats or input drift rather than a true data-volume change.
- •pg_stat_statements totals show the same call count and total time rising for the query, indicating planner-side change rather than traffic spike.
Likely causes
- •The application or migration silently rewrote the statement text, so the planner now hashes a different normalized query and picks a different plan.
- •Bound parameter values changed in a way that pushed the planner across a plan-choice boundary, often via a prepared statement whose generic plan is now suboptimal.
- •An ALTER TABLE from the deploy skewed column statistics (for example via default expressions, type changes, or extended statistics) so row estimates no longer match reality.
- •A migration added, dropped, or rewrote an index that the regressed query used to rely on, forcing the planner into a seq scan or a worse join order.
- •Concurrent autovacuum or a manual ANALYZE that ran during the deploy window produced statistics that the planner is now reading, even though the schema itself did not change.
First ten minutes
- 01Pin the exact statement text the application is sending to PostgreSQL by capturing it from pg_stat_statements or from application logs, so any later comparison is anchored to a real query.
- 02Pull EXPLAIN (ANALYZE, BUFFERS) output for that exact statement against a representative dataset, and note the top-level node, join order, and row-estimate versus actual-row divergence.
- 03Pull the same statement's EXPLAIN output from before the deploy — from a saved plan, a staging run, or a rollback environment — so you have a side-by-side of plan shape.
- 04Diff the two EXPLAIN plans node by node: operator type, join order, scan method, and row estimates at each node, treating each difference as a hypothesis to confirm.
- 05Inventory the planner's inputs at the moment of regression: the statement text, bound parameters, current statistics on the touched tables, and the visible index list, so each EXPLAIN difference can be matched to a concrete input change.
Evidence to collect
- •Side-by-side EXPLAIN (ANALYZE, BUFFERS) outputs for the regressed statement, captured before and after the deploy against a frozen dataset with identical parameter values.
- •pg_stats snapshots for every column the planner referenced in either plan, recording null fraction, most-common-values, and histogram bounds across the deploy boundary.
- •pg_stat_user_tables last_analyze, last_autoanalyze, and n_mod_since_analyze for the touched tables, so an ANALYZE during the deploy window is visible rather than inferred.
- •An index inventory from pg_indexes and pg_index for the touched tables, including expression and partial index predicates, before and after the deploy.
- •pg_stat_statements rows for the regressed query, showing normalized query hash, call count, and total time so the regression can be quantified against a baseline.
Where to look
- •pg_stat_statements, normalized query text, and per-call timing so you can isolate which statement changed its plan.
- •pg_stats and pg_class reltuples/relpages for the tables touched by the regressed query, since statistics are the planner's primary input.
- •pg_indexes plus pg_index.indpred for indexes on the touched tables, including expression and partial indexes that the planner may now ignore.
- •pg_stat_user_tables last_analyze, last_autoanalyze, and n_mod_since_analyze to see whether the deploy window changed the stats feeding the planner.
- •The application's parameter binding path, since prepared statement plans and generic-plan vs custom-plan choices are a common silent source of regressions.
Diagnostic steps
- 01Capture the current EXPLAIN (ANALYZE, BUFFERS) output for the regressed statement and label it 'after', and obtain the pre-deploy 'before' EXPLAIN for the same logical query on the same dataset.
- 02Diff the two plan trees top-down: at every node compare operation type, relation, index, join strategy, and estimated rows versus actual rows; mark each difference with the hypothesis it implies.
- 03Confirm whether the statement text itself changed by hashing the normalized query string before and after; if hashes differ, the planner is genuinely looking at a new statement and the rest of the diff is suspect.
- 04Inspect pg_stats for every column the planner referenced in both plans; record null fraction, most-common-values list length, and histogram bounds, and compare them across the deploy boundary.
- 05List the indexes available to each plan via pg_indexes and pg_index, then mark which indexes disappeared, appeared, or were redefined in the deploy; correlate each disappearance with a plan node change.
- 06Reproduce both plans on a frozen copy of the data using SET statement_timeout and SET work_mem to controlled values, so the comparison isolates planner inputs from runtime resource effects.
Common mistakes
- •Comparing EXPLAIN output without ANALYZE, so actual row counts are missing and you cannot tell a stats drift from a real data-volume change.
- •Diffing plans that were run against different parameter values or different datasets, so a plan-shape change is misattributed to the deploy instead of to the inputs.
- •Chasing the wrong regression by assuming the planner changed when in fact the application rewrote the statement text under the same logical name.
- •Treating an ANALYZE as a fix without proving it changed the specific statistics that explain the row-estimate divergence in the new plan.
- •Dropping and recreating indexes reactively, which masks the original input change and prevents a clean before/after proof.
Safe fixes
- •If statement text changed, treat that as the regression boundary: revert the application change that rewrote the query, since plan-shape diffs downstream of a text change are usually symptoms, not causes.
- •If row estimates diverge but the statement and indexes are unchanged, run ANALYZE on the touched tables and re-capture EXPLAIN (ANALYZE, BUFFERS); proceed only if the new plan's estimates now align with actual rows.
- •If an index disappeared or was rewritten, do not drop and rebuild speculatively; instead, in a non-production environment, recreate the prior index definition and confirm the old plan returns before touching production.
- •If a prepared statement is using a generic plan that is now suboptimal, force a custom plan via SET plan_cache_mode, then re-capture EXPLAIN to confirm the regression is tied to plan caching and not to schema or stats.
- •If no input can be shown to explain the plan change, revert the deploy as the default safe action and reopen the investigation with a frozen dataset that supports a controlled before/after comparison.
Prove the fix
- 01EXPLAIN (ANALYZE, BUFFERS) for the regressed query, run on a fixed dataset and fixed parameter values, shows the original plan shape, with estimated rows within a small multiple of actual rows at every node.
- 02pg_stat_statements reports total execution time for the regressed query returning to within a documented tolerance of its pre-deploy baseline over a representative time window.
- 03Each planner input identified in the diff — statement text, statistics snapshot, and visible index set — can be shown to match the pre-deploy value, so the plan match is explained rather than coincidental.
- 04Application-side latency and timeout-error counters for the affected code path return to their pre-deploy baseline during a verification window that excludes unrelated traffic variation.
Prevention and next steps
- •Capture and store EXPLAIN (ANALYZE, BUFFERS) output for known-critical queries as part of the deploy artifact so regressions can be diffed immediately, not reconstructed under pressure.
- •Treat statement text as a contract: any change to the normalized query string should be visible in code review as its own boundary, separate from data or schema changes.
- •Run ANALYZE explicitly after data-loading migrations so the planner's statistics move on the schedule you control, rather than waiting for autovacuum during the deploy window.
- •Track per-query timing in pg_stat_statements as a deploy-time signal so a plan regression is visible against a baseline rather than inferred from user complaints.
Safe commands and checks
SELECT query, calls, total_exec_time, mean_exec_time FROM pg_stat_statements WHERE query ILIKE '%<signature>%' ORDER BY total_exec_time DESC LIMIT 10;
EXPLAIN (ANALYZE, BUFFERS) <regressed query with bound literals substituted>;
SELECT schemaname, relname, last_analyze, last_autoanalyze, n_mod_since_analyze FROM pg_stat_user_tables WHERE relname IN (<table_a>, <table_b>);
SELECT schemaname, tablename, indexname, indexdef FROM pg_indexes WHERE tablename IN (<table_a>, <table_b>) ORDER BY indexname;
SELECT tablename, attname, null_frac, most_common_vals, histogram_bounds FROM pg_stats WHERE tablename IN (<table_a>, <table_b>) AND attname IN (<col_1>, <col_2>);
SELECT pg_relation_size('<index_name>'), pg_size_pretty(pg_relation_size('<index_name>'));