PostgreSQL · intermediate
PostgreSQL replica-lag checklist
PostgreSQL replica-lag checklist for engineers whose read replicas are falling behind the primary write stream. Walks through what to observe in pg_stat_replication and pg_stat_subscription, how to decide whether the bottleneck is network, disk, replay, or apply-side, and the safe, read-only verification steps to confirm recovery before reopening traffic.
The symptoms
- •Read queries on a streaming replication standby return stale rows minutes after a write commits on the primary.
- •Application-level user reports describe "I just saved it, why don't I see it?" while the primary's commit latency stays flat.
- •pg_stat_replication.replay_lag or replay_lsn distance grows monotonically during a write burst and does not recover after the burst ends.
- •Logical replication subscriptions show high apply worker queue depth in pg_stat_subscription with no matching network problem.
Likely causes
- •Standby replay is throttled by an under-provisioned disk or WAL volume on the replica, so redo cannot keep pace with WAL arrival.
- •Network or replication slot retention prevents WAL records from being applied because the replica can receive but not replay them.
- •Long-running read queries on the standby hold recovery conflicts, pausing replay via hot_standby feedback / vacuum_defer_cleanup_age / max_standby_streaming_delay.
- •Logical replication apply worker is bottlenecked by missing indexes on the subscriber side, schema drift, or constraint enforcement per row.
- •Write workload on the primary changed shape (large batch UPDATE, maintenance, bulk loads) and the replica's single apply path cannot absorb the rate.
First ten minutes
- 01Confirm the read path is actually hitting a replica and not the primary: check the application connection string, PgBouncer or RDS proxy pool, and the server's session_id or application_name.
- 02On the primary, run pg_stat_replication and record replay_lsn, write_lsn, flush_lsn, replay_lag, and the state column for each connected replica. Sort by replay_lsn distance.
- 03On the primary, inspect pg_replication_slots for active slot names and then query pg_stat_replication alongside the slot to map lag to a specific downstream.
- 04On each replica, inspect pg_stat_wal_receiver for connection status, last_msg_receipt_time, and last_msg_send_time to differentiate "no WAL arriving" from "WAL arriving but not replayed."
- 05Check the replica's receive vs. replay LSN from pg_stat_wal_receiver and the WAL sender's send vs. write vs. flush vs. replay LSN to see which stage is the bottleneck.
- 06Capture a 60-second sample of write/sync/replay counters from pg_stat_io or from the OS view of the WAL volume to see whether the gap is growing, flat, or shrinking.
- 07Decide the bottleneck stage — receive, write/flush, or replay — before changing any configuration.
Evidence to collect
- •pg_stat_replication row for the affected replica: pid, state, write_lsn, flush_lsn, replay_lsn, sync_priority, sync_state, replay_lag.
- •pg_replication_slots row showing active, restart_lsn, and confirmed_flush_lsn for the replica's slot.
- •pg_stat_wal_receiver row on the replica: status, last_msg_send_time, last_msg_receipt_time, received_lsn, last_msg_receipt_lsn.
- •pg_stat_subscription row on the subscriber for logical replicas: pid, state, received_lsn, last_msg_receipt_time, last_msg_receipt_lsn.
- •Replica-side pg_stat_activity for any long-running read query that may be holding a buffer pin or AccessExclusive lock.
- •PostgreSQL log on the replica filtered for "recovery conflict," "canceling statement due to conflict with recovery," and "FATAL: hot_standby."
Where to look
- •On the primary, the replication boundary: pg_stat_replication and pg_replication_slots expose per-downstream LSN stages and lag.
- •On the streaming replica, the apply boundary: pg_stat_wal_receiver exposes receive-side health, and the replay rate is observable via WAL receiver LSN deltas over time.
- •On the logical replication subscriber, the apply boundary: pg_stat_subscription tracks the apply worker; pg_stat_progress_create_index and pg_stat_user_tables index scan counts hint at apply-side cost.
- •On the replica's local storage, the WAL boundary: the pg_wal directory's write rate and the underlying disk's await/svctm or IOPS ceiling determine whether replay can keep up.
- •On the replica's query workload, the conflict boundary: pg_stat_activity plus hot_standby feedback state shows whether reads are blocking replay.
Diagnostic steps
- 01Compare replay_lsn to flush_lsn on pg_stat_replication. If replay trails flush by a large margin, replay is the bottleneck; if flush trails write by a large margin, the network or replica disk is the bottleneck.
- 02Compare last_msg_send_time on the primary to last_msg_receipt_time on the replica. A gap widening over many samples indicates network or TCP buffering, not apply.
- 03Confirm the replica's slot is active and restart_lsn is moving. A frozen restart_lsn with growing replay_lag often indicates the receiver is healthy but the apply worker is stuck on a single transaction.
- 04On logical replicas, run pg_stat_activity filtered to the apply worker PID and check its current query, wait_event, and locks. Apply time per transaction is the leading indicator.
- 05On the replica, search recent logs for "recovery conflict" entries. Each entry identifies the blocking backend PID and the conflict type (buffer pin, lock, snapshot, table space).
- 06Cross-reference the replica's recent maintenance: vacuum, autovacuum, ALTER TABLE, or CREATE INDEX can briefly stall replay; sustained lag points to a workload change, not a one-off event.
- 07Decide on the boundary: receive, replay, or apply. The fix differs at each boundary, so do not skip this step.
Common mistakes
- •Lowering hot_standby feedback or max_standby_streaming_delay as a first move. These are the only protection against read queries blocking replay, so changing them without evidence can cause read errors on the replica.
- •Creating a new replication slot to "reset" lag. New slots inherit the same downstream bottleneck and add WAL retention risk on the primary.
- •Pointing application reads at the primary to "fix" lag. This hides the symptom, transfers load to the primary, and removes the read-scaling property the replica exists to provide.
- •Assuming streaming replication lag implies a network problem. Most observed lag is replay-bound, not network-bound, and shows up as flush equal to write but replay trailing.
- •Confusing bytes-on-the-wire with rows-applied. On logical replicas, received LSN advances while row-level apply is slow due to missing subscriber indexes or triggers.
Safe fixes
- •If replay trails flush, raise the replica's WAL volume IOPS ceiling and confirm wal_receiver_timeout and wal_receiver_status_interval match the primary's settings before the next sample window.
- •If reads block replay (recovery conflict log entries), tune access on the standby route: add application_name to identify the consumer, then increase max_standby_streaming_delay only after measuring the current query tail latency.
- •If logical apply is slow, add the missing index on the subscriber before replaying the backlog, and pause the subscription with ALTER SUBSCRIPTION ... DISABLE only when the user has confirmed the change is reversible.
- •If the slot grew because of a paused replica, temporarily extend the primary's max_wal_size only after checking current WAL volume usage, and rebuild the replica's recovery state before relying on it again.
- •After any change, re-run the same pg_stat_replication select and confirm replay_lsn is closing the gap to flush_lsn for at least two consecutive 60-second samples.
Prove the fix
- 01On the primary, replay_lsn - flush_lsn stays at zero, or replay_lsn distance to flush_lsn monotonically decreases across at least two 60-second samples while a steady write load is applied.
- 02On the replica, a control write on the primary is visible at the replica's read consistency within the previously observed SLO, sampled under the same workload that produced the original lag.
- 03pg_replication_slots.restart_lsn advances at the same rate as write_lsn during the sample window, proving the slot is not holding back WAL.
- 04No new "recovery conflict" or "canceling statement due to conflict with recovery" entries appear in the replica log during the sample window after the fix.
- 05For logical replicas, pg_stat_subscription.received_lsn equals last_msg_receipt_lsn and apply worker CPU utilization returns to the pre-incident baseline.
Prevention and next steps
- •Alert on replay_lsn - flush_lsn exceeding a documented per-environment threshold, scoped per replica, not globally, so a single slow downstream does not fire every alert.
- •Keep a runbook that maps each replica to its slot, its disk class, and its read consumer, so the first responder can identify the bottleneck stage in the first ten minutes.
- •Schedule replica-side checks: confirm hot_standby feedback state, confirm wal_receiver_timeout is set, and confirm pg_wal size is below the disk's free-space threshold.
- •Before bulk loads on the primary, lower the replica's read traffic or pre-warm indexes on the subscriber for logical replicas, so a known workload change does not become a surprise.
Safe commands and checks
SELECT pid, state, write_lsn, flush_lsn, replay_lsn, replay_lag, sync_state FROM pg_stat_replication ORDER BY replay_lsn; SELECT slot_name, active, restart_lsn, confirmed_flush_lsn, wal_status FROM pg_replication_slots ORDER BY restart_lsn; SELECT status, receive_start_lsn, received_lsn, last_msg_send_time, last_msg_receipt_time, last_msg_receipt_lsn FROM pg_stat_wal_receiver; SELECT pid, state, received_lsn, last_msg_receipt_time, last_msg_receipt_lsn FROM pg_stat_subscription; SELECT datname, usename, application_name, state, wait_event, query_start, state_change FROM pg_stat_activity WHERE application_name = '<replica_consumer_name>'; SELECT pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), '0/0')) AS current_wal_bytes;