PostgreSQL replica replay lag: distinguish write pressure from apply blockage

The symptoms

• •pg_stat_replication.replay_lag or replay_lsn on the primary climbs steadily while replica's pg_last_wal_replay_lsn lags behind pg_last_wal_receive_lsn, indicating replay cannot keep up with receive.
• •Replica's pg_stat_replication on a cascading standby shows the upstream's send/flush/replay LSNs diverging, often while CPU and disk write latency on the replica remain low.
• •Application reads from the replica return stale data with increasing staleness, while primary writes succeed normally and other replicas stay current.
• •System-level symptoms differ by mode: high WAL receiver disk write throughput with low startup process CPU points to write pressure; high startup process CPU or visible waits on AccessShareLock or RowExclusiveLock on hot tables point to apply blockage.
• •Recovery is reportedly finished ("consistent recovery state reached") yet replay continues to fall behind, suggesting apply starvation rather than initial catch-up.
• •Long-running transactions or replication slot lag warnings appear without a corresponding spike in writes, pointing at a held-back apply.

Likely causes

• •Write pressure: WAL volume generated on the primary exceeds the replica's sustained fsync and write throughput on its pg_wal directory, even though apply is otherwise idle.
• •Apply blockage: the startup process is blocked on a lock held by a long-running query, a DDL, or a conflicting VACUUM FULL/REINDEX on a table involved in the replay path.
• •Missing or invalid replica identity: UPDATE/DELETE on a table without a replica identity forces the apply process to scan, and a missing index on the replica identity columns can stall apply for the affected table.
• •Synchronous commit configuration mismatch: synchronous_standby_names or synchronous_commit settings cause the primary to stall on commit, indirectly inflating apparent receive-side backpressure.
• •Long-running autovacuum, or a stuck replication slot on another consumer, holding back xmin so that tuples cannot be cleaned up and apply slows.
• •I/O starvation on the replica's WAL volume (different filesystem or slower device than data) producing fsync backpressure independent of the data volume.

First ten minutes

1. 01Confirm the topology: identify the primary and the affected replica(s) via pg_stat_replication on the primary, recording client_addr, state, sync_state, and the three LSNs (sent, flush, replay).
2. 02Separate receive from replay: on the replica, compare pg_last_wal_receive_lsn() to pg_last_wal_replay_lsn(); a stable gap is replay lag, a growing receive-side gap indicates primary-to-replica transport or write pressure.
3. 03Classify the mode using process-level evidence: observe the WAL receiver process for sustained disk writes (write pressure) and the startup process for CPU activity and wait events (apply work).
4. 04Inspect wait events on the startup process; any Lock waits, LWLock waits, or DataFileRead waits indicate apply-side contention rather than raw throughput.
5. 05Check pg_replication_slots for active, inactive-but-retained, and aborted slots, and any replication slot's confirmed_flush_lsn relative to the current WAL position.
6. 06Snapshot long-running queries and transactions on the replica that touch hot tables; an open xmin that pre-dates the lag onset is the strongest signal of apply blockage.

Evidence to collect

• •pg_stat_replication rows on the primary with sent_lsn, flush_lsn, replay_lsn, replay_lag, sync_state, and state.
• •pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn(), pg_last_xact_replay_timestamp() on the replica, sampled at fixed intervals to show the gap widening.
• •pg_stat_activity on the replica filtered to backend type "startup" plus any sessions whose xact_start pre-dates the lag onset, capturing wait_event_type and wait_event.
• •pg_replication_slots including active, active_pid, and confirmed_flush_lsn; if WAL retention is growing on the primary, slot lag is implicated.
• •Per-table replica identity declarations on the primary for tables involved in the workload, plus presence of indexes on the replica identity columns on the replica.
• •I/O and fsync latency evidence on the replica's WAL volume, scoped to the WAL receiver's writes versus the startup process's reads.

Where to look

• •Boundary between WAL transport and WAL apply: the gap between pg_last_wal_receive_lsn() and pg_last_wal_replay_lsn() is the only place the two modes separate cleanly.
• •Boundary at the startup process: its wait_event and wait_event_type are the apply-side diagnostic surface; a Lock, LWLock, or DataFileRead wait is apply-side contention.
• •Boundary at WAL persistence on the replica: WAL receiver write activity versus fsync latency on the pg_wal directory is the write-pressure surface.
• •Boundary at replication slots: pg_replication_slots on the primary determines WAL retention; if WAL cannot be removed, receive-side stalls can cascade into primary bloat even when apply is healthy.
• •Boundary at lock holders: any session on the replica whose xact_start pre-dates the lag onset is the candidate apply blocker, especially if its query targets a hot table.

Diagnostic steps

1. 01Sample the three LSNs on the replica at fixed intervals; a stable receive/replay gap localizes the problem to apply, while a widening gap localizes it to transport or write throughput.
2. 02On the replica, query pg_stat_activity where backend_type = 'startup' and record wait_event_type and wait_event; a Lock wait on AccessShareLock or RowExclusiveLock identifies a blocking query by relation.
3. 03Cross-reference any blocking session's xact_start with the lag onset time; a pre-existing transaction that has not advanced is the primary apply blocker candidate.
4. 04On the primary, list pg_replication_slots; a slot whose confirmed_flush_lsn trails current WAL by more than expected indicates that consumer's apply is slow, which can mask replica-apply problems.
5. 05For each table in the replay-path workload, verify on the replica that REPLICA IDENTITY matches the primary and that the replica identity columns are indexed; missing indexes turn apply into a sequential scan.
6. 06Confirm replica-side write pressure is not I/O-starved: sustained high WAL write throughput with elevated fsync latency on pg_wal indicates write pressure even when apply is idle.
7. 07Compare synchronous_commit and synchronous_standby_names on the primary against the deployed topology; a synchronous standby whose apply is slow becomes the system's bottleneck and looks like replica apply lag.

Common mistakes

• •Reading replay_lag in pg_stat_replication as a single number without also tracking receive-vs-replay LSNs, which prevents distinguishing write pressure from apply blockage.
• •Restarting the replica to "fix" apply blockage: a restart closes connections and clears long-running queries but does not address the original lock holder or missing replica identity index, so lag returns.
• •Dropping or advancing a replication slot to recover WAL space when slot lag is itself a symptom of a slow apply consumer; this can break other replicas.
• •Assuming a missing index on the primary is the cause of replica apply lag; replica apply depends on the replica's indexes on replica identity columns, not the primary's.
• •Conflating replica replay lag with logical replication apply lag, which uses a different worker model and a different lag surface (pg_stat_subscription).
• •Increasing wal_compression or full_page_writes as a mitigation; both affect transport volume and primary-side cost, not the replica's apply throughput.

Safe fixes

• •If the startup process is waiting on a lock held by a pre-existing query that is not critical, end that session with pg_terminate_backend(<pid>) where <pid> is obtained from pg_stat_activity on the replica; only do this after confirming the query is non-essential and its xact_start pre-dates the lag onset.
• •If REPLICA IDENTITY is DEFAULT on a table receiving UPDATEs and DELETEs, set REPLICA IDENTITY to a usable indexed column set on the primary and ensure the replica has an index on those columns; this only narrows apply cost when verified with EXPLAIN on the replica.
• •If WAL volume is the bottleneck, reduce per-transaction WAL bytes on the primary (smaller row width, fewer indexed updates, batched writes) and verify with a measured drop in pg_stat_replication.write_lsn delta per second.
• •If a replication slot is the bottleneck because its consumer cannot keep up, add capacity for that consumer first; only consider removing the slot after consumers are healthy.
• •If synchronous_commit is forcing the primary to wait on a slow replica, separate the synchronous peer from the lag-affected replica in synchronous_standby_names so the slow replica is no longer on the synchronous path.
• •If a stuck autovacuum is holding back xmin on the replica, identify and cancel only after confirming via pg_stat_progress_vacuum that progress has stalled; do not disable autovacuum globally.

Prove the fix

1. 01Replay gap closed: pg_last_wal_replay_lsn() advances within seconds of pg_last_wal_receive_lsn() on the affected replica for at least the duration of a representative workload.
2. 02Startup process wait events return to short DataFileRead or walreceiver-related waits with no sustained Lock waits, sampled over multiple intervals.
3. 03Replication slot confirmed_flush_lsn advances at the same pace as WAL generation on the primary, ruling out slot-induced retention pressure.
4. 04Application read-staleness observed during the incident (a known write-then-read round-trip) returns to baseline round-trip latency and value freshness on the affected replica.
5. 05No regression on the primary: write latency, fsync latency, and WAL retention remain within prior bands after any configuration change, verified by the same monitoring sources used before the change.

Prevention and next steps

• •Track both the receive/replay LSN gap and the startup process wait event as first-class signals, with separate alerting thresholds so write pressure and apply blockage are distinguishable in dashboards.
• •Enforce that every table receiving UPDATE/DELETE has an explicit REPLICA IDENTITY with a matching indexed column set on every replica; detect drift between primary and replica catalog views.
• •Cap transaction age on replicas: alert on any session with xact_start older than a defined budget, since pre-existing transactions are the dominant apply-blockage cause.
• •Size WAL volume and fsync capacity on replicas to peak WAL generation rate of the primary, and verify under load rather than assuming parity; track fsync latency on the WAL volume as a distinct signal from data volume latency.
• •Review replication slot ownership and purpose regularly; each slot represents a consumer whose apply health must be monitored independently.

Safe commands and checks

psql -d postgres -c "SELECT client_addr, state, sync_state, sent_lsn, flush_lsn, replay_lsn, replay_lag FROM pg_stat_replication;"
psql -d postgres -c "SELECT pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn(), pg_last_xact_replay_timestamp();"
psql -d postgres -c "SELECT pid, datname, usename, application_name, xact_start, wait_event_type, wait_event FROM pg_stat_activity WHERE backend_type = 'startup' OR xact_start < now() - interval '5 minutes';"
psql -d postgres -c "SELECT slot_name, plugin, active, active_pid, confirmed_flush_lsn FROM pg_replication_slots;"
psql -d postgres -c "SELECT relname, relreplident FROM pg_class WHERE relreplident <> 'f' AND relkind = 'r' ORDER BY relname;"
psql -d postgres -c "SELECT pid, datname, usename, application_name, state, xact_start, query FROM pg_stat_activity WHERE xact_start IS NOT NULL ORDER BY xact_start LIMIT 10;"
psql -d postgres -c "SELECT pg_terminate_backend(<pid>);" with <pid> obtained from the previous SELECT and only after confirming the session is non-essential and its xact_start pre-dates the lag onset.