What this usually means
Logical replication slot lag occurs when the subscriber cannot keep up with the rate of WAL generated on the primary. The slot prevents WAL cleanup, so old WAL segments accumulate until disk fills. Common causes: slow network, under-resourced subscriber, long-running transactions on subscriber blocking apply, or a stuck/down subscriber. The slot holds a restart_lsn that prevents removal of WAL segments the subscriber hasn't confirmed. If the subscriber is disconnected long enough, the primary may delete old WAL segments anyway (due to wal_keep_size or max_slot_wal_keep_size), causing the subscriber to fail with 'requested WAL segment has already been removed'.
The first ten minutes — establish facts before touching code.
- 1Run `SELECT slot_name, slot_type, active, restart_lsn, wal_status FROM pg_replication_slots;` to see slot state and WAL retention status
- 2Check `pg_stat_replication` for each slot: `SELECT application_name, state, write_lag, flush_lag, replay_lag, backend_start, pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS lag_bytes FROM pg_stat_replication;`
- 3Monitor disk usage: `df -h /var/lib/postgresql/data/pg_wal` (or your WAL directory)
- 4Verify subscriber connectivity: on subscriber, run `SELECT count(*) FROM pg_subscription;` and check `pg_stat_subscription` for errors
- 5Use `pg_waldump` to inspect WAL segment range: `pg_waldump -p /var/lib/postgresql/data/pg_wal` (limit to first few segments) to see if there's a pattern of large transactions
The specific files, logs, configs, and dashboards that usually own this bug.
- searchpg_replication_slots (restart_lsn, wal_status, confirmed_flush_lsn)
- searchpg_stat_replication (write_lag, flush_lag, replay_lag, state)
- searchpg_stat_subscription (subscriber side) – last_msg_send_time, last_msg_receipt_time, latest_end_lsn
- searchPostgreSQL logs on primary (typically pg_log/ or /var/log/postgresql/) for 'WAL segment already removed' or 'slot is not active'
- searchOS disk monitoring: dstat, iostat, df – especially for pg_wal directory
- searchpg_current_wal_lsn() and pg_wal_lsn_diff() to compute lag in bytes
Practical causes, not theory. These are the things you will actually find.
- warningSubscriber is under-provisioned: CPU, memory, or I/O cannot keep up with the write rate on primary
- warningNetwork latency or bandwidth limit between primary and subscriber causing increased write_lag
- warningLong-running transaction on subscriber holding the apply process (e.g., blocking triggers or foreign key checks)
- warningLogical replication slot is not consumed because subscriber is down or misconfigured (e.g., connection string wrong, password expired)
- warningwal_keep_size or max_slot_wal_keep_size set too low, causing primary to recycle WAL needed by a lagging slot
- warningBurst of large transactions (e.g., bulk loads) temporarily overwhelms subscriber apply rate
Concrete fix directions. Pick the one that matches your root cause.
- buildIncrease subscriber resources (CPU, memory, disk IOPS) to match primary write rate
- buildTemporarily pause replication on subscriber to catch up: `ALTER SUBSCRIPTION ... DISABLE;` then after lag reduces, `ENABLE`
- buildIf slot is dead (no subscriber), drop it: `SELECT pg_drop_replication_slot('slot_name');` to free WAL
- buildAdjust primary parameters: increase max_slot_wal_keep_size (or wal_keep_size) to tolerate larger lag
- buildOptimize subscriber: set synchronous_commit=off, increase max_worker_processes, tune apply parallelism (apply_work_mem, logical_decoding_work_mem)
- buildRestart subscriber apply worker: `ALTER SUBSCRIPTION ... DISABLE;` then `ENABLE` to reset connection and clear stuck state
A fix you cannot prove is a guess. Close the loop.
- verifiedAfter fix, run `SELECT slot_name, restart_lsn, pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS lag_bytes FROM pg_replication_slots;` and confirm lag decreases
- verifiedCheck `pg_stat_replication` for each subscriber: write_lag, flush_lag, replay_lag should trend toward zero
- verifiedMonitor WAL directory size over 10 minutes: it should stabilize or shrink
- verifiedOn subscriber, `SELECT count(*) FROM pg_subscription WHERE subenabled;` should show subscriptions active and `pg_stat_subscription` has recent last_msg_receipt_time
- verifiedRun a test insert on primary and measure time to appear on subscriber – should be within acceptable RPO
Things that make this bug worse or harder to find.
- warningDropping a replication slot that is still needed – this will break replication permanently and require resync
- warningIncreasing wal_keep_size without monitoring disk space – can accelerate disk fill if slot is already lagging
- warningIgnoring subscriber resource metrics – often the bottleneck is not the slot itself but the subscriber's ability to apply
- warningRestarting primary PostgreSQL to clear WAL – this will cause all subscribers to need resync and may cause data loss
- warningAssuming lag is always due to network – check CPU/I/O on subscriber first
- warningSetting max_slot_wal_keep_size to unlimited – can lead to disk full outages if subscriber fails permanently
Midnight Disk Full Outage Due to Logical Replication Slot Lag
Timeline
- 00:00Primary WAL directory growth alarm triggers (>100GB). On-call engineer alerted.
- 00:05Engineer checks pg_replication_slots: slot 'debezium_slot' shows restart_lsn 30GB behind current WAL insert, wal_status='lost'
- 00:10pg_stat_replication shows no active connection for that slot. Subscriber (Kafka Connect) crashed at 23:45 due to OOM.
- 00:15Debezium connector status shows 'FAILED' with 'java.lang.OutOfMemoryError'. Kafka broker logs indicate consumer group rebalancing.
- 00:20Engineer drops the stuck slot: SELECT pg_drop_replication_slot('debezium_slot'); WAL directory stops growing.
- 00:30After restarting Kafka Connect with increased heap (from 2GB to 4GB), a new slot is created by Debezium. Lag begins to catch up.
- 01:00Replication lag drops to <1 second. Disk space reclaims as WAL segments are recycled.
- 01:30Post-mortem reveals Kafka Connect OOM caused by high-volume topic. Added memory limits and monitoring.
I was the on-call DRE when the disk alarm fired at midnight. The primary's WAL directory had ballooned to 120GB in 15 minutes. My first check was pg_replication_slots. I saw a slot named 'debezium_slot' with restart_lsn far behind and wal_status='lost'. The subscriber column was empty – no active connection. The slot was holding onto old WAL, preventing cleanup.
I quickly checked pg_stat_replication: no rows for that slot. The Debezium Kafka Connect worker had crashed with an OOM error earlier. The slot was orphaned. I dropped it immediately with pg_drop_replication_slot, and the WAL directory growth stopped. Then I worked with the streaming team to restart the Kafka Connect worker with a larger heap (4GB instead of 2GB) and added memory alerts.
The root cause was that the Debezium connector couldn't keep up with a burst of large transactions, causing it to run out of memory and crash. The orphaned slot then caused WAL buildup. The fix was dropping the slot and re-creating it after the connector was stable. We also added monitoring of slot lag and set max_slot_wal_keep_size to limit damage from future orphaned slots.
Root cause
Debezium Kafka Connect worker crashed due to OOM from high-volume topic, leaving the replication slot orphaned and accumulating WAL.
The fix
Drop the orphaned replication slot, increase Kafka Connect heap, restart connector, and monitor slot lag.
The lesson
Always set max_slot_wal_keep_size to bound WAL retention. Monitor slot active status and lag. Ensure consumer processes have adequate resources and crash recovery automation.
Each logical replication slot tracks two LSNs: restart_lsn (oldest WAL the consumer might need) and confirmed_flush_lsn (last WAL the consumer has acknowledged). The primary will not remove any WAL segment that contains data after the minimum restart_lsn across all slots. If a slot becomes inactive (consumer disconnected), restart_lsn remains fixed, and WAL accumulates. The parameter max_slot_wal_keep_size (in PostgreSQL 13+) limits how much WAL a single slot can retain; if exceeded, the slot's wal_status becomes 'lost' and the consumer will need to resync. wal_keep_size is a separate parameter that retains extra WAL for all replicas (physical and logical) but is not slot-specific.
When lag grows, the most important diagnostic query is: SELECT slot_name, active, restart_lsn, pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS lag_bytes, wal_status FROM pg_replication_slots; A lag_bytes value > 0 indicates backlog. wal_status='lost' means the primary has recycled WAL needed by the slot – the consumer must be resynced.
Use pg_stat_replication on the primary to get per-subscriber lag components: write_lag (time between primary write and subscriber receive), flush_lag (time to flush to disk on subscriber), replay_lag (time to apply). If write_lag is high, network is the bottleneck. If flush_lag or replay_lag dominate, the subscriber is slow (I/O or apply processing). On the subscriber, pg_stat_subscription shows the latest received LSN and time. Compare with primary's current LSN to compute total lag.
Practical command: SELECT application_name, state, write_lag, flush_lag, replay_lag, pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS lag_bytes FROM pg_stat_replication WHERE state = 'streaming'; This gives a one-liner view of each subscriber's lag. If a subscriber has been disconnected, it won't appear here; check pg_replication_slots for active=false.
If the slot is still active but slow, first try to increase subscriber performance: scale up CPU/memory, increase max_worker_processes, set synchronous_commit=off on subscriber (if safe), and tune logical decoding parameters (logical_decoding_work_mem for large transactions). On the primary, you can also increase max_slot_wal_keep_size to prevent the slot from going 'lost' while the subscriber catches up.
Another technique is to temporarily pause replication on the subscriber with ALTER SUBSCRIPTION ... DISABLE; This stops the subscriber from receiving new WAL, but the slot remains. The primary will continue to accumulate WAL, but the subscriber can catch up when re-enabled. This is risky if disk space is limited. A better approach is to drop and recreate the subscription if the lag is too high, but that requires a full resync which may be slow for large databases.
Set up alerts based on pg_replication_slots: monitor wal_status for any value other than 'reserved' (or 'extended' if using max_slot_wal_keep_size). Also monitor lag_bytes using a query that compares restart_lsn to pg_current_wal_lsn. For disk space, alert when pg_wal directory exceeds 80% of its partition. Use tools like pg_stat_monitor or Nagios/CheckMK with custom checks.
Example check script: psql -c "SELECT CASE WHEN EXISTS (SELECT 1 FROM pg_replication_slots WHERE wal_status != 'reserved') THEN 1 ELSE 0 END AS slot_issue;" If returns 1, alert. Also compute lag in bytes and compare to a threshold (e.g., 10GB).
For production environments, consider automating the detection and removal of orphaned slots. A script can periodically check slots with active=false for more than N minutes and drop them after alerting. However, this can cause data loss in CDC pipelines, so it must be coordinated with the consumer team.
Better: implement health checks on consumers that auto-restart them. For example, Debezium connectors can be configured with a retry policy and health check endpoint. Use Kubernetes liveness probes or systemd services to restart crashed workers. On the PostgreSQL side, set max_slot_wal_keep_size to a reasonable value (e.g., 50GB) so that a stuck slot doesn't fill the disk; the slot will go 'lost' and the consumer can detect and resync.
Frequently asked questions
What does wal_status='lost' mean in pg_replication_slots?
It means the primary has discarded WAL segments that the slot needs, because the slot's restart_lsn is behind the oldest available WAL. This happens when the consumer is too far behind and the primary's wal_keep_size or max_slot_wal_keep_size limit has been exceeded. To recover, you must drop and recreate the slot, then resync the subscriber from scratch.
How can I clear a stuck replication slot without dropping it?
If the slot is stuck because the subscriber is disconnected, you can only clear it by dropping the slot (pg_drop_replication_slot). There is no built-in way to advance restart_lsn manually. If the subscriber is still connected but slow, let it catch up naturally. To speed up, you can increase subscriber resources or temporarily disable the subscription to allow the subscriber to apply pending WAL before re-enabling.
What's the difference between wal_keep_size and max_slot_wal_keep_size?
wal_keep_size specifies the minimum number of WAL segments to retain for any purpose (physical or logical replicas), independent of slots. max_slot_wal_keep_size (introduced in PG13) limits the total WAL retained by all replication slots combined. If a slot causes WAL retention to exceed max_slot_wal_keep_size, the slot's wal_status becomes 'lost'. Use wal_keep_size as a safety net for all replicas, and max_slot_wal_keep_size to prevent a single slot from consuming too much disk.
Can a long-running transaction on the subscriber cause replication lag?
Yes. The subscriber apply process is single-threaded per subscription (by default). If a long-running transaction on the subscriber (e.g., due to a trigger or foreign key check) blocks the apply worker, it cannot confirm WAL segments, causing lag to grow. Monitor pg_stat_activity on the subscriber for active queries with long duration. You can often fix by setting idle_in_transaction_session_timeout or tuning the subscriber's configuration.
Should I use synchronous replication for logical replication?
Logical replication can be synchronous by setting synchronous_commit = remote_write or remote_apply on the subscriber. This ensures the primary waits for the subscriber to confirm. However, this increases latency and can cause the primary to stall if the subscriber lags. It's only recommended for critical data where zero data loss is required. For most use cases, asynchronous replication is sufficient, with monitoring to detect lag.