Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

PostgreSQL replica: 'requested WAL segment has already been removed'

Technical article
Troubleshooting

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

The standby fell behind far enough that the primary recycled a write-ahead log (WAL) segment the standby still needed. Nothing on the primary was holding that segment: no replication slot, a wal_keep_size too small to cover the gap, and no archive the standby could read from. If a WAL archive exists and the standby can read it, the standby can catch up from there. If not, the PostgreSQL documentation says the standby must be reinitialised from a new base backup.

What the error means

In the REL_18_STABLE source, the message requested WAL segment %s has already been removed is raised at ERROR level in WalSndSegmentOpen in walsender.c, the callback that opens a WAL segment for a sending process. It fires when opening the file fails because it does not exist. It is therefore generated on the primary, so look for it in the primary's server log. The same code serves physical and logical replication. This article covers the physical (streaming standby) case.

The documentation explains the mechanism: if you use streaming replication without file-based continuous archiving, the primary might recycle old segments before the standby has received them. There are three ways to stop that: set wal_keep_size large enough, use a replication slot, or provide an archive the standby can read.

wal_keep_size is the minimum amount of past WAL kept in pg_wal for standbys. Its default is 0, and without a unit it is read as megabytes. At zero, "the system doesn't keep any extra segments for standby purposes", so what remains depends on the previous checkpoint and archiving status.

Likely causes

Cause What you see How to confirm
Standby was offline or disconnected longer than the primary retained WAL Error appears after reconnect following a maintenance window, network fault or crash Compare the standby's last replay time with the primary's oldest segment in pg_wal
No slot, wal_keep_size left at default 0 The standby never configured primary_slot_name; the primary shows no slot for it SHOW wal_keep_size; on the primary; SELECT slot_name, active, restart_lsn, wal_status FROM pg_replication_slots; returns no row for this standby
Slot exists but its WAL was removed because max_slot_wal_keep_size was exceeded Slot wal_status is lost The pg_replication_slots view reports lost ("this slot is no longer usable")
restore_command missing or failing on the standby The standby never reads the archive even though one exists restore_command is empty in the standby's configuration, or the archive path is unreachable
Heavy write bursts (bulk load, large rebuild) outran the retention window The standby was connected but lagging pg_stat_replication on the primary shows sent_lsn far ahead of replay_lsn before the failure

Prove the cause before changing anything

All checks below are read-only. The commands come from the documentation and have not been run.

On the primary:

  1. SHOW wal_keep_size; and SHOW max_slot_wal_keep_size;. The defaults are 0 and -1 (unlimited).
  2. SELECT slot_name, active, restart_lsn, wal_status, safe_wal_size FROM pg_replication_slots; A wal_status of lost means the slot is unusable. safe_wal_size is NULL for lost slots and when max_slot_wal_keep_size is -1.
  3. SELECT state, sent_lsn, write_lsn, flush_lsn, replay_lsn FROM pg_stat_replication; Any row means a sender is currently connected. state is one of startup, catchup, streaming, backup or stopping. No row for the standby means it is not connected.
  4. To express lag in bytes, use pg_wal_lsn_diff(sent_lsn, replay_lsn), which the documentation describes as calculating the byte difference between two WAL locations and as usable with pg_stat_replication.

On the standby:

  1. SELECT status, last_msg_receipt_time, latest_end_lsn FROM pg_stat_wal_receiver; (see the statistics views).
  2. Check whether restore_command is set, and whether the archive location is readable by the postgres user.

If the archive holds the needed segments, the standby may recover on its own once restore_command works. If the missing segment is in neither the primary's pg_wal nor the archive, only a rebuild will fix it.

Example (illustrative; names and values are invented): on PGPRIMARY01 the standby PGSTANDBY01 was offline for 8 hours of patching. The primary writes about 3 GB of WAL per hour, so roughly 24 GB passed. With wal_keep_size = 0 and no slot, the primary had recycled the segments by the time the standby returned. The primary's log shows the error and pg_stat_replication has no row for the standby.

Fix

Option 1: let the standby read the archive

If a WAL archive exists, configure restore_command on the standby. The documentation's example is restore_command = 'cp /path/to/archive/%f %p', and it notes the command should return immediately if the file does not exist. The standby needs standby.signal in its data directory to run in standby mode. The documentation states a standby with an archive "can always use the archive to catch up provided it retains enough segments". If the archive no longer has the segment either, go to option 2.

Option 2: rebuild the standby with pg_basebackup

Destructive for the old standby copy: the target directory must be empty, so confirm you are rebuilding the standby and not the primary. Do not run this against the primary's data directory. Stop the standby and move its old data directory aside (do not delete it until the new standby is confirmed).

The pg_basebackup page documents these options:

  • -D directory target directory; created if missing, must be empty if it exists.
  • -R creates standby.signal and appends connection settings to postgresql.auto.conf, including the slot name if one is used.
  • -X stream streams WAL during the backup (the default method).
  • -S slotname with -C creates the named slot before the backup starts; -C raises an error if the slot already exists.

A command built from those options:

pg_basebackup -h PGPRIMARY01 -U replicator -D /path/to/new_standby_dir -R -X stream -C -S standby01_slot -P

With -S, the primary keeps WAL for the new standby in the gap between the end of the backup and the start of streaming. The documentation states that the standby should then use the same slot name as primary_slot_name; -R records it. The -X fetch method is less safe here: the documentation says the backup fails and is unusable if the required WAL is recycled before the end.

If the old standby's slot is lost, it cannot be reused. Remove it and create a new one. On the primary, once you have confirmed the slot belongs to the old standby, SELECT pg_drop_replication_slot('slot_name'); drops it; the documentation describes the function as dropping "the physical or logical replication slot named slot_name". Dropping is destructive for that slot. The -C -S options above then create a new one.

Prevent recurrence: choose retention on purpose

  • Replication slot. The slot section says slots stop the primary removing WAL until all standbys have received it, and retain only what is known to be needed. The caution is that a slot "can cause the server to retain so many WAL segments that they fill up the space allocated for pg_wal". A disconnected standby with a slot keeps the primary's WAL growing.
  • max_slot_wal_keep_size. This caps what slots may retain. The default is -1, unlimited. Setting a limit protects the primary's disk but brings the original error back for a standby that falls behind further than the limit.
  • wal_keep_size. In the example above, 24 GB of WAL in 8 hours (3 GB x 8) means a value of at least 24576 (megabytes) if the standby should survive an 8-hour outage. This is a minimum, not a guarantee.
  • Archive. An archive the standby can read removes the dependence on the primary's retention, at the cost of running and monitoring archiving.

Which to choose is a trade-off between primary disk risk and standby survivability. The documentation does not pick one for you.

Confirm the fix

  • pg_stat_replication on the primary shows the standby with state of streaming, and pg_wal_lsn_diff(sent_lsn, replay_lsn) stays small and stable.
  • pg_stat_wal_receiver on the standby reports an active receiver with a recent last_msg_receipt_time.
  • If you use a slot, pg_replication_slots shows active as true and wal_status as reserved.
  • The proof that matters for a failover is a controlled test: promote a spare copy or rehearse a switchover in a non-production environment and confirm the application works. A standby that streams is not yet a standby you have tested.

What to watch afterwards

  • Lag in bytes and in time, alerting well before the retention window you chose.
  • Slot health: alert when wal_status is anything other than reserved, and on safe_wal_size shrinking when max_slot_wal_keep_size is set.
  • pg_wal disk use on the primary, particularly with slots and -1.
  • Standby disconnections. Each is the start of the clock on this error.
  • Recurrence signal: the same message after the standby is rebuilt means the retention choice does not cover your longest outage.

Risks and limits

  • Commands are taken from the PostgreSQL 18 documentation and source and have not been run or lab-tested.
  • Defaults and column names are for 18. Earlier releases may differ.
  • The message is quoted from the 18 source.
  • Rebuilding gives you a standby as of backup time plus streaming; it cannot recover data the primary lost.
  • Slots and max_slot_wal_keep_size are a direct trade-off; no setting removes both risks.

Sources

  • 26.2 Log-Shipping Standby Servers — PostgreSQL Global Development Group, PostgreSQL 18.7 documentation, date not shown, accessed 16 November 2026
  • 19.6 Replication — PostgreSQL Global Development Group, PostgreSQL 18.7 documentation, date not shown, accessed 16 November 2026
  • 53.20 pg_replication_slots — PostgreSQL Global Development Group, PostgreSQL 18.7 documentation, date not shown, accessed 16 November 2026
  • pg_basebackup — PostgreSQL Global Development Group, PostgreSQL 18.7 documentation, date not shown, accessed 16 November 2026
  • 27.2 The Cumulative Statistics System — PostgreSQL Global Development Group, PostgreSQL 18.7 documentation, date not shown, accessed 16 November 2026
  • 9.28 System Administration Functions — PostgreSQL Global Development Group, PostgreSQL 18.7 documentation, date not shown, accessed 16 November 2026
  • walsender.c, REL_18_STABLE — PostgreSQL source repository, source file, accessed 16 November 2026