Australian cloud for business software — built and run in Sydney since 2003

+61 2 8282 5656
support@netleverage.com
Talk to an engineer
Talk to an engineer
NetLeverage
NetLeverage

PostgreSQL physical backups: pg_basebackup, incremental backups and pg_combinebackup

Technical article
Backup & Recovery

Originally published:

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

PostgreSQL has two families of backup. Logical backups, made with pg_dump, export the contents of a database as SQL or an archive that can be loaded elsewhere. Physical backups copy the files of the whole database cluster, the server's data directory, and are restored by putting those files back and letting the server recover.

Physical backups are the basis for point-in-time recovery and for building standby servers. The standard tool is pg_basebackup. Since PostgreSQL 17, it can also take incremental backups that copy only changed blocks, with a companion tool, pg_combinebackup, that rebuilds a full backup from a chain of them.

This guide explains how physical base backups work, what incremental backups add and require, and how to check that a backup is fit to restore.

Why it matters

For a large database, a full physical copy every night can take hours and consume storage quickly. Incremental backups reduce both. The trade-off is that a restore now depends on several backups instead of one, and on rules that are easy to break quietly: a summary setting turned off, an old full backup deleted by a retention script, or a chain restored in the wrong order.

The continuous archiving chapter is direct about who is responsible for the chain: "PostgreSQL has no built-in mechanism to figure out which backups are still needed as a basis for restoring later incremental backups. You must keep track of the relationships between your full and incremental backups on your own, and be certain not to remove earlier backups if they might be needed when restoring later incremental backups."

How it works

What a base backup is

The pg_basebackup documentation says the tool "can take a full or incremental base backup of the database. When used to take a full backup, it makes an exact copy of the database cluster's files." It handles backup mode itself: "pg_basebackup makes sure the server is put into and out of backup mode automatically."

Two limits shape how it fits a backup plan:

  • "Backups are always taken of the entire database cluster; it is not possible to back up individual databases or database objects. For selective backups, another tool such as pg_dump must be used."
  • It "works with servers of the same or older major version, down to 9.1", with feature limits for older servers; "incremental backup (--incremental) only works with server version 17 and later."

A base backup is taken while the database keeps running, so its files are not consistent at a single moment by themselves. They become consistent when the server replays the write-ahead log (WAL), PostgreSQL's record of every change, from the backup period. That is why the WAL matters as much as the files.

Including the WAL

The -X / --wal-method option decides whether the WAL needed to make the backup consistent travels with it. The documentation says that, unless the method none is specified, "it is possible to start a postmaster in the target directory without the need to consult the WAL archive, thus making the output a completely standalone backup."

  • fetch: WAL files "are collected at the end of the backup." The server's wal_keep_size must be high enough, and "if the required log data has been recycled before it's time to transfer it, the backup will fail and be unusable."
  • stream: WAL is streamed "while the backup is being taken" over a second connection, so it "will require two replication connections not just one."
  • none: the backup relies on the WAL archive to supply what it needs.

A self-contained backup is easier to restore in an emergency. A backup that depends on the archive is only as good as the archive.

Two related options are worth knowing. -S / --slot uses a named replication slot during streaming; the documentation notes that if the backup will become a standby using a slot, "the standby should then use the same replication slot name as primary_slot_name". Checkpoint mode (-c fast or the default spread) decides whether the backup starts at once with an immediate checkpoint or waits for a gentler one. A fast checkpoint adds a burst of I/O at the start; a spread checkpoint means the backup may appear to wait before copying anything.

Tablespaces and format

In plain format, the main data directory goes to the target directory, but "all other tablespaces will be placed in the same absolute path as they have on the source server", unless --tablespace-mapping changes that. In tar format, "the main data directory's contents will be written to a file named base.tar, and each other tablespace will be written to a separate tar file named after that tablespace's OID."

The practical consequence: a backup taken in plain format on the same machine as the server can collide with the server's own tablespace paths. Plan tablespace mapping, or use tar format, before the first backup rather than discovering it during one.

The manifest and checksums

By default the server generates a backup manifest, which the documentation describes as "a list of every file present in the backup with the exception of any WAL files that may be included", with sizes, modification times and optional checksums. The --no-manifest option disables it. Leave it on: incremental backups and verification both depend on it.

If data checksums are enabled on the server, pg_basebackup checks them while copying: "By default, checksums are verified and checksum failures will result in a non-zero exit status. However, the base backup will not be removed in such a case". A backup job that ignores the exit status may keep a backup that already contains damaged pages.

Incremental backups

The continuous archiving chapter explains: "You can use pg_basebackup to take an incremental backup by specifying the --incremental option. You must supply, as an argument to --incremental, the backup manifest to an earlier backup from the same server."

In the resulting backup, "non-relation files will be included in their entirety, but some relation files may be replaced by smaller incremental files which contain only the blocks which have been changed since the earlier backup". So the saving is greatest for large tables with a small share of changed blocks.

To know which blocks changed, "the server uses WAL summaries, which are stored in the data directory, inside the directory pg_wal/summaries. If the required summary files are not present, an attempt to take an incremental backup will fail." The summaries "must cover all LSNs from the start LSN of the prior backup to the start LSN of the current backup."

Two settings control this, per the WAL configuration documentation:

  • summarize_wal "enables the WAL summarizer process". "The default is off," and the server "cannot be started with summarize_wal=on if wal_level is set to minimal."
  • wal_summary_keep_time sets when old summaries are removed. The documentation advises: "Typically, you should set this comfortably higher than the time that could pass between a backup and a later incremental backup that depends on it."

A common failure mode follows directly: if summaries are removed before the next incremental runs, for example after a long gap or a paused schedule, that incremental fails. If the required history has expired, a new full backup establishes a fresh starting point. The schedule must surface the failed incremental and confirm a valid replacement, rather than leave a silent gap.

The incremental backup is not usable on its own. The pg_basebackup page states: "An incremental backup cannot be used directly; instead, pg_combinebackup must first be used to combine it with the previous backups upon which it depends."

pg_combinebackup

The pg_combinebackup documentation says it is "used to reconstruct a synthetic full backup from an incremental backup and the earlier backups upon which it depends. Specify all of the required backups on the command line from oldest to newest."

Its checks are deliberately limited. It "will attempt to verify that the backups you specify form a legal backup chain", but "it is not designed to help you keep track of which backups depend on which other backups. If you remove one or more of the previous backups upon which your incremental backup relies, you will not be able to restore it." It also "only attempts to verify that the backups have the correct relationship to each other, not that each individual backup is intact; for that, use pg_verifybackup."

A useful property: "Since the output of pg_combinebackup is a synthetic full backup, it can be used as an input to a future invocation of pg_combinebackup." Periodically combining a chain into a new synthetic full backup shortens the chain a future restore depends on.

Options and limits to note:

  • --link uses hard links instead of copies. The documentation warns that "any modifications to that directory (for example, starting the server) can also affect the input directories", and that the option "is best used when the input directories are only copies that will be removed". Using it on your only copy of the backups risks changing them.
  • --clone and --copy-file-range can share or optimise copying on supporting filesystems.
  • Checksums: "pg_combinebackup does not recompute page checksums when writing the output directory." If checksums were enabled partway through a chain, the result "may contain pages with invalid checksums", and the documentation recommends "taking a new full backup after changing the checksum state of the cluster".

Incremental backups also inherit every requirement of full ones: "you still need all of the WAL segment files generated during and after the file system backup, and any relevant WAL history files. And you still need to create a recovery.signal (or standby.signal) and perform recovery".

Verifying backups

The pg_verifybackup documentation describes checking "the integrity of a database cluster backup taken using pg_basebackup against a backup_manifest generated by the server at the time of the backup." It works on plain and tar formats, but "WAL verification is supported only for plain-format backups", so tar backups need --no-parse-wal.

It is explicit about its limits: "the validation which is performed by pg_verifybackup does not and cannot include every check which will be performed by a running server when attempting to make use of the backup. Even if you use this tool, you should still perform test restores and verify that the resulting databases work as expected and that they appear to contain the correct data."

What to choose and why

These are our recommendations, based on the documentation above:

  • Keep a physical and a logical layer. Base backups restore the whole cluster quickly and support point-in-time recovery; pg_dump exports let you recover a single database or table and move across major versions.
  • Use incremental backups when the database is large and most of it is unchanged between backups. For smaller clusters, a full base backup each time is simpler to keep and restore.
  • If you use incrementals: turn on summarize_wal, set wal_summary_keep_time well above your longest gap between backups, and alert on a failed incremental.
  • Record each chain explicitly: which full backup and which incrementals depend on each other, in a catalogue that the retention process reads before deleting anything.
  • Keep chains short. Take a new full backup on a regular cycle, or combine chains into synthetic full backups, so a restore needs a handful of backups, not dozens.
  • Verify every backup with pg_verifybackup, and check the exit status of the backup itself.
  • Test restores onto a separate server or directory on a schedule, combining a real chain, replaying WAL and checking the data.

What the recovery environment must provide

The continuous archiving chapter supplies the supported recovery procedure. Preserve the original data and available WAL before selecting a restore target: unarchived WAL may still be present on the failed system. The replacement needs compatible PostgreSQL binaries, tablespace destinations, required extensions, permissions and the chosen recovery configuration. Configuration files outside the data directory require their own backup, and WAL replay does not restore later edits to configuration files. Incremental chains must first be reconstructed into a full backup before recovery can use their contents.

Our recommendation is to rehearse this on a separate server rather than over the original data directory, so a mistake costs a rehearsal rather than the database.

Risks and limits

  • Deleting the base of a chain makes every later incremental unrestorable. Retention must be chain-aware.
  • Missing WAL summaries can prevent a new incremental. Missing WAL needed to make a base backup consistent can prevent recovery altogether; a later WAL gap limits how far recovery can proceed.
  • --link can alter the source backups if the output is started.
  • Checksum state changes within a chain need a new full backup.
  • Version limits. Incremental backup needs PostgreSQL 17 or later on the server. A client tool accepting an older server is not permission to restore physical files into a different major server version. Plan compatible server binaries, architecture and extensions; major-version migration uses a supported upgrade or logical migration method.
  • Nothing here was run. Options are quoted from the documentation for planning.

Sources

This article was originally published on .