Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

duplicity 'found the following orphaned backup file(s)': broken chains, full-if-older-than and restores

Technical article
Backup & Recovery

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

duplicity stores a backup as a chain: one full set, then incremental sets that each carry on from the one before. An orphaned backup file is an incremental set that no chain can accept, for example because the full set or a set in the middle is missing from the storage. Those sets cannot be restored from. The fix is not to hunt for the file name in the message but to read the collection status, find where the chain breaks, and start a fresh full.

The same family of messages includes "No signatures found, switching to full backup." That one is about duplicity's local record of what was backed up, and it costs you a full upload rather than any lost restore points.

What the error means

In duplicity 3.2.2, the code groups the files on the storage into sets and then builds chains. A full set starts a chain. An incremental set is added to a chain only if its start time equals the end time of the chain's last set. If no chain accepts it, duplicity logs an informational line that it is "Ignoring incremental Backupset" with the start time it has and the start time it needed, and the set becomes orphaned.

During normal runs the visible result is a warning headed "WARNING. found the following orphaned backup file(s):" followed by the set names. The collection-status action prints a summary that includes "Also found N backup set(s) not part of any chain" and says these "may be deleted by running duplicity with the "cleanup" command". Incomplete sets, from a run that died, are reported separately as "found missing difftar(s) in backup sets, possibly left from aborted session".

The man page explains why one missing set matters: "When restoring, duplicity applies patches in order, so deleting, for instance, a full backup set may make related incremental backup sets unusable." Both full and incremental sets use the same tar-based format, and an incremental set holds only the changes from the previous set.

Signatures are a separate record. The man page says signature files "are not required to restore a backup set, but without an up-to-date signature, duplicity cannot append an incremental backup to an existing archive". duplicity keeps them locally in the archive directory (default ~/.cache/duplicity/) and, encrypted if encryption is on, remotely. The man page adds that if the archive directory is deleted, duplicity "will have to recreate it from the remote repository".

Likely causes

Cause What you see How to confirm
The full set (or a middle set) was deleted from the storage Orphaned sets after the missing one. "No backup chains with active signatures found" may also appear, but it also appears when only the signature record is lost (see the signatures row) collection-status lists the chains and orphans; compare with a listing of the storage
A storage lifecycle rule or a manual clean-up removed old files The break appears at a date that matches the rule Check the bucket or folder rules and its deletion log for that date
A run was aborted part-way Incomplete sets reported, "missing difftar(s)" collection-status shows "incomplete backup set(s)"
The local signature record was lost or does not match "No signatures found, switching to full backup." on the next backup Check the archive directory and the --name and --archive-dir settings
An incremental was requested with no signatures Fatal error: "Unable to start incremental backup. Old signatures not found and incremental specified" The command used the incremental action instead of plain backup
The chain was never cut Long chains; every restore needs every set collection-status shows one chain with many sets

Prove the cause before changing anything

Run only read-only actions first.

duplicity collection-status sftp://uid@backup01/srv/backup

The man page says this action prints "the chains and sets found, and the number of volumes in each". Use the same target URL and the same --name and --archive-dir options your backup job uses, or you may look at a different archive directory than the one the job uses. Read three things in the output: the primary chain with its matching signature chain; any secondary chains; and the orphaned and incomplete counts. A good result is one primary chain, no orphans and no incomplete sets. A bad result lists orphans, or says no chains with active signatures were found.

To list the files in the latest backup:

duplicity list-current-files sftp://uid@backup01/srv/backup

The man page says this information is "extracted from the signature files, not the archive data itself", so the whole archive is not downloaded and, in the man page's words, "if the archive has been deleted or corrupted, this action will not detect it". It cannot show that the chain can be read. Use verify for that.

For a stronger check, duplicity verify downloads each archive file and checks that it can restore the archive and that the restored file matches the stored signature. The man page says verify "does not actually restore". It reads every file in the chain, so on a large repository it takes time and incurs download charges.

If you have an encrypted chain, these actions need the passphrase. Have it ready before you start.

Check the storage side too. List the full and incremental file names in the target folder (full sets start with duplicity-full, increments with duplicity-inc) and compare dates with the last day the backup was healthy.

Fix

A set is missing. Nothing you do in duplicity brings the file back. If the storage has versioning or a bin, restore the deleted files there first. If not, the restore points after the break are lost. Start a new chain:

duplicity full /srv/data sftp://uid@backup01/srv/backup

The man page says of full: "A new backup chain is started even if signatures are available for an incremental backup." The old chain stays in place until you remove it.

Risk: a full uploads every file again, which costs time, bandwidth and storage. Nothing is deleted, so there is nothing to back out except removing the new chain later with the remove actions. The same applies to --full-if-older-than and --full-if-n-inc below: they add extra full uploads.

Orphaned or incomplete sets you no longer need. The man page says cleanup deletes "the extraneous duplicity files on the given backend", never non-duplicity files or files in complete data sets, and that --force is needed "to delete the files instead of just listing them". Run it without --force first to see the list.

Disruptive step: duplicity cleanup --force <url> deletes the listed files from the storage. There is no way back from duplicity once they are gone, so only run it after you have confirmed that the listed sets are orphans you do not want and that the chain you rely on is complete.

"No signatures found, switching to full backup." duplicity is warning, not failing: no signature chain matched, so it will upload a new full. If the cause was a moved or renamed archive directory, point --archive-dir and --name back at the original and the next backup can stay incremental. If the signatures are truly gone, accept the full.

Fatal error on incremental. Use the plain backup action, which the man page says performs an incremental "if old signatures can be found" and otherwise starts a new chain.

Long chains. Make the software start a new full by itself. --full-if-older-than <time> makes duplicity perform a full "if an incremental backup is requested, but the latest full backup in the collection is older than the given time". The man page example is a full a month, --full-if-older-than 1M, and says a month is always 30 days. --full-if-n-inc <number> does the same after a given number of increments. duplicity logs "Last full backup is too old, forcing full backup" or "Latest chain has too many incrementals, forcing full backup" when it does.

Then thin out old chains. remove-all-but-n-full <count> deletes sets older than the count-th last full ("keep the last count full backups and associated incremental sets"), and remove-older-than <time> leaves sets that newer sets depend on. Both only list files unless --force is given.

Disruptive step: with --force, remove-all-but-n-full and remove-older-than delete backup sets from the storage, and the restore points in them are gone. There is no way back from duplicity, so run each command first without --force, check the listed sets, and keep the chain you rely on.

Example (illustrative): fs01 backs up /srv/data to backup01 every day. The first full is 100 GB and each daily increment is 2 GB. With no full for 90 days, the chain is 1 full + 89 increments = 90 sets, and restoring the latest point reads 100 + 89 × 2 = 278 GB. If the set from day 10 is lost, restore points for days 11 to 89, which is 79 points, cannot be rebuilt from this chain. With --full-if-older-than 1M, a new full is forced after 30 days, so a chain holds about 30 sets (100 + 29 × 2 = 158 GB to read) and the same lost set affects days 11 to 29, 19 points. The price is two extra 100 GB uploads in 90 days (300 GB of fulls instead of 100 GB).

Confirm the fix

  • collection-status shows one primary chain with a matching signature chain and "No orphaned or incomplete backup sets found."
  • The next scheduled backup runs as an incremental and ends without a warning.
  • A restore of a few files, and a restore with --time set to an early point in the chain, both complete into an empty test folder.
  • If you used --full-if-older-than, the notice "Last full backup is too old, forcing full backup" appears once in the log on the day the limit passes, and not on every run.

What to watch afterwards

  • The number of sets in the primary chain. A rising count without a new full means the limit is not working.
  • Storage-side lifecycle rules and delete permissions. A rule that removes old objects will orphan the chain whenever it removes a full that newer sets still need.
  • The archive directory: keep its location stable, include --name if several jobs share a user account, and do not delete it casually.
  • An occasional restore into an empty directory. Only a restore proves the data can be read back.

Risks and limits

  • Messages and behaviour here come from duplicity 3.2.2. Older releases word the orphan warning differently, and other backends handle failed uploads differently.
  • --dry-run calculates what would be done without performing backend actions, but it is no substitute for checking the list printed by cleanup and the remove actions.
  • A full backup uploads every file again. On slow links or metered storage that costs time and money.
  • Shortening chains does not repair damage to a stored file. Verify and restore tests find that.
  • If the passphrase is lost, no part of an encrypted chain can be read.
  • These commands have not been run or lab-tested.

Sources