Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

restic 'repository contains errors' and 'pack ... does not exist': repair the index and snapshots

Technical article
Backup & Recovery

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

Fatal: repository contains errors is how restic check ends when it finds a mismatch between the index (restic's list of which pack file holds which data), the pack files themselves and the snapshots. A line such as pack <ID>: does not exist means the index points at a pack file that is no longer in the repository. Do not start by deleting anything. Copy the repository, find out why files went missing, then repair in the order restic documents: index first, backups again, snapshots last.

Two repair commands remove data that restic cannot bring back: restic repair packs removes damaged packs and keeps only their intact blobs, and restic repair snapshots --forget deletes the original damaged snapshots. restic repair index only rebuilds the index and deletes the old index files. That is why the order and the copy matter.

What the error means

A restic repository stores file data in pack files, keeps an index of what is in each pack, and records each backup as a snapshot that points to trees and data blobs. The restic troubleshooting page says backup and prune "can be interrupted at any point in time without damaging the repository", but that a repository "might be damaged if some of its files are damaged or lost", which "can occur due to hardware failures, accidentally removing files from the repository or bugs in the implementation of restic".

In the v0.19.1 source, check reports problems in five kinds of message, and the text differs for each:

Output Meaning Severity in the source
pack <ID>: does not exist The index lists a pack that is not in the repository Error
pack <ID>: unexpected file size: got N, expected M The pack exists but is the wrong size (for example cut short) Error; the pack is marked for repair packs
pack <ID>: not referenced in any index A pack exists that no index lists Counted as non-critical; shown with -v, with a hint to run prune
error for tree <ID>: followed by id ... not found in repository or file "..." blob ... not found in index A snapshot refers to data that cannot be found Error
The repository index is damaged and must be repaired. You must run 'restic repair index' to correct this. The index itself fails to load or is inconsistent Error

Duplicate packs are non-critical, you can run 'restic repair index' to correct this is a hint, not a failure. The exit status follows the usual restic rules: 0 for success, 1 for any error, and 11 if the repository is already locked.

The troubleshooting page shows only id ... not found in repository under a tree error, and the final Fatal: repository contains errors. The other texts above, including the file "..." blob ... not found in index form (internal/checker/checker.go), come from the source and may be worded differently in your version. If what you see is not in this list, quote it in full when you ask for help.

Likely causes

The restic page names three causes: hardware failures, accidentally removing files from the repository, and bugs. The other rows below are reasoning from the error texts, not restic statements.

Cause What you see How to confirm
Pack files deleted or lost on the storage pack <ID>: does not exist, then tree errors The storage's own log, a clean-up job, a restore from an older storage snapshot, or a quota event
Interrupted or partial copy of the repository unexpected file size on several packs Compare file counts and sizes with the source copy
Hardware or storage fault Ciphertext verification errors, or damage that returns after repair Disk, RAID, memory and controller logs; the restic page says to fix hardware first, otherwise "the repository will soon be damaged again"
Index damaged or out of date "The repository index is damaged" The message itself
A prune or forget running while files were copied or removed outside restic A mixture of missing and orphaned packs Job logs and the times files changed on the storage
Bug or old version The same fault repeats with no hardware cause Upgrade; the page advises the latest restic because it "can contain bugfixes, and improvements to simplify the repair"

Prove the cause before changing anything

These commands come from the restic documentation and manual pages.

restic check
restic check --read-data

check verifies structure and snapshots. The manual says --read-data adds reading "all data blobs", and --read-data-subset=n/t or a percentage reads part of it. The troubleshooting page warns that --read-data "will download the whole repository", so avoid it where downloads cost money, and keep the output: "Keep a copy of the check output as it might be necessary later on!"

By default both commands take an exclusive lock, so they need a quiet window and fail with exit code 11 if a backup holds a lock. In the v0.19.1 source, check --no-lock skips the lock.

Read the output for:

  1. Which of the five message types above appear. Missing packs and tree errors point to lost files. A "ciphertext verification failed" warning is different: the page says to ask the restic forum, because such errors "are often caused by hardware problems which must be investigated and fixed".
  2. Whether check itself prints restic repair packs <IDs> and restic repair snapshots --forget. The source prints them when it finds damaged packs, with a pointer to the troubleshooting page. The page says: "If check detects damaged pack files, it will show instructions on how to repair them using the repair packs command. Use that command instead of the 'Repairing the index' section."
  3. What changed on the storage around the time of the first failure.

Example (illustrative): the repository for fs01 lives in /srv/backup/restic-fs01 on backup01 and holds 1.2 TiB. restic check reports two lines of the form pack <ID>: does not exist and a tree error. A full --read-data at 100 MiB/s would read 1.2 x 1,024 x 1,024 = 1,258,291 MiB, which is 12,583 s or about 3.5 hours. Reading a tenth with --read-data-subset=1/10 takes about 21 minutes. The disk's log shows a file-system error the previous week, so the fault is a hardware problem to fix first. The rate and size are invented for the example.

Fix

Follow the order on the troubleshooting page. Stop at any step where restic check passes.

1. Stop other work and copy the repository. The page says: "Create a full copy of the repository if possible. Or at the very least make a copy of the index and snapshots folders. This will allow you to roll back the repository if the repair procedure fails." It also says to disable regular operations: "Especially, forget or prune must be disabled as they could remove data unexpectedly." If the storage is suspect, fix the hardware before any repair.

2. Repair the index.

restic repair index

The manual says it "creates a new index based on the pack files in the repository". In the page's sample output it reports removing not found pack file <ID>, rebuilds the index and deletes the old index files. Disruptive step: old index files are deleted. Back out by restoring the copy of the index folder from step 1. The page adds: "it is not recommended to repair the index unless the repository is actually damaged". --read-all-packs ("read all pack files to generate new index from scratch") exists, and reads every pack, so it is slow and costly on remote storage.

3. If check named damaged packs, use repair packs instead of step 2.

restic repair packs <ID> [<ID> ...]

The manual says it "extracts intact blobs from the specified pack files, rebuilds the index to remove the damaged pack files and removes the pack files from the repository". In the source it first saves a copy of each named pack into the current folder as pack-<ID>, so run it from a directory with enough free space. Disruptive step: the damaged pack files are removed from the repository; only the intact blobs are kept. restic documents no restore from the pack-<ID> copies it saves; they are raw copies of the damaged packs. The real way back is the repository copy from step 1.

4. Run all backups again. The page says that with a correct index "the backup command guarantees that newly created snapshots can be restored successfully. It can also heal older snapshots, if the missing data is also contained in the new snapshot". If the original files still exist, this may restore the missing data without any data loss. Then run restic check again.

5. Remove what is still missing from snapshots. See what would change first:

restic repair snapshots --dry-run

Look for would save new snapshot messages. Then:

restic repair snapshots --forget

Disruptive step: the manual warns "Repairing and deleting broken snapshots causes data loss! It will remove broken directories and modify broken files in the modified snapshots." With --forget, restic removes the original damaged snapshots after saving repaired ones. There is no way back except the copy from step 1. The manual adds that if the content is still available, "the better option is to run 'backup'", and to use repair snapshots only to recover "an old and broken snapshot". Without --forget, you must remove the old snapshots yourself with restic forget <ID>. Disruptive step: restic forget <ID> deletes that snapshot record. There is no way back except the copy from step 1.

Confirm the fix

  1. restic check --read-data ends with no errors were found (the troubleshooting page uses this text in its final check). A subset read is a weaker proof; say so in your record.
  2. Restore the newest snapshot of each backed-up path to a scratch directory and compare it with the source. Then restore a file from the oldest snapshot you still need. A repaired snapshot can be valid yet missing files.
  3. Compare the list of snapshots before and after. Anything that repair snapshots replaced is a snapshot with less data than it had.

What to watch afterwards

  • Schedule restic check regularly in a window with no backup, and keep the output.
  • Alert on repository contains errors, not only on a failed backup.
  • Keep a second copy of the repository somewhere that does not share the failed hardware. Repair on a single copy depends on luck.
  • If the same damage returns, the page says to open an issue with the restic project and include the check output.

Risks and limits

  • These commands have not been run or lab-tested. They come from the restic documentation, manual pages and v0.19.1 source.
  • The restic documentation says it does not cover all kinds of damage and that you should ask for help if the steps do not work or you are unsure.
  • The message wording comes from the restic 0.19.1 source; other versions may differ.
  • Repair cannot recreate data that no longer exists anywhere. It can only make the remaining repository consistent, and each snapshot may end with fewer files.
  • A repository on a remote service may have its own rules (such as write-once storage) that block deletion or replacement of files, so some repair steps may not be possible there.

Sources