Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Proxmox Backup Server verify failed: corrupt or missing chunks and what to do

Technical article
Backup & Recovery

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

A failed verify means that at least one chunk a snapshot needs could not be read, was missing, or did not match its recorded size or checksum. In the project's source the snapshot is then marked as failed and a corrupt chunk file is renamed to <digest>.<n>.bad, with n from 0 to 9. Do not delete anything yet. First find which chunk and snapshots are affected and why the storage returned bad data, then repair from a good copy or take a new backup.

What the error means

Proxmox Backup Server stores data as chunks. A chunk is possibly encrypted data with a CRC-32 checksum at the end, identified by the SHA-256 checksum of its content. Each snapshot has a manifest (index.json) listing its files with sizes and checksums, and index files (.fidx or .didx) that give "an ordered list of references to the chunks that the original file was split into". Chunks are shared between snapshots, so one bad chunk can fail many snapshots.

A verify job reads every chunk a snapshot references and compares it with what the index expects. The documentation states that verification ensures backup data is intact, can be scheduled as verify jobs, and can be run on a whole datastore, a group or a snapshot.

In the public source, the task log lines you may see include:

  • can't verify chunk, load failed - <reason>: the chunk file could not be read.
  • can't verify missing chunk with digest <digest>: used for S3-backed datastores when the object is not there.
  • corrupt chunk renamed to <path>: the bad chunk has been moved aside.
  • chunk <digest> was marked as corrupt: a later snapshot in the same run references a chunk already found bad.
  • verify <datastore>:<snapshot>/<file> failed: <reason> and chunks could not be verified.
  • A summary line ending in (<n> errors) per file.

These strings come from the source, not the user guide. They read the same on the main branch and in tag v4.2.10, the newest tag. The documentation describes 4.2.11-1, which has no tag, so releases can still differ.

Two limits matter. For encrypted chunks the server holds only the checksum of the plaintext, so "only the CRC-32 checksum gets checked" without the key. And a renamed .bad chunk is tracked separately by garbage collection, whose status counts bad chunks removed and bad chunks still present.

Likely causes

Cause What you see How to confirm
Storage returned wrong data (bit rot, failing disk, controller fault) (<n> errors) on a few chunks; load or checksum failures; the same datastore fails again after repair Storage health tools for your disks or pool; the documentation recommends re-verifying monthly because drives degrade over time
Chunk files deleted or lost (manual clean-up, partial copy, unmounted or restored storage) can't verify chunk, load failed with the reason the load returned; a missing file is a likely reason, so expect many chunks at once (this link is reasoning, not a documented rule) List the chunk path <datastore>/.chunks/<first 4 hex digits>/<digest>; check whether the datastore was recently copied, moved or restored
Interrupted write (power loss or hard shutoff) The backups written around the event fail verify Compare the failure time with host events; check the datastore sync-level setting
Ransomware or tampering on the datastore Many snapshots fail at once, including old ones (this pattern is reasoning, not a documented rule) The documentation says encrypted backups no longer match their recorded SHA-256 checksums, so verification fails
Index and chunk encryption mismatch chunk CryptMode ... does not match index CryptMode ... Inspect the chunk with proxmox-backup-debug (below)

The documentation's tuning section says the sync-level default, filesystem, triggers a syncfs after a backup but before the task returns OK, while file runs an fsync on every chunk insertion. It also notes that the underlying device must protect itself against power loss.

Prove the cause before changing anything

Start with checks that only read.

  1. In the web interface, open the datastore's Content tab and look at which snapshots show a failed verification, then open the verify task log. Note the first error, not the last, and write down the chunk digests.
  2. Look at the snapshot's referenced chunks: proxmox-backup-debug inspect file drive-scsi0.img.fidx lists the chunks a .fidx index references. A good result is a list of digests. A bad result is an error reading the index.
  3. Find which index files use a given chunk: proxmox-backup-debug inspect chunk <digest> --reference-filter /path/in/a/datastore/directory. Without a filter it prints only the CRC and encryption status.
  4. Check the file exists and its size looks right, using normal file tools on <datastore>/.chunks/<4 hex digits>/<digest>.
  5. Check the storage layer with your storage tools. Proxmox's administration guide shows zpool status -v for watching a ZFS resilver.

Risk: running the verify command is not read-only. proxmox-backup-manager verify <datastore> --read-threads 1 --verify-threads 4 --ignore-verified false writes the verify result into each snapshot's manifest and, per the source, renames chunks it finds corrupt. Run it deliberately, not as a harmless look. Renaming moves the file aside; it does not delete it.

Fix

If you have a good copy on another Proxmox Backup Server. The documentation describes a sync job option, resync-corrupt, which "will re-sync all snapshots that have failed to verify during the last Verification". Run a verification first, then the sync job. It "needs to check the manifests of all snapshots in a datastore and might take much longer than regular sync jobs". The protected flag of snapshots is not synced, so set it again on the target if you need it.

Risk: the re-sync rewrites the snapshots that failed verification on the target with the copy from the other server. If that copy is also damaged, the target gains nothing, so verify the other server first.

If the source still holds the data. Take a fresh backup. In the source, the server does not use a failed latest snapshot as the starting point for the next one, so the new backup should not depend on the damaged one. The documentation does not state how much data the client re-sends in that case. Watch the transfer and the duration.

If nothing intact remains and you need the readable parts. Try proxmox-backup-debug recover index drive-scsi0.img.fidx /path/to/.chunks. The --skip-crc option "will speed up the process slightly and allow for trying to restore (partially) corrupt chunks". The documentation recommends always trying without --skip-crc first. Restore to a new location, not over live data.

Removing the failed snapshot. Disruptive step: proxmox-backup-client snapshot forget <snapshot> deletes the snapshot. There is no way back from this command. The documentation says "The prune command removes only the backup index files, not the data", so space is freed when garbage collection runs. Protected snapshots are skipped, because the protected flag prevents pruning or manual deletion. Do this only once the affected restore points are replaced.

Garbage collection. Disruptive step: proxmox-backup-client garbage-collect deletes chunks that no snapshot references and that are older than a cutoff of 24 hours and 5 minutes, or the oldest running backup. No way back. Wait until the repair is finished and verified.

If the cause is hardware, fix or replace it before repairing. The same bad storage will return bad data again.

Confirm the fix

  1. Re-verify the affected snapshots, or the datastore with --ignore-verified false so already verified snapshots are checked again. Expect (0 errors) for each file and no new .bad files.
  2. Restore a recent snapshot to a new guest, not over the live one. The documentation recommends testing restores this way.
  3. Check that the next scheduled backup finishes and that its snapshot verifies.

Example (illustrative): backup01 holds vm/100, a 100 GiB disk stored in 4 MiB chunks. 100 × 1,024 ÷ 4 = 25,600 chunks. Verification finds 3 bad chunks. Those 3 chunks are 12 MiB, 0.012% of the 25,600 chunks. If they sit in an unchanged part of the disk, every snapshot of vm/100 since they were first stored references them, so all snapshots fail. A re-sync from the remote copy restores the good chunks. A single verify pass reads each unique chunk once per run, because chunks already verified in the run are skipped, so the first pass costs about one read of the unique data: 100 GiB at 200 MiB/s is 102,400 ÷ 200 = 512 seconds, a little under 9 minutes.

What to watch afterwards

  • Run a verify job that checks new backups hourly or daily, and a second job that re-verifies everything weekly or monthly. The documentation recommends re-verifying at least monthly, even when earlier verification passed.
  • Turn on notifications for verification and garbage collection results.
  • Compare error counts over time. Errors that return after repair point to hardware.
  • Keep a copy of the datastore on another server. The documentation's 3-2-1 guidance recommends off-site copies and describes sync jobs that do not remove snapshots that vanished from the source.

Risks and limits

The documentation does not list the log strings above. They come from the project's public source (main branch and tag v4.2.10) and may change. Verification of encrypted chunks without the key checks only the CRC-32 checksum. Verification tells you the stored data matches its checksums, not that the guest will boot after a restore. Do the restore test.

These commands have not been run or lab-tested.

Sources