Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Which restore point should a restore test use in an incremental chain?

Technical article
Backup & Recovery

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

Do not test only the newest one. The newest point proves that the newest chain can be rebuilt, which is a good start, but it never reads data that only older points hold, such as a file deleted last week. A sensible rotation tests the newest point, a point in the middle of a chain, the oldest point you promise to keep, and the first point after a new full backup or a merge. Vary the pick each cycle and keep a record of which points you have covered.

What does a restore from an incremental chain actually read?

In a forward incremental chain, each restore point is rebuilt from a full backup plus every incremental after it up to the chosen point. Veeam states it directly: "you need to have a full backup file and a chain of subsequent incremental backup files on disk", and if you delete an incremental before the point you want, you cannot restore because later incrementals depend on earlier ones.

That makes the newest point a strong test of the chain's links, because the files that hold blocks of the newest state must all be readable. Veeam's health check page describes where the data lives: "Data blocks that are required to 'compose' the VM latest state are typically spread out across several backup files in the backup chain". So the test reads several files, and from each file only the blocks the newest state needs. The file counts in the example below are a simple model: it counts every file from the full to the newest increment, although a real restore may need blocks from only some of them.

Blocks that belong only to a middle point are not needed. If a document existed on Tuesday and was deleted on Wednesday, its data lives in older files and is never read when you restore Friday. A corrupted block there passes a Friday test and fails a request for Tuesday's version.

How do the common checks differ in which point they use?

  • Veeam health check verifies only the latest restore point. It does a CRC for metadata and a hash check for data blocks, by default at 05:00 on the last Saturday of the month. It is a check of stored data, not a test of a booted machine.
  • Veeam SureBackup can verify the latest restore point or a specific one. In the Restore Point window you give an approximate date, and Veeam picks "the most recent restore point prior to this time". Its recovery verification starts machines from the backup in an isolated virtual lab and runs tests, and its optional backup file validation recalculates checksums for the backup file.
  • Bareos restore offers "Select the most recent backup for a client" and "Select backup for a client before a specified time". For a chosen time, Bareos selects "the most recent Full backup and all subsequent incremental and differential backups that started before the specified date and time". Neither option selects a job that failed.
  • ASD Essential Eight states for backups that "[r]estoration of data, applications and settings from backups to a common point in time is tested as part of disaster recovery exercises". It does not say which restore point to use, so the choice is yours.

Which restore points are worth testing?

Restore point What it exercises What it misses
Newest The latest full and every increment after it; the point a recent failure would need Blocks that only older points hold
Mid-chain Part of the chain, and files from earlier days Later increments
A full backup itself Whether the full file can be read alone Increment links
Oldest point you promise to keep The retention promise; the oldest chain Recent changes
First point after a new full or a merge The changed structure after a retention step Long chains

On a forever forward incremental chain, Veeam's retention step injects the first incremental into the full backup file and removes that incremental, so the full file is rewritten in normal operation. The first point after that step has a different structure from the one before, which makes it a good candidate for a test.

What does a rotation look like in numbers?

Example (illustrative): fs01 is backed up nightly to backup01 with a full on each Sunday and incrementals on the other days. Number the nights 1 to 28, with fulls on nights 1, 8, 15 and 22. A restore of night N needs the latest full at or before N and each increment up to N.

Test Night Files read Count
Newest 28 full 22, incrementals 23 to 28 7
Full only 22 full 22 1
Mid-chain 18 full 15, incrementals 16 to 18 4
Oldest kept 2 full 1, incremental 2 2

The four tests read 7 + 1 + 4 + 2 = 14 files in total. Night 22 is inside the files already read for night 28, so the unique set is 7 + 4 + 2 = 13 of the 28 files. The newest point alone touches 7 of 28, which is 25%. The rotation raises that to 13 of 28, about 46%, and at the file level only. Within each file, the blocks you read still depend on what each chosen night needs, so repeat the rotation with different nights.

How do you run a restore test safely?

  1. Choose the point from your rotation, and write down the date and the reason.
  2. Restore to a new location. Proxmox Backup Server's documentation advises restoring "to a new guest and not to overwrite your current guest", and recommends restoring random samples when automation is not feasible.
  3. Risk: do not restore over live data or put a test machine on the production network. Veeam's SureBackup starts machines in an isolated virtual lab. If you restore by hand, isolate the restored system from production.
  4. Open real data: start the application, read a file written that day, and compare a known checksum if you have one.
  5. Record the restore point, the start and end times, any errors, and who checked the result.
  6. Disruptive step: delete the test copy only after the result is recorded. Deleting it cannot be undone, but it is a copy and the backup itself is untouched.

How often, and what should a failed test change?

The right rhythm depends on how much the data matters and how long a test takes. A reasonable pattern is a small, quick test of the newest point at a steady interval, and a slower test of an older point on a longer cycle. The related pieces on small frequent checks and on the annual exercise explain how the two fit together. What matters here is only the choice of point: each cycle should use a different one, and the record should show which points have been covered and which have not.

When a test of an older point fails but the newest passes, treat that as information about the chain, not as a one-off. Three questions help:

  1. Does the failure follow the point, or the full backup beneath it? Try the full on its own and the next point up the chain.
  2. Is the failure in the stored data or in the restore environment? A checksum failure points at the data. A boot or permission failure on restored files points at the environment.
  3. Which points could have the same fault? In a forward chain, every point after the faulty one is affected. See the piece on corrupted blocks for how far damage reaches.

If the newest point fails, restore the next older point to learn how much recent data you can still recover, and check the storage before the next backup runs.

How do you check this on your own system or with your provider?

On Veeam, look at the backup job's health check setting and the SureBackup job. In SureBackup, the Start action lets you pick the latest point or an approximate date.

On Bareos, the restore command in bconsole lists the options above. Pick the "before a specified time" option for an older point.

On Proxmox Backup Server, restore a snapshot to a new guest.

If a provider runs this for you, ask in writing:

  1. Which restore points do you test, and how are they chosen?
  2. How often, and from which part of the chain?
  3. Is the test a restore of real data, or a checksum check?
  4. Does any test cover the oldest retained point and the point after each merge?
  5. Where is the record, and can I see the last one?

Risks and limits

The file counts in the example come from a simple model of fulls and incrementals. Real products may read fewer files, because blocks sit in a few of them, or more, because of merges. The Veeam, Bareos and Proxmox pages describe their own products and versions, and the ASD page sets no rule on the choice of restore point. A passing restore test shows that the tested point can be restored on the day. It does not guarantee other points, so keep rotating.

These commands have not been run or lab-tested.

NetLeverage includes a daily full disaster-recovery backup of each hosted server at no extra charge. Talk to an engineer.

Sources