Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Bacula and Bareos Verify jobs: what VolumeToCatalog, DiskToCatalog and Data actually check

Technical article
Backup & Recovery

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

A Verify job is a green tick that means different things depending on its Level. VolumeToCatalog reads the file attribute data written to the backup volume and compares it with the catalogue, but both Bareos and Bacula state that the file data itself is not compared. DiskToCatalog compares the files as they are on the client's disk today with the catalogue, so it tests the source, not the backup. The Data level, which the Bacula manual documents and the Bareos 25 documentation does not describe, reads data back from volumes and checks size and checksum. None of the three is a restore.

Which Verify levels exist?

A Verify job is a job whose Type is Verify. Its Level selects what is compared with what. The Bareos 25 Director documentation lists four Verify levels: InitCatalog, Catalog, VolumeToCatalog and DiskToCatalog. The Bacula 15.0.x manual lists those four and adds a fifth, Data. The Bareos 25 source also defines a Data level (see the Data section below).

InitCatalog and Catalog are a "tripwire" feature. InitCatalog scans a FileSet and stores the attributes in the catalogue without saving any file data. Catalog later compares the current files with that stored state. The documentation describes them as a way to detect system intrusion on files that should not change, such as /sbin, /boot, /lib and /bin. They say nothing about whether a backup can be read, so this page concentrates on the other three.

Bareos describes Verify jobs in general as comparing "the contents of the catalog to the file system, or to what was backed up". It also says a verify can check that a tape that was written can be read. That sentence is about the purpose of Verify jobs. How much of that purpose each level covers is the reasoning in the sections below, not a statement in the manuals.

What does VolumeToCatalog check?

Both manuals say the level makes the system read the file attribute data written to the volume from the last backup of the job named in the Verify Job directive (Bareos treats Verify Job as an alias of JobToVerify). That attribute data is compared with the values saved in the catalogue, and differences are reported. The manuals add that the attribute data includes the signatures (MD5 or SHA1), but "the actual file data is not compared (it is not in the catalog)".

Bareos adds a limitation in plain words: when a VolumeToCatalog job runs, the file data is not checksummed and compared with the recorded checksum. File data errors introduced between the checksumming in the File Daemon and the checksumming of the block by the Storage Daemon are therefore not detected.

Taken together, the manuals' descriptions suggest this level answers one question: does what the volume says about each file still match what the catalogue says? That is a reasoned consequence, not a manual statement. What the manuals do state is that the file data itself is not compared, so the level does not tell you that the bytes of a document will come back intact.

Three operational details come from the manuals:

  • The job needs a client to extract the metadata, but it does not have to be the original client. Bareos suggests using the client on the backup server for maximum performance.
  • If you run no Verify Job directive, the last job that ran is the one compared, so the verify must follow the backup immediately. Naming the job lets you run all backups first and then verify the ones you choose "so that the tape just written is re-read".
  • Risk: VolumeToCatalog modifies the catalogue while it runs. Both manuals warn that two VolumeToCatalog jobs on the same client at the same time give incorrect results. Do not schedule two at once for one client.

What does DiskToCatalog check?

DiskToCatalog reads the files as they currently are on the client's disk and compares their attributes with what the catalogue saved from the last backup of the named job. The manuals say it differs from VolumeToCatalog because it compares against a previous backup rather than a previous verify job. You must supply verify options on the Include statements, and those options decide which attribute fields are compared.

Bareos lists the option letters, including i (inodes), p (permission bits), n (number of links), s (size), m (modification time) and 1 (the signature). It calls pins1 "a useful set of general options" for the Catalog and DiskToCatalog levels.

The documented use is diagnosing the disk: it "can be very useful if you have disk problems because it will compare the current state of your disk against the last successful backup". Two limits follow from the manuals:

  • It compares the source, not the stored copy. A clean DiskToCatalog result says the live files match the catalogue. It says nothing about whether the volume holds them.
  • The current implementation "does not identify files that have been deleted". Both manuals say so.

On a busy file server a DiskToCatalog run also reports every file that legitimately changed since the backup. Expect differences and judge them against the backup time.

What does the Data level check, and does Bareos have it?

The Bacula 15.0.x manual describes Data as reading back "the data stored on volumes" and checking "data attributes such as size and the checksum of all the files". That is the level closest to "was the file stored correctly". The manual notes three things:

  • You can name the job to check with the jobid parameter of the run command.
  • The Verify job must name the correct Storage resource, which can be changed in bconsole.
  • Data with accurate=yes can replace VolumeToCatalog, because the accurate option also checks catalogue records.

The manual's example is run job=VerifyData level=Data jobid=10 accurate=yes, where VerifyData is the example's job name and 10 its job ID.

The Bareos 25 Director documentation does not describe a Data level. The Bareos source on the bareos-25 branch does define one: dird_conf.cc lists Data among the Verify levels, job_level.h describes it as "Verify data on volume", and the verify code accepts it instead of reporting an unimplemented level. What a Bareos Data verify reads has not been tested. If you run Bareos, check your own version and its job report before relying on this level either way, and do not assume the Bacula description applies.

How do the levels compare?

Level What it reads What it compares with Catches Does not catch
VolumeToCatalog File attribute data (with signatures) on the volume The catalogue Volume and catalogue disagreeing on file attributes Bad file data; Bareos states file data is not checksummed
DiskToCatalog Files on the client's disk now The catalogue from the last backup Source disk changes or damage since the backup Anything wrong on the volume; deleted files
Data (documented in the Bacula manual; defined in Bareos source, not documented) Data stored on volumes Size and checksum of each file Stored data that no longer matches its checksum Whether the restore will work end to end

How long will it take?

The manuals do not give run times, so measure them yourself. Reading file data back costs at least a full read of the job from the volume.

Example (illustrative): fs01 backs up /srv/data, a full of 1.2 TB (1,200,000 MB in decimal units) to disk volumes on backup01. If the storage can be read at a steady 200 MB/s, one pass over the full takes 1,200,000 ÷ 200 = 6,000 seconds, which is 100 minutes. A weekly Data check of that full needs a 100-minute read window, and only if nothing else is using the device. A VolumeToCatalog run is documented as reading attribute data, so it can finish sooner, but the manuals do not say how much of the volume is touched. A DiskToCatalog run depends on the number of files on fs01, not on the 1.2 TB. Time each level once on a quiet system before you put it in a schedule.

How should you choose?

  • Use VolumeToCatalog straight after a backup when you want to know the volume and catalogue agree. Name the job with Verify Job so the right backup is checked.
  • Use DiskToCatalog when you suspect the client's disk, for example after disk errors, to see what changed against the last good backup.
  • Use Data (documented in the Bacula manual) when you want the stored bytes checked against their checksums, and plan the read window. On Bareos, check your version first, because its documentation does not describe this level.
  • Add a real restore of a sample into a scratch location. Bareos documents the restore command with the option "Select the most recent backup for a client" and an option to select a backup before a specified time. A restore proves the full path, including that the volume is mounted and readable.

How do you check what your own jobs do?

Read the Job and FileSet resources in the Director configuration rather than assuming from a job's name. Look for:

  1. Type = Verify and the Level of each verify job.
  2. The Verify Job (Bacula) or JobToVerify (Bareos) directive. If it is missing, the job compares the last job that ran.
  3. For DiskToCatalog and Catalog, the verify options in the FileSet Include block.
  4. For Bacula Data, the Storage resource named in the Verify job.
  5. The schedule, so that two VolumeToCatalog jobs never overlap for one client.

Then read the job report from a recent run and note what it says it compared. If a provider runs the system for you, ask which Level each verify job uses, which backup job it names, when it last ran, and whether any file data is read back.

Risks and limits

A Verify job reports differences. It is not a restore and does not prove an application will start from the restored files. The Bareos limitation above means file data errors can pass VolumeToCatalog.

Both manuals describe their own versions: Bareos 25 and Bacula 15.0.x. Directive names, levels and behaviour can differ in other versions, and the Bareos 25 documentation describes no Data level although its source defines one. The Bacula manual's sentence about the implementation not identifying deleted files carries a version number from the 1.32 era, so confirm the behaviour on your own version before relying on it.

These commands have not been run or lab-tested.

Sources