Incremental backups suddenly run as full after a CBT reset

Last night's backup was meant to be an incremental, and it read the whole disk. The job took far longer than usual and moved far more data. Your backup product may report that it could not use changed block tracking, or it may simply record a full backup.

The cause can be that the hypervisor's record of changes was lost or deliberately discarded. That record is Changed Block Tracking (CBT). A backup can no longer rely on an invalid saved change position and may need a new valid baseline; the product determines the supported data-read workflow. This article explains the evidence, limitations and acceptance checks for recovery of protection.

What is actually happening

CBT is a feature of ESXi that records which areas of a virtual disk have been written. It is off by default. When it is on, each tracked disk gets a small companion file in the VM folder. For a disk called vmname.vmdk the companion is vmname-ctk.vmdk, and a disk inside a snapshot has its own, such as vmname-000001-ctk.vmdk (KB 320557).

Two kinds of setting in the .vmx file switch it on. ctkEnabled controls CBT for the whole VM, and each disk has its own entry such as scsi0:0.ctkEnabled. Setting a disk entry to FALSE turns CBT off for that disk only (KB 320557). A VM can therefore be tracked on one disk and not on another.

Your backup product asks the host what changed using a vSphere API call named QueryChangedDiskAreas. The caller passes a change ID, which is an identifier for a point in the past, and gets back the areas written since then. The caller typically reads that ID from the disk's backing information (API reference, 8.0.3). The VDDK programming guide, the developer kit backup products are built on, describes the ID as a sequence number in the form <UUID>/<nnn>. If the UUID part changes, tracking information has become invalid and a full backup is required. A first backup of a snapshot has no saved ID, which the guide treats as a baseline (full) backup (VDDK guide).

Backup 1 (full)    no saved ID, reads the disk      saves change ID  U1/a
Backup 2 (incr)    asks "since U1/a"                saves change ID  U1/b
   ... CBT reset: the tracking files are gone ...
Backup 3           the saved ID cannot be used: full backup, new ID saved

The decision to read everything belongs to your backup product. The guide also describes a special ID, *, for the baseline case. It returns only the active portions of the disk, the disk must be on VMFS, and the VM must have had no snapshots when tracking was enabled (VDDK guide). KB 320557 says the same about zero snapshots. KB 339974 and KB 320557 both say it plainly: after a reset, or after disabling and enabling CBT, the first backup is a full backup.

How to prove it

All steps in this section are read-only.

1. Read the backup job log for the night of the full. Note the exact line your backup product gives for the reason. The wording differs per product.

2. Check the VM folder for tracking files. On the ESXi host:

ls -l /vmfs/volumes/<datastore_name>/<VM_name>/ | grep -i ctk

After power-on, every CBT-enabled disk should have a <vmname>-ctk.vmdk file (KB 315370). Judge by presence only. A missing file on a disk that should be tracked is a finding. File times mean nothing here, because tracking files change whenever the guest writes.

3. Check the settings in the .vmx file:

grep -i ctkEnabled /vmfs/volumes/<datastore_name>/<VM_name>/<VM_name>.vmx

You expect ctkEnabled = "TRUE" plus one scsiX:Y.ctkEnabled = "TRUE" line for each disk the backup covers (KB 320557). A disk with no line, or with FALSE, is not tracked. An error such as "incorrect changed block tracking configuration" on a disk points the same way (KB 320557).

4. Search the VM logs for tracking messages. In our experience the reset can sit in a rotated log, so search all of them:

grep -i "change tracking" /vmfs/volumes/<datastore_name>/<VM_name>/vmware*.log

KB 322285 lists "Could not open change tracking file" and "Change tracking invalid; reinitializing" among the symptoms of a failed power-on on ESXi 7.0 and 8.0. KB 408791 describes a failure to create the -ctk-mirror.vmdk file during snapshot creation, snapshot deletion or storage vMotion, on ESXi 6.x, 7.x and 8.x. Finding none of these lines proves nothing either way, so go on to step 6.

5. List the snapshots in Snapshot Manager and note their dates against the date CBT was enabled. A snapshot that existed before CBT was enabled can make QueryChangedDiskAreas return no error and incorrect data (KB 315370, KB 320557).

6. Open the VM's Tasks and Events and look at the window before the first full. Look for reconfigure tasks, power operations and migrations. We have not found an official page that names a vCenter event for a CBT reset, so this step shows what happened to the VM and does not name the reset.

Missing tracking files, disabled flags or relevant errors are clues requiring investigation. An event preceding the full backup does not itself prove causation. Present files, TRUE flags and quiet logs neither rule out a tracking fault nor establish integrity; check the backup product's change IDs, job decisions and recovery evidence (KB 320557).

Documented causes

Someone disabled and re-enabled CBT. Disabling and then enabling CBT results in a full backup (KB 320557). Check Tasks and Events for a reconfigure.

Hard shutdown or power failure. A power failure or hard shutdown while the VM is on can make CBT reset and lose track of incremental changes (KB 320557). Evidence is a power event just before the first full.

Damaged tracking files. KB 408791 describes invalid, locked or corrupt tracking files on ESXi 6.x–8.x and a reset remedy. KB 322285 addresses a different failed-power-on recovery involving process termination and descriptor changes. Keep those cases separate and obtain supplier/Broadcom guidance; neither is a generic repair sequence supplied here.

Tracking module not loaded after an upgrade. KB 317887 covers ESXi 7.0 U1, where the CBT module failed to load after boot and consolidation failed with "ESXi tracking filter not available". It is fixed in 7.0 U2. To check the module:

esxcli system module list --enabled=true | grep cbt
esxcli system module list --loaded=true | grep cbt

The historical workaround changes host module state and does not persist after reboot. Its command is withheld here; confirm the exact defect and supported update path with the platform supplier.

CBT enabled while snapshots existed. The query may return no error and wrong data (KB 315370). Ask the backup supplier to assess affected recovery points and establish a supported new baseline; do not initiate an automatic reset.

Enabled on a running VM but never started. Setting CBT on a powered-on VM through PowerCLI or the API saves the setting, but tracking does not start until you create and delete a snapshot or power cycle the VM (KB 315370). Until then the -ctk.vmdk files are missing.

Recovery decisions and escalation

Enabled flags and tracking files show configuration, not tracking integrity. A reset is not a routine response to every full backup. Ask the backup supplier to establish whether the job lost its valid change position, followed a policy-driven full, or encountered a build-specific defect.

Before a reset is considered, identify the exact VM and every protected disk, ESXi build, backup-product version and selected supported workflow. Let active backup and consolidation tasks finish. Preserve configuration and a separately demonstrated recovery route. Reconcile Snapshot Manager with the VM's actual per-disk chain and relevant configuration; an empty display alone does not establish that no snapshots exist. If the records disagree, stop before changing tracking or snapshot metadata.

A supplier-approved reset must specify power state, snapshot restrictions, preservation of metadata, disk coverage and the supported mechanism that activates tracking again. KB 320557 and the reset sources describe configuration changes; activation can also require a lifecycle event. Such events themselves affect availability or the disk chain. This guide withholds generic metadata-move commands and snapshot/power-cycle instructions pending validation of the selected product/build procedure.

After the chosen workflow, verify configuration and tracking activation, but do not call that restored protection. Agree with the backup supplier how to obtain a fresh valid baseline without trusting suspect change information. A synthetic full assembled from existing backups is not evidence of a fresh source read. Product terminology and the actual data-read behaviour must be checked.

In an isolated lab, record known synthetic changes independently, restore the new baseline and check those records at application level. Make further known changes, take the subsequent supported incremental, restore that recovery point and check again. Retain logs, disk coverage and application results. A completed full followed by a job labelled incremental is insufficient acceptance. Stop and escalate if either restore fails or the backup supplier cannot establish the baseline's validity.

No reset, backup or restore was executed for this article; these are proposed acceptance checks, not test results.

What not to do

• Do not repeatedly retry an incremental without investigating an invalid change ID. The product may need a new baseline; its logs and supported workflow determine the response.
• Do not enable CBT while snapshots exist. The query can return incorrect data with no error (KB 315370), and an incremental built on it cannot be trusted.
• Do not move the .vmsd file while snapshots exist. It holds the snapshot tree, and the VM loses that metadata (KB 408791).
• Do not hand-edit VMDK descriptors to disable tracking based solely on a KB; require case-specific support and disposable-copy validation. KB 322285 asks for a backup of the .vmx file but gives no backup step for the descriptors.
• Do not load the CBT module by hand and assume it is fixed. It does not survive a reboot (KB 317887).

Acceptance checklist

These are required checks, not completed test results.

• [ ] Backup-product reason for full read investigated.
• [ ] Actual chain state reconciled with snapshot displays.
• [ ] Exact identity, disk coverage, versions and supplier workflow established.
• [ ] Metadata changes withheld when snapshot state is uncertain.
• [ ] Activation checked separately from integrity.
• [ ] New baseline data-read behaviour confirmed with supplier.
• [ ] Baseline and subsequent incremental restores validate independent synthetic changes.

Sources

1. KB 320557, Changed Block Tracking (CBT) on virtual machines. https://knowledge.broadcom.com/external/article/320557/changed-block-tracking-cbt-on-virtual.html. ESXi 8.x and 9.x.
2. KB 339974, Resetting Changed Block Tracking for VMware vSphere virtual machines. https://knowledge.broadcom.com/external/article/339974. ESXi 8.x, 7.x and 6.x.
3. KB 315370, Enabling or disabling Changed Block Tracking (CBT) on virtual machines. https://knowledge.broadcom.com/external/article/315370. ESXi 7.x and 8.x.
4. KB 317887, Error: "ESXi tracking filter not available" when consolidating snapshots. https://knowledge.broadcom.com/external/article/317887. vSphere 7.0.x, fixed in ESXi 7.0 U2.
5. KB 322285, VM power-on fails with error: Change tracking invalid or disk in use. https://knowledge.broadcom.com/external/article/322285/vm-poweron-fails-with-error-change-trac.html. ESXi 7.0 and 8.0.
6. KB 408791, "Could not open or create change tracking file" error during Snapshot creation, snapshot deletion, or vMotion. https://knowledge.broadcom.com/external/article/408791/creation-of-snapshot-and-vmotion-fails.html. ESXi 6.x, 7.x and 8.x.
7. Low Level Backup Procedures, Virtual Disk API (VDDK programming guide). https://developer.broadcom.com/xapis/virtual-disk-api/latest/vddkBkupVadp.9.3.html. No release named on the page; ESXi 4.0 and later, hardware version 7 or higher.
8. QueryChangedDiskAreasRequestType, Virtual Infrastructure JSON API. https://developer.broadcom.com/xapis/virtual-infrastructure-json-api/8.0.3/data-structures/QueryChangedDiskAreasRequestType/. Version 8.0.3.