Australian cloud for business software — built and run in Sydney since 2003

+61 2 8282 5656
support@netleverage.com
Talk to an engineer
Talk to an engineer
NetLeverage
NetLeverage

Changed Block Tracking was lost after a Storage vMotion, an HA restart or a power event

Troubleshooting article
VMwareBackup & RecoveryTroubleshooting

Your backup product reported that an incremental backup turned into a full one, or that it could not use its saved change position, and the only thing that happened to the virtual machine was a move, a failover or an unclean stop. Sometimes there is no error at all, but the incremental backups look the wrong size.

Changed Block Tracking (CBT) is the feature in which the ESXi host keeps a record of which parts of a virtual disk have changed, so a backup product can copy only those parts. When that record is reset, an old change position may no longer support an incremental and the product needs a supported new baseline. Incorrect tracking can cause missing data without a reported error.

The sources reviewed here do not establish a universal CBT-reset rule for every Storage vMotion or HA restart. KB 320557 says power failure or hard shutdown might reset tracking. Investigate the event and the exact build/product behaviour; a known defect and an unexplained event are different cases. No reset or recovery procedure in this article has been lab validated.

What is actually happening

CBT is switched on per virtual machine and per virtual disk. In the virtual machine configuration file (the .vmx file) it appears as ctkEnabled for the machine and scsiX:Y.ctkEnabled for each disk. For each tracked disk the datastore holds a small tracking file ending in -ctk.vmdk. KB 320557 says each CBT-enabled disk should have a matching file of that kind.

The backup product asks the host for the areas changed since a saved marker, called a change ID. Three outcomes are possible:

Tracking intact     -> host returns the changed areas       -> correct incremental
Tracking reset/gone -> marker is unknown or invalid         -> product establishes a new valid baseline
Tracking wrong      -> host returns an answer, incomplete   -> silent gap in the backup

The first two are visible. The third is the dangerous one, because nothing fails. KB 320557 and KB 315370 both describe a case where the change query returns no error but incorrect data.

How to prove it

All steps here are read-only. Host shell commands below are general shell and VMware CLI use, not taken from a Broadcom page. Quote paths that contain spaces.

  1. Find the event in the vCenter event stream. In the vSphere Client open the virtual machine, then Monitor and Events. Take the time of the first wrong backup and work backwards. Look for a migration task, a failover or restart entry, a power-off, a reconfiguration, a snapshot revert or a disk extension. Storage migration messages appear in the Events tab (vSphere 7.0 documentation).

  2. Look for the warning the backup job may have raised. KB 320557 and KB 315370 give these messages:

Disk "Hard disk #" has incorrect changed block tracking configuration.
One or more VM disks have incorrect changed block tracking configuration.

If absent, a reset is still possible.

  1. Check the configuration flags. For an already powered-off machine, KB 320557 says to search its advanced parameters for ctk. Do not power off a running VM just to perform this read. From existing approved host access:
grep -i ctk "/vmfs/volumes/datastore_name/VM_name/VM_name.vmx"

Configured: ctkEnabled = "TRUE" plus scsiX:Y.ctkEnabled = "TRUE" for every disk. Missing or FALSE: tracking is off, and KB 320557 says a backup after CBT is disabled may copy full disks or become a full backup.

  1. Check the tracking files:
ls -l "/vmfs/volumes/datastore_name/VM_name/" | grep -i ctk

KB 320557 says each tracked disk should have a -ctk.vmdk file, and that after a backup and consolidation no snapshot tracking files such as vmname-000001-ctk.vmdk should remain. A missing file after the machine has been power cycled is a fault. Right after an enable through the API it may simply not have started yet, because KB 315370 says tracking begins only after a stun and unstun.

  1. Search the logs. The virtual machine log vmware.log covers the current power-on session only; earlier sessions are kept as vmware-N.log files in the same folder (general knowledge, check your folder). On the host:
grep -i "DISKLIB-CTK" "/vmfs/volumes/datastore_name/VM_name/"vmware*.log
grep -i "change tracker" /var/log/hostd.log

KB 408791 (ESXi 6.x, 7.x and 8.x) shows DISKLIB-CTK: ChangeTracker_Mirror: Failed to create CTK file and Could not open or create change tracking file. KB 408231 (ESXi 8.x and ESX 9.x) shows Invalid change tracker error code (7228) in hostd.log, with a file fault on the relocate task in the vCenter Server log vpxd.log.

  1. Check that no snapshots exist before any reset. Snapshot Manager can miss a broken chain, so also run:
vim-cmd vmsvc/getallvms
vim-cmd vmsvc/snapshot.get VMID
ls "/vmfs/volumes/datastore_name/VM_name/" | grep -E -- '-[0-9]{6}'

These checks are clues, not proof of an empty active chain. Reconcile the VM configuration and every disk's parent chain, including disks outside the VM folder. Unexplained files or metadata disagreement stop any proposed reset.

None of steps 1 to 6 proves tracking integrity. They can identify visible faults, but missing evidence does not rule faults out. Representative application-level restores and independently recorded test changes provide stronger evidence about the selected recovery points; one successful sample does not prove every backup is complete.

Documented causes

Power failure or hard shutdown. KB 320557 (ESXi 8.x and 9.x) states that after a power failure or hard shutdown while machines are powered on, CBT might reset and lose track of incremental changes. Evidence: a power-off event in the stream.

HA restart after a host failure. KB 387832 (vCenter Server 8.0.x, 7.0.x, 6.x.x) says that when vSphere HA records clnPwrOff=false the machine was not powered off cleanly and HA attempts a restart. The sources reviewed here do not establish a universal HA-restart reset rule. Linking the two is our inference: an unclean stop is the condition KB 320557 names, so treat the machine as possibly reset.

Tracking turned off and on again. KB 320557 states that disabling and re-enabling CBT results in a full backup. Evidence: a reconfiguration event on the machine.

Snapshots that existed before CBT was enabled. KB 320557 and KB 315370 say the change query may return no error but incorrect data.

Disk extension. KB 320557 mentions disabling and re-enabling CBT after a disk is extended past a 128 GB boundary. KB 313039 (ESXi 8.0 U2) describes the worse case: hot extending a disk on 8.0 U2 makes change tracking inconsistent, so backups run after the extend may miss changed data. It appears only after hot extends, not cold ones, on all datastore types. Fixed in ESXi 8.0 U2b build 23305546 or later. Backups already taken cannot be fixed retroactively.

Storage vMotion. KB 442717 (ESXi 8.x and ESX 9.x) says a datastore-only Storage vMotion migrates the disk files and recreates the associated metadata and tracking files on the target datastore. That source does not establish a universal change-ID survival rule; check the installed product and event evidence without treating every migration as a reset. Damaged tracking files also make a Storage vMotion fail: KB 408231 describes failure at about 33% on both running and powered-off machines, and KB 408791 covers snapshot and migration failures. KB 442717 adds that a missing, locked or misaligned tracking file can fail a compute vMotion with Received invalid swap bitmap lengths in vmware.log.

Unsupported states. KB 320557 lists hardware version below 7, physical-mode raw device mappings and independent disks as unsupported. That a disk mode or hardware change can put a machine there is our reasoning.

Snapshot revert. The reviewed sources do not establish a general CBT-reset rule after revert. Ask the backup supplier to assess the event and existing recovery points.

Recovery decisions and escalation

Record the event, exact VM identity, disk coverage, ESXi build, backup-product version and last valid change position. Preserve logs and existing recoverable backups. No single remedy follows from an unexplained HA or migration event.

  • A known ESXi 8.0 U2 hot-extend case: KB 313039 identifies the defect and fixed build 8.0 U2b, 23305546. Have the platform and backup suppliers confirm applicability, patch strategy and treatment of potentially affected backups. Its reset/active-full guidance must not be generalised into a vendor requirement for every HA restart, power event or Storage vMotion. See https://knowledge.broadcom.com/external/article/313039/changed-block-tracking-cbt-is-inconsiste.html.
  • Missing or invalid tracking without that exact defect: establish the supported backup-product workflow. A supplier-approved reset requires proven target identity, complete disk coverage, a preserved restore route, no conflicting tasks and actual chain-state verification rather than an empty snapshot display alone. The procedure must include tracking reactivation and backup acceptance. Unscoped name-based reconfiguration commands are intentionally withheld.
  • A migration or snapshot failure involving a damaged tracking path: the descriptor-change case in KB 408231 is specialist recovery, not routine reset advice. Preserve configuration, every relevant descriptor and disk chain, including components outside the VM folder. Require case-specific support and disposable-copy validation before changes. Retrying migration successfully is only an intermediate result.
  • The cause remains unknown: preserve evidence and ask the backup supplier how to establish a new trustworthy baseline. Do not toggle flags, delete tracking files or repeatedly retry incrementals in place of diagnosis.

Every selected branch must finish with documented tracking state, a supplier-confirmed valid baseline, an isolated application-level restore and a subsequent protection test. A fresh source read must not be confused with a synthetic full assembled from existing backups. Use independently recorded synthetic changes to check both the baseline and the following incremental restore. Configuration flags, new tracking files and successful migration alone cannot demonstrate recoverability.

Any automation must resolve exactly one intended VM on the explicit vCenter, verify its immutable identity and disk inventory, and stop on ambiguous results before a reconfiguration is approved. No reset, descriptor edit, migration or restore was tested for this article.

What not to do

  • Do not treat an incremental that finished without error as complete after a power event. The silent case gives no error.
  • Do not accept "the backup product took its saved marker" as proof of correct tracking.
  • Do not delete tracking files while snapshots exist. KB 339974 requires no active snapshots.
  • Do not enable CBT through the API on a running machine and assume it started. It needs a stun and unstun.
  • Do not hot extend disks on ESXi 8.0 U2 before the fixed build.

Acceptance checklist

These are required checks, not completed test results.

  • [ ] Event, immutable target identity, disk inventory and versions recorded.
  • [ ] Known hot-extend defect separated from unexplained events.
  • [ ] Ambiguous targets or snapshot state stop reconfiguration.
  • [ ] Exceptional descriptor recovery assigned to case-specific support.
  • [ ] Every branch includes tracking reactivation and protection checks.
  • [ ] Baseline and subsequent incremental restore accepted at application level.

Sources

  1. KB 320557, Changed Block Tracking (CBT) on virtual machines. https://knowledge.broadcom.com/external/article/320557 (ESXi 8.x and 9.x)
  2. KB 339974, Resetting Changed Block Tracking for VMware vSphere virtual machines. https://knowledge.broadcom.com/external/article/339974 (ESXi 8.x, 7.x and 6.x)
  3. KB 315370, Enabling or disabling Changed Block Tracking (CBT) on virtual machines. https://knowledge.broadcom.com/external/article/315370 (ESXi 7.x and 8.x)
  4. KB 408231, Storage vMotion or Cloning failure: Inconsistent CBT file (7228). https://knowledge.broadcom.com/external/article/408231/storage-vmotion-or-cloning-failure-incon.html (ESXi 8.x, ESX 9.x)
  5. KB 408791, "Could not open or create change tracking file" error during snapshot creation, snapshot deletion, or vMotion. https://knowledge.broadcom.com/external/article/408791/could-not-open-or-create-change-tracking.html (ESXi 6.x, 7.x, 8.x)
  6. KB 313039, Changed Block Tracking (CBT) is inconsistent after resizing VM disk in vSphere 8.0 U2. https://knowledge.broadcom.com/external/article/313039/changed-block-tracking-cbt-is-inconsiste.html (ESXi 8.0 U2)
  7. KB 442717, vMotion failure - Received invalid swap bitmap lengths. https://knowledge.broadcom.com/external/article/442717/vmotion-failure-received-invalid-swap-b.html (ESXi 8.x, ESX 9.x)
  8. KB 387832, vSphere High Availability (HA) unexpectedly powered on a virtual machine (VM) that was previously in a powered-off state. https://knowledge.broadcom.com/external/article/387832 (vCenter Server 8.0.x, 7.0.x, 6.x.x)
  9. Migrate a Virtual Machine to New Storage, vSphere 7.0 documentation. https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/7-0/vcenter-and-host-management/migrating-virtual-machines-host-management/migrate-a-virtual-machine-with-storage-vmotion-host-management.html (vSphere 7.0)
  10. Respond to Host Failure, vSphere 7.0 documentation. https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/7-0/vsphere-availability/creating-and-using-vsphere-ha-clusters/configuring-cluster-settings/configuring-responses-to-failures/respond-to-host-failure.html (vSphere 7.0)