Australian cloud for business software — built and run in Sydney since 2003

+61 2 8282 5656
support@netleverage.com
Talk to an engineer
Talk to an engineer
NetLeverage
NetLeverage

A backup failed and a production disk is still attached to the backup proxy

Troubleshooting article
VMwareBackup & RecoveryTroubleshooting

You open vCenter and the production VM shows a "Virtual machine disks consolidation is needed" warning. Consolidation fails with "Failed to lock the file" or "One or more disks are busy". The backup proxy VM, the helper machine your backup product uses to read other VMs' disks, lists a hard disk you never gave it.

That combination means a backup used HotAdd, a transport that attaches the production VM's snapshot disk to the proxy so the proxy can read it, and the job ended without detaching the disk. The proxy still holds a lock on files the production VM needs to merge. This article shows how to prove that, how to release the disk through scoped recovery decisions, and which order applies when vSphere refuses to remove the disk.

Version scope: KB 321365 covers ESXi and vCenter Server 7.x to 9.x, KB 374141 covers 8.x and 9.x, KB 416996 covers 7.x to 9.x, KB 375284 covers ESXi 7.x and 8.x, and KB 316576 covers 6.x to 8.x. No recovery procedure here has been lab validated, including those described by vendor sources.

What is actually happening

A snapshot freezes the VM's base disk and sends new writes to a delta file, a separate file holding the changes since the snapshot. Deleting the snapshot merges the delta back into the base disk, and the merge needs exclusive access to those files.

A correct backup closes every disk before it deletes the backup snapshot, and detaches HotAdd disks from the proxy when the job ends, including after a failure (KB 416996). If a job fails or is killed first, the disk stays attached to the proxy. KB 375284 describes the result: the proxy keeps a read-only lock on the base disks, cannot release them because they are still in an active snapshot chain, and the production VM cannot consolidate because the proxy holds the lock. KB 316576 notes that backup appliances create extra locks when disks are hot-added. KB 374141 adds that the chain keeps growing until it reaches the 255 limit.

production VM folder                     backup proxy VM
  prod.vmdk            (base disk)  <---- attached, read-only lock held
  prod-000001.vmdk     (snapshot delta) <- may be attached too

The proxy's settings point to a file in the production VM's folder. That is the signature of an orphan.

How to prove it

All steps here are read-only. Close any settings dialog with Cancel.

  1. Confirm that no backup job is running against the affected VM. Check your backup product's console (KB 321365, KB 375284). A disk attached during a live job is not an orphan.

  2. Read the symptoms. These strings appear on the pages cited:

  • "Failed to lock the file" and "One or more disks are busy" in the client; vmware.log in the VM folder shows "Device or resource busy" and "Failed to lock the file"; vmkernel.log shows locks that are "not free" (KB 321365).
  • "An error occurred while consolidating disks: msg.snapshot.error-DISKLOCKED" and "Unable to access file ... since it is locked" (KB 316576).

  1. Open the proxy VM's Edit Settings and expand the Hard disk entries. Look for a disk you did not give it, whose path is in another VM's folder.

  2. Identify the lock holder from the host. The lock owner is shown in the vSphere Client only for a failed Power On task (8.0 Update 2 and later, KB 316576 and KB 314365). Do not try to power on a running VM to get it. Use the host commands from KB 314365 instead. On the host where the production VM is registered, list the VM folder to get the real snapshot file names, then:

ls -l /vmfs/volumes/<datastore-uuid>/<vm-name>/ vmfsfilelockinfo -p /vmfs/volumes/<datastore-uuid>/<vm-name>/<snapshot-file>.vmdk

The page reports a lock mode: none, exclusive, read-only or multi-writer. Add -v <vCenter IP> -u <admin user> to let the tool query vCenter for the owning host. Then, on the owning host:

lsof | egrep 'Cartel|<snapshot-file>.vmdk' esxcli vm process list

A vmx line against the file shows the Cartel ID of the VM holding it. Match it to the "VMX Cartel ID" in the esxcli output. If the name is the proxy and not the production VM, the disk is orphaned. If the proxy and the production VM share a host, KB 374141 says to compare the process IDs and look for a vmx that does not belong to the production VM.

  1. If no process appears, KB 314365 gives a loop that searches every registered VM's .vmx file for the disk name. Replace the placeholder with the VMDK name. Untested procedure, run in a lab first; it appears to be written to run from /vmfs/volumes.

for i in $(vim-cmd vmsvc/getallvms | grep -v Vmid | awk -F "/" '{print $2}' | awk '{print $1}'); do echo $i && find ./ -iname $i | xargs grep vmdk | grep -Ei _VMDKS_TO_LOOK_FOR_ ; done

It lists every VM's .vmx, and a matching line shows which VM has the disk. VM names with spaces are cut to the first word. From 8.0 Update 3 the embedded vCLS VMs show paths under /var/run/crx/.

The disk is an orphan when the proxy lists it, the proxy holds the lock and no job is running.

Documented causes

The job failed or was killed before it detached the disk (HotAdd). Evidence: the extra disk on the proxy and a proxy vmx process in lsof. Reference: KB 321365 path A, KB 416996, KB 375284.

The backup software did not follow the disk workflow. KB 416996 and KB 321365 path C: it must close every disk, and after a crash or kill call VixDiskLib_Cleanup and VixDiskLib_EndAccess. Evidence: the same orphan returns after each failed job.

Hung process on a proxy using NBD, NBDSSL or SAN. KB 321365 path B says to end the hung process or, with no jobs running, reboot the proxy. KB 416996 names NBD disk handles left open. Proof steps 3 and 4 will not single this cause out, because no disk is attached to the proxy in the way described above. Work with your backup vendor.

A host service. If lsof shows hostd holding the lock, KB 374141 says restarting hostd releases it. It gives no command; KB 316576 points to KB 320280 for restarting management agents.

NFS datastores. Locks are .lck- files, covered by KB 316576 part B and outside this article.

Recovery decisions and escalation

Before either branch, identify the protected VM, proxy, exact disk path and ownership, storage type, ESXi build and backup product. Confirm no backup job is using either target, no consolidation is running, sufficient capacity exists and a separately demonstrated restore route is preserved. Review disruption to every job sharing the proxy, and obtain agreement for the outage and snapshot restore points affected. If identity or ownership is uncertain, stop and escalate.

Normal release: use the backup supplier's supported cleanup for the confirmed attachment. A qualified operator must distinguish it from the proxy's own disks and must never select deletion of datastore files. If removal succeeds and the conflicting holder is released, assess whether the protected VM still requires the separately approved consolidation. If it does not, proceed directly to verification.

Removal refused: KB 375284 describes the specific ESXi 7.x/8.x HotAdd case where the disk or its parent remains part of a snapshot. Have the backup supplier or Broadcom confirm that symptom and scope. Its exception powers off the proxy, consolidates the target, and removes the attachment from the still-powered-off proxy after successful consolidation. Do not apply this exception to arbitrary holders or unknown errors. Once those actions succeed, verify and stop; do not consolidate a second time merely to follow the normal branch. See https://knowledge.broadcom.com/external/article/375284.

Ownership remains unresolved, removal still fails, or a vSAN lock remains: preserve evidence and obtain a case-specific support procedure. KB 321365 includes a vSAN lock-file operation and KB 375284 references proxy configuration editing; this guide withholds wildcard moves, manual configuration edits and forced termination. They are not a default escalation ladder.

An agent or proxy restart is proposed: require a separate impact and recovery plan for all affected jobs and workloads. For a host-agent procedure, follow the current platform restrictions; if HA monitoring is temporarily disabled, record and restore its previous state and verify cluster protection. A guest reboot and a full proxy power cycle are different actions, but neither is an automatic next step.

Acceptance includes the intended attachment state, correct active disk chains, cleared fault, guest/application checks and a successful subsequent backup with cleanup and a representative restore. Keep normal and exception paths separate in the record. No branch has been lab validated here.

What not to do

  • Do not tick the delete-files option when removing the disk from the proxy. The file belongs to the production VM.
  • Do not delete delta files or edit disk descriptors by hand to clear the error. The chain still has to be merged.
  • Do not run consolidation repeatedly while the lock is held. Each attempt fails for the same reason.
  • Do not reboot the proxy during a backup job. KB 321365 requires that no jobs run, and KB 374141 warns other jobs may be affected.
  • Do not use a guest reboot of the proxy as the fix for a lock that persists. KB 374141 requires a full power-off.
  • Do not leave unexpected attachments unresolved. Disk growth consumes capacity; a technical failure ceiling is not a safe operational target. Escalate before capacity or chain limits are approached.
  • Do not run other snapshot operations on a VM during its backup (KB 416996).

Acceptance checklist

These are required checks, not completed test results.

  • [ ] Protected VM, proxy, disk and all active jobs identified.
  • [ ] Shared prerequisites satisfied before either modifying branch.
  • [ ] Normal release distinguished from the exact removal-refused exception.
  • [ ] No datastore-file deletion or generic lock/configuration manipulation.
  • [ ] Already recovered means verify and stop, not consolidate again.
  • [ ] Temporary HA changes restored where applicable.
  • [ ] Application, backup cleanup and representative restore verified.

Sources

  1. KB 321365, Snapshot Consolidation Fails Due to Third-Party Backup Locks. https://knowledge.broadcom.com/external/article/321365. ESXi and vCenter Server 7.x to 9.x.
  2. KB 374141, Snapshot Consolidation Failure: "Failed to lock the file". https://knowledge.broadcom.com/external/article/374141. ESXi and vCenter Server 8.x and 9.x.
  3. KB 375284, Invalid configuration for device '0' error when removing a VMDK from Backup Proxy VM. https://knowledge.broadcom.com/external/article/375284. ESXi 7.x and 8.x.
  4. KB 416996, Stale file locks on VMDKs left by 3rd party backup solutions. https://knowledge.broadcom.com/external/article/416996. ESXi and vCenter Server 7.x, 8.x, 9.x.
  5. KB 316576, Troubleshooting Issues Resulting from Locked Virtual Disks. https://knowledge.broadcom.com/external/article/316576. ESXi 6.x to 8.x; lock owner in task from 8.0 Update 2.
  6. KB 314365, Investigating Virtual Machine file locks on ESXi Host(s). https://knowledge.broadcom.com/external/article/314365. ESXi 7.x to 9.x, vSAN 8.x.