A VMDK says it is locked and I need to find out who holds the lock

A power-on fails with "Unable to access file ... since it is locked", a consolidation fails with a disk lock error, or a datastore file operation returns "Device or resource busy". Something holds the file, and the message does not say what.

A lock is a record that one host writes so other hosts leave a file alone. On VMFS it carries an owner, normally the MAC address of a network card in the holding host. Read the owner, name the host, then find the process or VM responsible.

The read commands below report state; none is a release step. Every procedure here is untested from our side: run it in a lab first.

What is actually happening

Several hosts use one VMFS datastore at once, so VMFS keeps lock records inside its own metadata. KB 314365 states that VMFS volumes do not have .lck files.

Each lock has a mode (KB 453939, KB 314365):

A running VM legitimately holds locks. A lock remaining after that VM powers off needs investigation of other consumers, including backup proxies and shared-disk configurations; power state alone does not make a lock safe to remove.

A heartbeat record is a record on the datastore that shows a host is alive. KB 453939 notes it is not a lock; it protects locks. The "hb offset" in lock output is the position of the owner's heartbeat record.

Atomic Test and Set (ATS) is a storage command that updates one disk sector in a single step; KB 319797 says VMFS uses it to implement locking. To see each datastore's locking mode:

esxcli storage vmfs lockmode list

KB 416798 describes ATS switched off on a host: reservation conflicts, hung VMs and datastores reporting no capacity. If many VMs fail at once, check this first. Healthy output shows Int Value and Default Int Value both at 1:

localcli system settings advanced list -o /VMFS3/HardwareAcceleratedLocking

How to prove it

On NFS datastores vmkfstools -D and vmfsfilelockinfo do not show locks (KB 412870); use the NFS section.

Run these as root over SSH on the host that runs the VM.

1. Read the failure in vCenter. From vSphere 8.0 Update 2 the lock owner appears in the failed power-on task (KB 316576).

2. Find the disk file the VM is writing to. A disk is a small descriptor file plus a flat file of base data, and each snapshot adds a delta file. From the VM folder (KB 313833):

grep -i vmdk *vmx

3. Dump the lock record. KB 314365 lists the flat, delta or sesparse VMDK as the file to test; use the file named in the error, not the descriptor:

vmkfstools -D /vmfs/volumes/<datastore>/<vm-folder>/<file>-flat.vmdk

4. Read the Lock line and the line after it, which carries mode and owner. KB 323140 shows this layout, owner masked on the page:

Lock [type 10c00001 offset 60088320 v 160, hb offset 3646464
gen 80, mode 1, owner 604931b4-e9227831-####-### mtime 532684]

KB 453939 says the owner would normally indicate the MAC address of the owning network card; KB 323140 says 00:50:56:##:##:## is shown as 00505###### in the output.

5. If the owner is all zeros while the mode is not 0, KB 453939 reads the MAC from the heartbeat record at the hb offset. Print 512 bytes from that offset in .vh.sf:

hexdump -C /vmfs/volumes/<datastore>/.vh.sf -n 512 -s <hb offset>

The page's example has hb offset 3579904, which is 0x36a000. Its first three rows are:

0036a000  02 ef cd ab 00 a0 36 00  00 00 00 00 07 00 00 00
0036a010  00 00 00 00 d9 a4 d7 d7  fe 00 00 00 80 9d ac 4c
0036a020  6d b0 4d 37 1d 8e 02 1a  64 e6 22 67 c1 23 59 00

The page says the 7th to 12th byte hold the MAC. That means bytes 7 to 12 of the third row (offset plus 0x20, 38 bytes into the record): 02 1a 64 e6 22 67. Bytes 7 to 12 of the first row are not the MAC. The page words this loosely, so match the row.

6. Turn the MAC into a host. On each candidate host list its adapters (KB 323140, KB 453939):

esxcli network nic list
esxcfg-vmknic -l
esxcfg-nics -l

The vSphere Client shows the same. KB 453939 matched one MAC to vusb0, so any device with a MAC can own a lock. KB 313833 says a second MAC on a single file is the abnormal lock; act on that host. With HA enabled, esxcli network ip neighbor list can help.

7. Optional shortcut (KB 314365). It prompts for the vCenter password and prints "Host owning the lock on the vmdk is , lockMode: ". Run it one file at a time (KB 313833):

vmfsfilelockinfo -p /vmfs/volumes/<datastore>/<vm-folder>/<file> -v <vCenter IP> -u <admin user>

8. Find what holds the file. KB 314365 gives this, then esxcli vm process list. Match the Cartel ID from lsof to the "VMX Cartel ID" line in that list; its entry names the VM. The World ID in the same output is a different number and is not the match:

lsof | egrep 'Cartel|<file>'

9. For a suspected stale delta, check the file named in the lock is in the temporary location, not under /vmfs/volumes (KB 313833):

ls -l /dev/deltadisks

Documented causes

A backup proxy still has the disk attached. A hot-added disk is one VM's disk attached to another VM, normally the proxy, during a snapshot backup (KB 316576, KB 453939). Evidence: the owner MAC is the proxy's host and its settings list the disk. The lock often survives a host reboot (KB 323140).

A management agent holds it. KB 313833 lists vpxa-worker (vCenter agent), hostd and hostd-profiler (host agent).

A stale delta disk under /dev/deltadisks, per step 9.

A lock with no process behind it. KB 313833 says the locking host has to be rebooted. This is the last resort.

Recovery decisions and escalation

A lock record identifies a holder; it does not by itself authorise downtime or deletion. Do not shut down a healthy running VM simply to make the diagnostic example fit. Identify the fault, exact disk and datastore, owner host, process, VM and active backup/snapshot tasks first.

For a confirmed backup attachment, obtain the backup supplier's supported release procedure. Confirm the protected disk's full path and identity against the proxy's own disks, preserve a demonstrated restore route and never delete the disk's datastore files. If removal is refused, stop for the separately scoped exception described in the orphaned-proxy article rather than improvising.

For a management-agent fault, review the effect on every workload and running task on that host, including management dependencies. Arrange independent administration access and a recovery plan before the applicable individual-service operation in KB 320280. If cluster HA monitoring must change temporarily, record and restore its previous setting and verify protection afterwards. A broad agent restart is not a diagnostic shortcut.

Deletion in scratch directories, lock-file movement, host maintenance and reboot are withheld from this general guide. Before any such operation, a specialist must establish the exact object and topology, active users, capacity/protection after evacuation, recovery access, abort conditions and application acceptance. Do not infer that a similar-looking datastore path is equivalent to a documented scratch directory.

A host shown as not responding may be network-isolated and still writing. Establish its actual power and storage-access state through independent management and storage evidence; do not break locks from another host on the basis of vCenter connectivity alone. Escalate unresolved ownership to Broadcom with the diagnostic outputs.

NFS requires its own investigation

NFS lock-file handling is different from VMFS metadata locking. Confirm the NFS version and the source's applicability. A read-only listing and inspection of a specifically identified lock file may help establish its owner; absence of an expected filename does not establish that storage is unlocked. Do not apply VMFS or vSAN release steps to it.

Any proposed NFS stale-lock intervention requires case-specific confirmation of the owner and all clients that could write, the required power states and a preserved recovery route. This article supplies no lock-move, deletion or maintenance commands. If ownership is uncertain or a lock reappears, stop and investigate the remaining holder.

After approved recovery, verify the intended operation, guest/application correctness, cluster protection and backup operation. All recovery options remain untested here.

What not to do

• Do not delete .vmdk or delta files under /vmfs/volumes to free a lock. A delta holds data written since the last snapshot.
• Do not reboot a host first. A proxy's lock can survive the reboot (KB 323140).
• Do not read an all-zero owner as "not locked". Check the mode and use the heartbeat record.
• Do not turn off ATS to clear a lock. KB 319797 warns that on ATS-only volumes and vSAN this removes locking and causes an outage.
• Do not restart agents with services.sh restart or from the DCUI (KB 320280).

Acceptance checklist

These are required checks, not completed test results.

• [ ] File, lock mode, owner host and process recorded.
• [ ] Cartel matched to VMX Cartel ID, not World ID.
• [ ] Storage type and source applicability confirmed.
• [ ] No shutdown or deletion inferred solely from a lock.
• [ ] Host-wide effects and independent recovery access assessed.
• [ ] Disconnected host not assumed powered off.
• [ ] Application, protection and backup checks define completion.

Sources

1. KB 313833, "VMware virtual machine file lock on VMFS datastore", https://knowledge.broadcom.com/external/article/313833, ESXi 8.0.x, 7.0.x, 6.x.
2. KB 314365, "Investigating Virtual Machine File Locks on ESXi Hosts", https://knowledge.broadcom.com/external/article/314365, ESXi 7.x, 8.x, 9.x; lock owner shown in the vSphere Client from ESXi 8.0 Update 2.
3. KB 316576, "Troubleshooting Issues Resulting from Locked Virtual Disks", https://knowledge.broadcom.com/external/article/316576, ESXi 6.x, 7.x, 8.x.
4. KB 319797, "Disabling hardware accelerated locking (ATS) in ESXi", https://knowledge.broadcom.com/external/article/319797, ESXi 8.0.x and 7.0.x.
5. KB 323140, "Device or resource busy / file is locked error when making changes to a file on an ESXi datastore", https://knowledge.broadcom.com/external/article/323140, ESXi 8.0, 7.0, 6.0.
6. KB 412870, "VM Backup jobs and disks consolidation tasks fail due to stale NFS locks", https://knowledge.broadcom.com/external/article/412870, ESXi 8.x and 9.x.
7. KB 416798, "VMs become hung and datastores become inaccessible due to SCSI reservation conflicts", https://knowledge.broadcom.com/external/article/416798, ESXi, including 8.x.
8. KB 453939, "Viewing and interpreting VMFS metadata", https://knowledge.broadcom.com/external/article/453939/viewing-and-interpreting-vmfs-metadata.html, vSphere ESXi, no version stated.
9. KB 320280, "Restarting Management Agents in ESXi", https://knowledge.broadcom.com/external/article/320280, vSphere 7.x, 8.x, 9.x; VMware Cloud Foundation 4.x, 5.x.