Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Cannot extend a virtual disk while the VM is running

Technical article
VMware

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

A VM is running low on disk space. You open Edit Settings, raise the hard disk size, and the task fails. Or the size field will not change at all. Hot extend is documented for a flat VMFS disk with no snapshots on a PVSCSI controller (from vSphere 7.0) or NVMe controller (from vSphere 8.0.1) (KB 344854, KB 322255, KB 323131). For any other controller, power the VM off first. ESXi names the cause in the error, and each reason code is covered below.

What the error means

KB 322255 gives the general form:

The disk extend operation failed: REASON

Where REASON is one of msg.disklib.NOSPACE, msg.fileio.lock, msg.disklib.UNSUPPORTEDFEATURE, msg.disklib.INVAL, msg.disk.hotextend.badMode, 22 (Invalid argument), 5 (Input/output error) or 12 (Cannot allocate memory).

Some failures carry a longer message. KB 323131 shows: "Hot-extend is currently supported only for VMFS flat virtual disks without snapshots opened in persistent mode." In the VM's vmware.log it shows:

HotExtend: Extending scsi1:0 to 167772160
[msg.disk.hotextend.Failed] Failed to extend disk scsi1:0.

If the size box is greyed out, KB 344854 says: "If this option is greyed out, the disk may be running on snapshots or the disk may be at the maximum allowed size depending on the block size of the datastore."

Likely causes

Reason code Cause Source
msg.disk.hotextend.badMode One or more snapshots on the disk KB 322255
msg.disklib.NOSPACE Not enough free space on the datastore KB 322255
msg.fileio.lock Disk "already actively in use and locked by some ESXi host" KB 322255
msg.disklib.UNSUPPORTEDFEATURE "ParentCID parameter in the virtual disk's descriptor file is not set to ffffffff" KB 322255, 323131
"not supported" on a shared disk Multi-writer flag set KB 425578
msg.disklib.INVAL / 22 (Invalid argument) Sparse disk, new size smaller, unknown datastore space, or past 2 TB on a powered-on VM KB 322255
5 (Input/output error) Host "has a problem accessing the underlying storage device or datastore" KB 322255
12 (Cannot allocate memory) Storage stack out of heap memory, or many stale snapshot files KB 322255
Option refused in the vSphere Client IDE disk KB 344854
Running VM on a controller other than PVSCSI or NVMe KB 344854 names only those two as hot-extend capable KB 344854

The text "Hot-extend is currently supported only for VMFS flat virtual disks without snapshots opened in persistent mode" appears for the ParentCID case (KB 323131) and for the multi-writer case (KB 425578). Steps 4 and 6 below tell them apart.

KB 318825: "Virtual machine disks cannot be increased in size while snapshots exist, regardless of whether the virtual machine is powered on or off."

Prove the cause before changing anything

1. Read the exact reason. Take it from the failed task in Recent Tasks, or from the VM's vmware.log.

2. Look for snapshots. Check Snapshot Manager on the VM. KB 318825 warns: "Snapshots taken by third-party software (via API) may not appear in the Snapshot Manager. Routinely check for snapshots using the command line." Then check from the ESXi shell, read-only. KB 344559 gives these:

vim-cmd vmsvc/getallvms
vim-cmd vmsvc/get.snapshotinfo vmid
less VM_deploy.vmx |grep vmdk
ls -lrth | grep *sesparse.vmdk

Run the last two in the VM's folder (/vmfs/volumes/Datastore_name/Virtual_Machine_name/). The vmid is the first column of getallvms. KB 316575 explains the file name test: a disk entry in the .vmx or in Edit Settings that uses a -00000X.vmdk file is running on a snapshot, as in scsi0:0.fileName = "TestVM-000001.vmdk". To look across every datastore the host can see, KB 316545 gives:

find /vmfs/volumes/*/ -name "*000*.vmdk" | xargs ls -lh | awk '{ print $5,"\t"$6,$7,$8"\t",$9"\t",$10}'

Bad: a -00000X disk in use or delta files that Snapshot Manager does not list. Good: every disk points at its base file.

3. Check the disk type and controller. In Edit Settings, note the controller (SCSI, PVSCSI, NVMe or IDE), the sharing mode and the disk mode. KB 344854: "Increasing the size of an IDE virtual disk using the vSphere Client is not supported. SCSI virtual disks are the only supported disk that can be expanded via the vSphere Client." It also lists "PVSCSI controller allows hot extend disk from vSphere 7.0" and "NVMe controller allows hot extend disk from vSphere 8.0.1."

3b. Check the host build. From the ESXi shell run vmware -v (KB 345263). KB 313039 reports a defect on ESXi 8.0 U2 that is "fixed in ESXi 8.0U2b build 23305546 or later". On an 8.0 U2 host below that build, do not hot extend. See the Risk at the start of the Fix section.

4. Check for a multi-writer disk. If sharing is set to multi-writer, KB 425578 applies: "VMware vSphere does not support Hot Extend (online expansion) for virtual disks with the Multi-writer flag enabled."

5. Check datastore free space. Compare it with the size increase, allowing for thick disks that use the full new size at once.

6. Read the descriptor's parent ID. For UNSUPPORTEDFEATURE, view the small descriptor file (not the -flat file), at a path of the form /vmfs/volumes/datastore/VM_Directory/vm.vmdk (KB 323131), with a read-only viewer such as less, and read the parentCID= line. KB 323131 shows a bad value as parentCID=72ca12de on a base disk; a base disk should read parentCID=ffffffff. Note that this fault fails "regardless of the virtual machine's power state."

Example (illustrative): a file server's data disk will not grow. The task shows msg.disk.hotextend.badMode. Snapshot Manager is empty, but Edit Settings shows the disk as fs01_1-000002.vmdk. A backup job failed last week and left two snapshots it created through the API. The fix is to remove those snapshots with the backup owner's agreement, confirm the disk is back on its base file, and then extend.

Fix

Risk, before any hot extend: ESXi 8.0 U2 below build 23305546. KB 313039: vSphere 8.0 Update 2 changed how disks are extended, which led to incorrect change tracking, backups that miss data and "corrupt restores". KB 313039 also lists data corruption of the VM after a powered-on vMotion. The defect is "fixed in ESXi 8.0U2b build 23305546 or later". On a host below that build, power the VM off and extend it cold: "Grow disks only when the VM is powered off." That means downtime. If a disk was already hot extended there, KB 313039 lists ways to force the disk to be re-opened: power cycle the VM, take a snapshot, suspend and resume, or remove and hot-add the disk. Protect future backups by resetting CBT or running an active full backup. Either makes the next backup a full one.

Snapshots (badMode). KB 322255: "Delete the snapshot or snapshots that are on the disk needing to be extended. Once the snapshots are deleted, perform the extend operation again." KB 318825: "Increasing the size of VMDK disks running on a snapshot should never be attempted, even using the CLI." If Snapshot Manager lists them, delete them there. If it lists none but step 2 found a disk running on a snapshot, KB 316575 gives the shell route:

vim-cmd vmsvc/getallvms
vim-cmd vmsvc/snapshot.get [Vmid]
vim-cmd vmsvc/snapshot.create [VmId] [snapshotName] [snapshotDescription] [includeMemory] [quiesced]
vim-cmd vmsvc/snapshot.removeall [Vmid]

KB 316575: when the VM shows consolidation needed and no snapshots are present, "creating a new snapshot and deleting it clears the entire hierarchy", and it needs a small amount of free space. Disruptive step: removal merges the snapshot data into the base disk, cannot be undone, and loads the storage for as long as it takes. KB 316575: "The remove snapshot process can take a long time to complete if the snapshots are large." KB 318825: "Do not cancel a running deletion/consolidation task, as it can lead to data corruption." Check that no backup job still needs the snapshots. If a removal fails on the running VM, KB 316575 says to retry in a window with the VM powered off, and then to clone the disk with vmkfstools -i or clone the VM.

No space (NOSPACE). KB 322255: "either create more free space in the datastore by migrating or deleting unnecessary virtual machines and files, or increase the size of the datastore."

Lock (msg.fileio.lock). Find which host holds the lock before you act. KB 322255 notes that "these locks may not be released when the virtual machine is powered off" and points to the file-lock guide, KB 314365. Read-only, from the ESXi host where the VM is registered:

vmfsfilelockinfo -p /vmfs/volumes/<UUID>/VM_name/VM_name-000001-delta.vmdk
lsof | egrep 'Cartel|VM_name-000001-delta.vmdk'
esxcli vm process list

The first command names the host that holds the lock. lsof finds the Cartel ID holding the file, and esxcli vm process list matches that ID to a VM name (KB 314365). KB 314365's removal options are to power off the VM holding the lock and reboot the host that holds it, or, on NFS only, to move the .lck file of a powered-off VM into a backup folder. Disruptive step: a host reboot takes every VM on that host with it.

Multi-writer disk. KB 425578: "the disk must be resized in an offline state (Cold Extend)." "Shut down all identified Virtual Machines. (The disk cannot be expanded if even one attached VM is still running)." Disruptive step: every VM that shares the disk goes down together.

ParentCID wrong (UNSUPPORTEDFEATURE). KB 323131 edits the descriptor so the base disk reads parentCID=ffffffff, using vi /vmfs/volumes/datastore/VM_Directory/vm.vmdk. It says: "Ensure to take a backup of the virtual machine before proceeding with these steps." Disruptive step: a wrong edit to a descriptor can make the disk unreadable. Copy the descriptor first and check that the disk really is a base disk with no snapshot chain. Way back: put the copied descriptor back over the edited file.

Invalid argument. Check the new size is larger than the current one, and that the disk is not sparse. For disks over 2 TB, KB 321231 says "ESXi 6.5.x and above now supports hot extending a VMDK beyond 2 TB." The 2 TB limit in KB 322255 applies to older hosts.

Clustered disks. KB 344854: "If the disks are clustered, they cannot be expanded through vSphere Client but instead through CLI. Use this command as an example":

vmkfstools -X 6G -d eagerzeroedthick vm.vmdk

Disruptive step and risk: this extends a shared disk. KB 344854's checks are that all snapshots are removed and the VM is powered off, and KB 425578 says the disk cannot be expanded if even one attached VM is running, so shut down every VM that shares it. Take a backup first (KB 344854). 6G and -d eagerzeroedthick are the KB's example values; set the new size and disk format for your disk. An extend cannot be undone. The way back is to restore from the backup.

IDE disk. KB 344854: "Increasing the size of an IDE virtual disk using the vSphere Client is not supported."

Controller not named as hot-extend capable. If the VM is running and the controller is neither PVSCSI nor NVMe, power the VM off, extend the disk and power it on. Disruptive step: this is downtime, and the extend cannot be undone, so back up first.

Error 5 (Input/output error). KB 322255 points to its storage connectivity articles for LUN, Fibre Channel, iSCSI and NFS. Fix the path, then retry.

Error 12 (Cannot allocate memory). KB 322255 gives two causes. For stale snapshot files, use the find command in step 2 and the snapshot steps above. For VMFS heap exhaustion on ESXi up to 7.0.x, KB 344097 shows Heap vmfs3 already at its maximum size. Cannot expand. in the vmkernel log.

Confirm the fix

  • The extend task completes, and Edit Settings shows the new size.
  • vmware.log has no new msg.disk.hotextend.Failed line.
  • The guest sees the larger disk. KB 344854: "the size of the disk partition does not increase as observed inside the guest operating system." Grow the partition inside the guest next, using the guest operating system's own steps.

What to watch afterwards

  • Leftover snapshots from backup tools that do not show in Snapshot Manager.
  • Change tracking after any hot extend on an ESXi 8.0 U2 host below build 23305546. See the Risk at the start of the Fix section.
  • Datastore free space after thick disks grow.

Risks and limits

  • KB 344854 recommends "that valid virtual machine backups are taken before increasing the size of virtual disks."
  • Forcing an extend on a shared disk is unsafe. KB 425578 explains: "The hypervisor cannot safely resize the underlying VMDK while multiple VMs hold active write locks on the file, as this risks data corruption and metadata conflicts."
  • KB 344854 also lists "The virtual machine must be powered off." among its checks. When unsure, extend cold.

NetLeverage runs its own hosting on VMware vSphere, vCenter and ESXi. Talk to an engineer.

Sources