Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Grew the LUN but the VMFS datastore did not get bigger: rescan, then Increase Capacity

Technical article
VMware

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

Growing a LUN on the storage array does not grow the VMFS datastore on it. Broadcom's article puts it plainly: "VMFS volumes do not automatically expand." The sequence is: grow the LUN on the array, rescan storage on every host that sees it, check that the device backing the datastore reports the new size, then run Increase Capacity and choose to expand the existing extent.

If the wizard lists no device with free space, the cause is one of a few things, and each can be proved from the host before anything is changed.

What the symptom means

A VMFS datastore sits on a partition on a device. When the array grows the LUN, the device gets bigger but the partition and the file system do not. Increase Capacity grows the partition and then the file system. For that, vCenter first has to see the new device size, and it applies filters before it offers a device.

Broadcom's article on expanding through the Host Client describes two options in the Increase Capacity wizard:

  • Add an extent to the existing datastore. This spans a new volume onto the datastore and creates a multi-extent datastore.
  • Expand an existing VMFS datastore extent. This increases capacity of the existing device itself, which is what you want after growing the LUN.

The filters exist to stop corruption. Broadcom says vCenter offers extents only if the LUNs are used as datastores on that host or any other host (force-mounted volumes excepted), are not used as raw device mappings, and are seen the same by all hosts. A log line from vpxd.log shows the filter at work: "[QueryAvailableDisksForVmfs] Removed all candidates because all LUNs in the datastore are not visible by the same hosts or have the same transport types". Broadcom recommends against switching the filters off, because doing so may result in storage device corruption or performance degradation.

Likely causes

Cause What you see How to confirm
LUN grown but not rescanned Device Backing still shows the old size Rescan, then check Configure, Device Backing, Extent Name
Only some hosts rescanned Datastore capacity flips between the old and new size in the vSphere Client Rescan on all hosts with access to the datastore
LUN ID differs between hosts Wizard lists nothing; the vpxd.log filter line above Compare the LUN number for the device on every host
Datastore was force-mounted as a snapshot Wizard shows no device with free space vim-cmd hostsvc/datastore/info <name> shows a non-null forceMountedInfo; see "VMFS datastore missing after a LUN is presented again: 'unresolved VMFS volume' snapshot, mount or resignature"
Backup GPT table not at the end of the disk Host Client error "Failed to update disk partitions" partedUtil getUsableSectors prints "The backup GPT table is not at the end of the disk, as it should be"
Multi-extent datastore grown at the wrong extent "Operation failed! Cannot change the host configuration." or stale capacity The head extent was grown rather than the tail
Host Client error with the end sector "Cannot parse "NaN" as a long value" The wizard used the whole device; leave a few MB at the end

Prove the cause before changing anything

Checks one to six in the numbered list that follows change nothing, apart from the rescan in check two, on the host or in the vSphere Client.

  1. Device size. In the vSphere Client open the datastore, go to Configure, Device Backing, and select the Extent Name. Device Details should show the new capacity. Broadcom's example is a datastore of 250 GB whose device was grown to 400 GB.
  2. Rescan (a rescan does not cause a service interruption, Broadcom says, but it is not a read-only check). In vCenter 8.0 Update 3 and later: Datastore, Hosts, select all hosts, Actions, Rescan Storage. In 8.0 Update 2 and older: Hosts and Clusters, right-click the cluster, Storage, Rescan Storage. If the LUN is visible to hosts in more than one cluster, rescan every one of those clusters. From a host's shell: esxcli storage core adapter rescan --all, then vmkfstools -V.
  3. File system view. esxcli storage filesystem list lists the volumes the host sees, and esxcli storage vmfs extent list lists the VMFS extents. vmkfstools -Ph -v 10 /vmfs/volumes/<datastore name> is the command Broadcom's page 390604 uses to count the extents; its sample output reads "file system spanning 3 partitions" for a three-extent datastore.
  4. Force mount check. vim-cmd hostsvc/datastore/info <datastore name> | grep -A2 forceMountedInfo. A null value is good.
  5. Partition table. partedUtil getptbl /vmfs/devices/disks/<naa id> and partedUtil getUsableSectors /vmfs/devices/disks/<naa id>. A backup-GPT warning in either is the fifth row of the table above.
  6. LUN ID. Compare the LUN number for the device on each host. Broadcom says it must remain consistent for the same LUN across all hosts.

Example (illustrative): ds-prod01 is a 500 GiB datastore on one LUN, and the array team grows the LUN to 800 GiB, so 800 − 500 = 300 GiB of new space. After a cluster rescan, Device Backing for ds-prod01 shows 800 GiB but the datastore still reports 500 GiB. That is the normal state: the device is bigger and the file system has not been told. Increase Capacity, with "Expand an existing VMFS datastore extent", is the step that uses the 300 GiB. If the wizard lists no device here, row four and row three of the table are the ones to check.

Fix

Normal path (vSphere Client)

  1. Grow the LUN on the array.

  2. Rescan storage on every host and cluster that sees the LUN, as above. Broadcom recommends a cluster-level rescan from vCenter after any expansion, because without it vCenter may not detect the change and may show a space mismatch.

  3. Check Device Backing shows the new size.

  4. Right-click the datastore and choose Increase Capacity. Select "Expand an existing VMFS datastore extent", choose the VMFS partition, set the amount with the slider, and finish.

    Disruptive step: this grows the partition and the file system of a datastore that may be in use. There is no way to shrink it back, so the only way out is a restore from backup. Take a backup first. Do it from one host's vSphere Client session at a time, finish it, then rescan before you start anything else on that datastore.

  5. If the host-level wizard fails with the "NaN" error, Broadcom says to read the usable sectors with partedUtil getptbl and partedUtil getUsableSectors and expand to slightly less than the whole device, leaving a few MB at the end.

  6. Rescan again across the cluster.

If you did this directly on a host rather than in vCenter, Broadcom says to follow it with a cluster-level rescan so every host agrees. Otherwise capacity can flip between sizes in the client and raise "increased in capacity" alerts.

Force-mounted datastore

Broadcom says a force-mounted datastore cannot be expanded. It needs resignaturing, which is a disruptive change: see "VMFS datastore missing after a LUN is presented again: 'unresolved VMFS volume' snapshot, mount or resignature". Broadcom adds that if a growth was attempted from the vSphere Client on a force-mounted datastore, the partition table may have grown without the file system, and it asks you to contact Broadcom support to finish the extension from the command line.

Backup GPT table

For the "backup GPT table is not at the end of the disk" message, Broadcom's article gives partedUtil fixGpt /vmfs/devices/disks/<naa id>, followed by vmkfstools -V and a retry of the wizard.

Disruptive step: this rewrites the partition table of a live datastore device. Never run it on a raw device mapping disk (Broadcom's prompt says so). Take a backup first. There is no way back other than a restore. Broadcom's final fallback is partedUtil resize with the existing start sector and a new end sector one less than the last usable sector, then one vmkfstools --growfs command in which the partition path is given twice, once as each argument (vmkfstools --growfs "/vmfs/devices/disks/<naa id>:1" "/vmfs/devices/disks/<naa id>:1"), and then a rescan. Broadcom directs that case to a support ticket to validate the partition size, so treat it as supported only with Broadcom's help.

Multi-extent datastores

Expand the tail extent, not the head. Broadcom says VMFS-6 concatenates extents in order, so space added to a head extent cannot be used, and recommends single-extent datastores as best practice. Its repair for a wrongly expanded head is to build a new single-extent datastore and Storage vMotion the VMs across.

Disruptive step: adding an extent from the command line with vmkfstools -Z <new device:1> <existing device:1> formats the new device and the host prompts "All data on ... will be lost". The extent is named first and the original second. Broadcom's page, written for a host whose UI is not reachable, uses full device paths (/vmfs/devices/disks/naa...:1) and first gives the new device a partition table and a VMFS partition with partedUtil; without the full path the command fails with "Device path name ... is not a valid absolute or relative path". A wrong device order or device destroys data and there is no way back.

Confirm the fix

  • The datastore capacity in the vSphere Client reflects the grown device, and the same value shows on every host.
  • esxcli storage filesystem list shows the same size on every host.
  • Capacity no longer flips between old and new in the client.

What to watch afterwards

  • Make a cluster-level rescan part of every LUN growth change.
  • Keep LUN numbers consistent across hosts when presenting storage.
  • Check the LUN has not exceeded the maximum LUN size in the Configuration Maximums document for your ESXi version, which Broadcom's article names.

Risks and limits

  • These commands have not been run or lab-tested. They are taken from Broadcom's knowledge base articles.
  • Broadcom's pages differ on the options. KB 315433 describes creating a new VMFS partition on the free space and using it as an extent candidate for the original volume. KB 314373 lists both "Add an extent to existing VMFS datastore" and "Expand an existing VMFS datastore extent". The steps above use the second option, because KB 315433 itself sends ESXi 8.x readers to KB 314373 and to the Increase VMFS Datastores section of the vSphere 8.x Storage Guide. KB 315433 also says, "on a safer side", to have all virtual machines on the volume powered down and no active virtual swap files on it; Broadcom's wizard pages (KB 313382 and KB 314373) do not repeat that precondition, so decide from your own change rules.
  • Menu names differ by vSphere build.
  • Any change to a partition table or file system on a live datastore can lose data. Back up first, change one host's view at a time, and use Broadcom support for the command-line steps where its articles say so.

Sources