Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

VMware disk made bigger but Linux still shows the old size: rescan, growpart, pvresize and lvextend in order

Technical article
VMware

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

If you enlarged a virtual disk in vSphere and df or lsblk inside the Linux guest still shows the old size, the virtual disk itself may be fine. Broadcom's own wording is that "Extending a virtual disk only extends a virtual machine's disk. This does not automatically increase the size of the file system (such as NTFS for Windows and ext3 for Linux) within the guest." Linux has up to four layers between the disk and your files, and each one has to be told about the new space, in order, from the bottom up.

Work from the bottom. Check that the kernel sees the bigger disk, then grow the partition, the LVM physical volume, the logical volume and finally the file system. Do not skip the read-only checks, because one wrong device name at the partition step can damage data.

What the symptom means

Four sizes sit one above the other, and a vSphere extend changes only the lowest.

  1. The virtual disk. This is what you changed in vSphere. The VMware log records it. In one Broadcom example the log shows capacity = 1258291200 sectors - 600 GB followed by new capacity = 1363148800 sectors - 650 GB. At 512 bytes a sector, 1,258,291,200 × 512 ÷ 1,073,741,824 = 600 GiB, and 1,363,148,800 × 512 ÷ 1,073,741,824 = 650 GiB. Broadcom's conclusion for that case: "VMware disk space expansion is working as designed from the VMware side."
  2. The kernel's idea of the disk. The guest may not notice the change until it rescans the device or reboots. Broadcom's KB 323136 says of an online extend: "you must reboot the guest operating system (OS) in the virtual machine so that it is aware of the size change of the VMDK disk." Broadcom's KB 316393 adds that "If the extended vmdk does not reflect the new size under Linux guest OS", you rescan the SCSI device with echo 1 > /sys/block/sda/device/rescan.
  3. The partition and, if you use LVM, the physical volume, volume group and logical volume. The growpart manual page describes the partition step: it rewrites the partition table "so that the given partition takes up as much space as it can". The pvresize page says it "resizes a PV" and that the PV "may already be in a VG and may have active LVs allocated on it." lvextend then extends the logical volume.
  4. The file system. resize2fs states that it "does not manipulate the size of partitions", and that "The size of the file system may never be larger than the size of the partition." xfs_growfs expands an XFS file system, which "must be mounted to be grown".

Whether the virtual disk could be extended on a running VM at all depends on the controller. Broadcom lists that a "PVSCSI controller allows hot extend disk from vSphere 7.0" and an "NVMe controller allows hot extend disk from vSphere 8.0.1". Its guidance also says all snapshots must have been removed first. If the extend itself failed in vSphere, see "Cannot extend a virtual disk while the VM is running".

Likely causes

Cause What you see How to confirm
The extend never took effect on that disk The VM settings show the old size, or you extended a different disk vSphere disk size in the VM settings; Growing disk lines in the VMware log
The kernel has not re-read the disk size lsblk shows the disk at the old size Compare the disk row in lsblk with the size in vSphere
The partition is still the old size The disk row is large but the partition row is small lsblk shows free space after the last partition
The LVM physical volume is the old size The partition is large but pvs shows the old PV size pvs and vgs show no free space in the volume group
The logical volume is the old size The volume group has free space but the logical volume does not use it vgs shows free space; lvs shows the old logical volume size
The file system is the old size The logical volume or partition is large but df shows the old size df -h on the mount point against lsblk or lvs

Prove the cause before changing anything

These commands only read. Run them as root or with sudo, and write down the device names you see before going further.

lsblk
df -hT
pvs
vgs
lvs

lsblk lists block devices as a tree. Compare the size of the disk row (for example sda) with the size of the partition under it (for example sda3). If the disk row is still the old size, the kernel has not seen the extend: go to Fix 1. If the disk is larger than its last partition, the partition is the next layer. df -hT shows the size and the file system type of each mounted file system, and you need the type (ext4 or xfs) to choose the last command. pvs, vgs and lvs report the LVM physical volumes, volume groups and logical volumes; read the sizes and any free space in the volume group.

To see what growpart would do without changing anything, use its dry-run option. Its manual page documents -N | --dry-run as "Only report what would be done", and the exit status is 0 "if the partition was successfully grown or if --dry-run was specified and it could be grown", 1 if it "could not be grown due to lack of available space" and 2 if an error occurred.

growpart --dry-run /dev/sda 3

Example (illustrative): web01 has a 100 GiB disk sda. Partition sda3 is an LVM physical volume in volume group vg_data, holding logical volume lv_data, an XFS file system mounted at /var/www. The disk is extended to 150 GiB in vSphere, a 50 GiB increase (150 − 100 = 50; 50 × 1,024 = 51,200 MiB). lsblk still shows sda at 100 GiB, so the kernel has not seen the change. After the rescan, sda shows 150 GiB but sda3 is unchanged, so the partition needs growpart. After that, pvs shows the PV unchanged until pvresize runs, and vgs then shows 50 GiB free. lv_data and df stay at the old size until lvextend and the file system step run. Each layer moves only when you move it.

Fix

Take a full backup of the virtual machine first, with all snapshots removed before the vSphere extend (KB 344854 says they must be). Broadcom's guidance is that "VMware recommends to take a complete backup of the virtual machine prior to making these changes." Fixes run in this order, and you stop at the layer that is already correct.

Fix 1: make the kernel see the new disk size

If lsblk shows the old disk size, tell the kernel to rescan the device. Replace sda with your disk:

echo 1 > /sys/block/sda/device/rescan

Run it as root. Broadcom's KB 316393 gives this command as the way to rescan a SCSI device when the extended disk does not show the new size. Broadcom's KB 323136 says a reboot of the guest is needed for it to become aware of the change, so if lsblk still shows the old size after the rescan, reboot the guest in a planned window.

Disruptive step: a guest reboot interrupts everything running on the VM. Back out by waiting for the guest to start; there is no change to undo, because nothing on disk has changed.

Fix 2: grow the partition

Check the dry-run output first, then run it for real:

growpart /dev/sda 3

Risk: this rewrites the partition table of the disk. Use the right disk and the right partition number, and the number counts from 1. A mistake can make data unreachable, and the manual page documents no undo. Back out only by restoring from the backup you took. The page does not say whether the partition may be mounted while it is changed; Broadcom's LVM article powers the VM off and restarts it around its partition change, so plan a window if the data matters.

If the disk has no partition table (LVM directly on the disk), this layer does not exist and you go straight to pvresize.

Fix 3: resize the LVM physical volume

pvresize /dev/sda3

The manual page example reads "Expand a PV after enlarging the partition." Run vgs afterwards and look for free space in the volume group.

Risk: pvresize rewrites LVM metadata on a physical volume that may hold active logical volumes. The manual page documents no undo, so the way back is a restore from backup. Take the backup first and run it on the right device.

Fix 4: extend the logical volume and the file system together

lvextend -l +100%FREE -r /dev/vg_data/lv_data

The lvextend page shows the form lvextend -l+100%FREE -r vg01/lvol01 and describes it as extending "an LV to use all remaining free space in volume group and all resize its filesystem with fsadm(8)". The -r option is documented as "Resize underlying filesystem together with the LV using fsadm(8)", and fsadm says it "tries to use the same API for ext2, ext3, ext4, ReiserFS and XFS filesystem". To add a fixed amount instead of all the free space, use -L with a plus sign; the manual page example is lvextend -L +54 vg01/lvol10 /dev/sdk3.

Disruptive step: this changes the logical volume and the file system. Growing avoids the data-loss risk of shrinking, but if the command is interrupted or the file system is damaged, recovery is from backup. Do not use the same commands to shrink: shrinking is a different procedure with a real chance of data loss.

Fix 5: grow the file system by hand (without -r, or without LVM)

Disruptive step: growing a file system changes it on disk. The resize2fs and xfs_growfs manual pages document no undo for a growth, so the way back is a restore from backup. Take the backup first. For XFS, xfs_growfs -n prints the geometry and checks the arguments without growing anything, so run it first.

For ext4, resize2fs can enlarge a mounted file system if the kernel and file system support online resizing:

resize2fs /dev/vg_data/lv_data

With no size given it "will default to the size of the partition" (or logical volume). For XFS, give the mount point:

xfs_growfs -d /var/www

The manual page says the -d option grows the data section "to the largest size possible" when no size is given, and that "The existing contents of the filesystem are undisturbed".

Confirm the fix

  1. lsblk shows the disk, the partition and any LVM volume at the new sizes.
  2. df -hT shows the mount point at the new size, with the same file system type as before.
  3. vgs shows the free space you expected to remain (none, if you used +100%FREE).
  4. The application on the volume writes data without a space error. Keep the old and new df output with the change record.

What to watch afterwards

  • Free space on the file system. The point of the change is headroom. Alert before the volume fills again, and extend in small steps while you learn the growth rate.
  • Snapshots. Broadcom's guidance says virtual machines running on snapshots cannot have a disk extended. If you use snapshots, remove them before the next extend.
  • Layout. If extends keep needing downtime, the controller type or the layout is the cause; see "Growing a hosted disk: extend easily, shrink rarely, and plan the volume layout".
  • Other disks. If more than one disk was extended, repeat the checks for each. A rescan of one device does not cover another.

Risks and limits

  • These commands have not been run or lab-tested. They come from the Broadcom articles and the Debian manual pages listed below, and the exact options may differ in your distribution.
  • The manual pages are for Debian bookworm package versions. Other distributions and newer releases may behave differently, particularly growpart, which comes from the cloud-utils family (package cloud-guest-utils on Debian).
  • Broadcom's LVM article is written for Red Hat and CentOS guests and shows an older procedure (a new partition, pvcreate, vgextend). The steps above use growpart and pvresize instead, because the manual pages describe them as the way to grow an existing partition and physical volume. Pick one method; do not mix them.
  • The sources do not describe growpart on non-LVM, encrypted (dm-crypt) or multiple-partition layouts beyond the manual-page examples. Check those layouts by hand.

Sources