Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

VMFS datastore missing after a LUN is presented again: 'unresolved VMFS volume' snapshot, mount or resignature

Technical article
VMware

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

If a datastore vanished after a storage change, the LUN is still presented, and the device shows as "Not Consumed", ESXi is treating the LUN as a snapshot or replica of a datastore it already knows. The mount error reads "An unresolved VMFS volume with signature naa.#### has been detected on disk". Run esxcli storage vmfs snapshot list on the host. It shows the volume, whether it can be mounted and whether it can be resignatured.

You then choose between two actions. A force mount keeps the old signature and is a temporary way to get at the data. A resignature gives the datastore a new identity and changes how virtual machines find their files. Pick before you click, because the choice has consequences either way.

What the symptom means

A VMFS datastore carries an identity in its metadata, including the device and LUN it was created on. When the host reads a LUN, it compares the identity it sees now with the identity stored on disk. If they differ, ESXi flags the volume as a snapshot or replica and refuses to mount it automatically, to avoid two volumes with the same signature colliding.

Broadcom's articles list these triggers: replacing SAN hardware, firmware upgrades, SAN replication, disaster-recovery tests, some HBA firmware upgrades, a replaced local storage controller that reports a new device ID, and a LUN ID that no longer matches the ID the datastore was created with. The host log shows it. In /var/run/log/vmkernel.log look for "detected to be a snapshot" followed by a "queried disk ID" line and an "on-disk disk ID" line. In Broadcom's first example the LUN number changed from 17 to 36 because the LUN was unmapped and remapped. In the second the hardware ID hash changed completely.

Broadcom's vSphere Client symptoms: the datastore is missing though the LUN is presented; the datastore device shows "Not Consumed"; one host sees the datastore while others see only the LUN; or an attempt to expand a datastore fails.

Likely causes

Cause What you see How to confirm
LUN ID changed when re-presented "detected to be a snapshot" with different lun values in the queried and on-disk IDs vmkernel.log
Array replica or DR copy presented A second volume with the same VMFS UUID esxcli storage vmfs snapshot list shows it
Replaced controller, new device ID A new device ID (naa, t10) but the same data Broadcom's controller case: the device reports a new ID, so compare the device ID now with the ID recorded before
Array migration or replication Data copied to a different serial number or WWN The "h(id)" values in the queried and on-disk lines differ (Broadcom's second example)
Original and copy both visible to one host Force mount refused; copy can only be resignatured The original datastore is mounted and has the same UUID
Duplicate extents "Can mount: No (duplicate extents found)" and "Can resignature: No (duplicate extents found)" esxcfg-volume -l
Multi-extent datastore with extents missing "Some extents missing" esxcfg-volume -l; all constituent LUNs must be visible
Already force-mounted on another host in the same datacenter A second host cannot mount it from vCenter; "Resolve VMFS Volumes" fails with "Cannot change the host configuration" The vCenter task list

Prove the cause before changing anything

All read-only. Sign in to the ESXi host over SSH.

esxcli storage vmfs snapshot list
esxcfg-volume -l
esxcli storage vmfs extent list

For each unresolved volume esxcli storage vmfs snapshot list prints a block with the volume name, VMFS UUID, "Can mount", "Reason for un-mountability", "Can resignature", "Reason for non-resignaturability" and "Unresolved Extent Count". If it prints nothing, the host does not see a snapshot, and Broadcom shows that force-mounting by name then fails with "No unresolved VMFS snapshots with volume label ... found". esxcfg-volume -l lists the volumes detected as snapshots and is the command Broadcom uses to report duplicate and missing extents. esxcli storage vmfs extent list lists the VMFS extents available on the host, which shows which devices back datastores that are already mounted.

Check whether the datastore was force-mounted before. Broadcom's command is vim-cmd hostsvc/datastore/info <datastore name> | grep -A2 forceMountedInfo. A null value means not force-mounted. A block with persist = true and mounted = true means it is, and that the mount survives reboots.

Example (illustrative): during a disaster-recovery test, the array presents a replica of datastore Prod-VMFS01 to esx01 as LUN 36. The original is on LUN 17 and is still mounted on esx01. esxcli storage vmfs snapshot list lists the replica, with its "Can mount" and "Can resignature" lines. Because the original is online with the same signature, a force mount is not allowed. Only a resignature will bring the replica up, as the new datastore snap-<id>-Prod-VMFS01. The three test VMs on it must then be registered from their .vmx files.

Fix: choose between force mount and resignature

Broadcom is direct that keeping the existing signature "will cause problems in the future", such as preventing the datastore from growing and preventing it from mounting automatically on new hosts. Force mounting is described as a temporary step to gain immediate access. Broadcom also says a force mount is allowed when the original is not online, and that when both the original and the copy are visible to the host, a resignature is the only route.

Option 1: force mount (keeps the signature)

Use it only when the original is not online. For a mount that persists across reboots: esxcli storage vmfs snapshot mount -l <label> or -u <uuid>. For one that does not: add -n. The ESXCLI reference defines -n as mounting the volume so that it will not be automounted after a restart. The vSphere Client route is Datastores, create a new datastore, VMFS, select the device with a value in the Snapshot Volume column, and choose "Keep existing" under Mount Options.

Risk: a force-mounted datastore cannot be expanded, and once one host has force-mounted it, Broadcom says other hosts in the same datacenter cannot mount it from vCenter Server (the datastore "can be mounted to one host only until it is resignatured"). Plan the resignature. To back out of the mount, unmount it (see the unmount command under Option 2) once nothing uses it.

Option 2: resignature (new identity)

Disruptive step: resignaturing changes the datastore's UUID and label. Broadcom says not to resignature datastores that have running VMs, because the path to the VMs and disks changes and causes an outage. Registered VMs prevent the unmount and keep a location record that is wrong after the change, so they will not power on. There is no way back to the old signature. Treat it as a maintenance-window change.

Broadcom's sequence:

  1. Free the datastore of I/O. Either use Storage vMotion to move VMs off (placing the datastore in maintenance mode), or shut down and unregister the VMs and note their .vmx paths.
  2. If this copy was force-mounted earlier (the forceMountedInfo check above shows mounted = true), unmount it on every host, because Broadcom says a force-mounted datastore must be unmounted from all hosts before a resignature. Broadcom's unmount command takes a label (esxcli storage filesystem unmount -l <datastore_name>), and the ESXCLI reference also lists --volume-uuid | -u. Disruptive step: a snapshot copy carries the same label as the original, so a label can name your live production datastore. Take the UUID of the copy from esxcli storage vmfs snapshot list and unmount with -u <uuid>. Never unmount by a label that the original also uses. If the copy was never mounted, there is nothing to unmount, so skip this step. Back-out: a datastore unmounted by mistake can be mounted again from the vSphere Client or with esxcli storage filesystem mount, but running VMs on it will already have lost their storage.
  3. Resignature: esxcli storage vmfs snapshot resignature -l <label> or -u <uuid>. The volume is mounted immediately with a new UUID. Alternatively, add the storage on one host in the vSphere Client with "Assign a New Signature". Broadcom's option list also gives the esxcfg-volume -r form.
  4. Expect the new label. ESXi assigns snap-<snapID>-<oldLabel>. Rename it if you wish.
  5. Rescan on every host. Broadcom's article says a rescan does not cause a service interruption.
  6. Browse the new datastore, find each .vmx, register it in the inventory, and power it on. A VM that lives mostly elsewhere but has a disk here may need that disk removed and re-added from the new location.

Broadcom's vCenter-recovery case shows the same flow: resignature, rescan across the cluster, browse snap-XXXX-, register the .vmx, power on, rename.

If the datastore cannot be mounted or resignatured

For "duplicate extents found", Broadcom says to engage the storage vendor so only one snapshot LUN is presented, or to detach the duplicates. For "some extents missing", confirm the host can see all the LUNs of the multi-extent datastore before trying.

Never choose Format. When the Keep and Assign options are greyed out, the vSphere Client may offer only "Format the disk". Format creates a new, empty file system on the device, which is not a way to reach the data on it.

Confirm the fix

  • The datastore appears under /vmfs/volumes/ on every host after a rescan.
  • esxcli storage vmfs snapshot list no longer lists the volume.
  • Registered VMs power on and their disks open.
  • For a resignatured volume, the old datastore name is not referenced by any VM or backup job.

What to watch afterwards

  • Before storage changes, note each datastore's device ID and LUN ID. A change to either is what sets this off.
  • Present a replica to a test cluster, not to hosts that mount the original with the same signature.
  • Update job definitions, scripts and monitoring that refer to the old datastore name or UUID, since a resignature changes both.
  • Remove any force mount that was only meant to be temporary, by resignaturing in a window.

Risks and limits

  • These commands have not been run or lab-tested. They are taken from Broadcom's knowledge base articles and the ESXCLI reference.
  • Broadcom's sources name the UI route for ESXi 7.x and 8.x. Menu names may differ by build.
  • A force-mounted datastore that was expanded from the vSphere Client may have a grown partition table and an ungrown file system; Broadcom says to engage support to finish that from the command line.
  • Which option is right for a given site depends on whether the original is online, how many hosts are involved and whether VMs run on the datastore.

Sources