Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

NFS datastore inaccessible on ESXi: read 'esxcli storage nfs list', then test the network path and the export

Technical article
VMware

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

When vCenter shows an NFS datastore as inaccessible, do not remount it yet. Run esxcli storage nfs list on the host first. The Accessible and Mounted columns separate three different faults: a mount that exists but cannot reach the server, a mount that is gone after a shutdown or reboot, and a mount the server refuses.

Remounting before you know which one you have can hide the cause, and removing a datastore that still has running VMs is disruptive. Read first, then change.

What the error means

ESXi keeps an NFS mount definition and a live connection to the server. The esxcli storage nfs list output has a column for each. In the Broadcom example for NFS 3 or NFS 4.1 on vSphere 7.x and 8.x, the columns are Volume Name, Host, Share, Vmknic, Accessible, Mounted, Connections, Read-Only, isPE and Hardware Acceleration (KB 410891).

  • Accessible false, Mounted true: the mount is defined, but the host cannot talk to the server. KB 410891 traces this to the network between host and storage. KB 339587 documents the same pair after a network change, with vmkping to the storage IP showing 100% packet loss.
  • Accessible false, Mounted false: the mounts are not active. KB 394144 shows this for every NFS datastore after a site was restored from a maintenance window, with the network fine.
  • A mount that fails with an error: the host reached the server, which refused the mount (for example "Permission denied" for NFS 4.1), or the host could not connect at all.

The log files are /var/run/log/vmkernel.log and, at boot, /var/run/log/boot.log. A host that has lost the server logs lines such as WARNING: NFS: 331: Lost connection to the server nfs-server mount point /share (KB 339587) or, for a failed connection, NFS Error: Unable to connect to NFS server and WARNING: NFS: 898: RPC error 12 (RPC failed) trying to get port for Mount Program (100005) Version (3) Protocol (TCP) (KB 323107).

Likely causes

Cause What you see How to confirm
Packet loss or MTU mismatch on the path Accessible false, Mounted true; high latency, VMs slow or unresponsive vmkping with the do-not-fragment flag shows loss; esxcli network nic stats get shows CRC or carrier errors
Server IP unreachable after a network change Accessible false; Lost connection to the server in vmkernel.log vmkping to the NFS IP gets 100% loss
Port 2049 or 111 blocked, or export refuses the host Unable to connect to NFS server, RPC error 12 nc -vz <array-IP> 2049 times out; check the export for no_root_squash and access
Mounts not restored after a site shutdown Accessible false and Mounted false, network fine esxcli storage nfs list; vmkping succeeds
NFS 4.1 datastore missing after every host reboot (ESXi 8.x) Datastore has 0 bytes; boot.log shows NFS41_FSMount failed: Timeout /var/run/log/boot.log
NFS 4.1 mount refused: mount sent from the management IP Mount failed: Unable to complete Sysinfo operation, then Permission denied vmkernel.log NFS41FSGetRootFH lines; array allows only the NFS VMkernel IP

Prove the cause before changing anything

These commands are taken from the Broadcom pages cited and have not been run.

1. Read the mount state.

esxcli storage nfs list

Good: Accessible and Mounted both true. Bad: either column false. A bad pair of false and true points at the path (steps 2 to 4). A false and false pair with a working path points at restoring the mounts.

2. Find the NFS VMkernel port and its MTU. Use esxcfg-vmknic -l and note the vmk and MTU (KB 323107).

3. Test the path from that port.

vmkping -I vmkN -s <size> -d <NFS-server-IP>

The -s value is the MTU minus 28 bytes: 1472 for MTU 1500, 8972 for MTU 9000 (KB 344313). -d sets the do-not-fragment bit, so a correct-size reply also proves the MTU. Good: 0% packet loss with replies. Bad: 100% packet loss (path down), or partial loss. KB 410891 shows a bad case where a 2000-packet test returned only 20 replies:

vmkping -I vmk1 <NFS IP> -d -s 1472 -c 2000 -i 0.01
2000 packets transmitted, 20 packets received, 99% packet loss

If the datastore uses a separate TCP/IP stack, KB 344313 notes -S with the stack name (for example -S vmotion) for the vMotion stack; use the stack your NFS port actually belongs to.

4. Look for physical errors.

esxcli network nic stats get -n vmnic1

KB 410891's bad example shows Receive CRC errors of 626,429 and Transmit carrier errors of 146,293 on the uplink, with zero dropped packets. Counters like these point at a cable, optic or switch port. They need the hardware vendor or network team, not an NFS change.

5. Test the NFS port.

nc -vz <array-IP> 2049

Good: Connection to ... 2049 port [tcp/nfs] succeeded!. Bad: Connection timed out (KB 323107). KB 308081 adds that ports 111 and 2049, TCP and UDP, must be open at the host, storage and physical switch levels.

6. Read the logs. Search /var/run/log/vmkernel.log for the lines in the table. At boot, /var/run/log/boot.log holds the NFS 4.1 mount failures.

Example (illustrative): host esx01 shows the datastore nfs-prod01 as inaccessible. esxcli storage nfs list shows Accessible false, Mounted true. vmkping -I vmk2 -d -s 8972 10.0.30.10 returns 100% packet loss, but -s 1472 succeeds. The physical ports are set to MTU 9000 on the array side only. The cause is an MTU mismatch on the path. Remounting would not have helped.

Fix

Path or packet loss

Fix the cause on the physical network (MTU on every hop, cable or optic, port errors). KB 339587 says the volume reconnects automatically once the server is reachable, and KB 410891 sends the CRC and carrier-error case to the hardware vendor and the physical network team.

If the original path is permanently gone, KB 339587 gives this workaround:

  1. Power off and unregister all VMs on the datastore.
  2. Unmount the inaccessible datastore from the host.
  3. Remount it using a reachable server IP.
  4. Confirm it returns to Operational status, then re-register and power on the VMs.

Disruptive step: this takes every VM on the datastore offline. Do not do it with running VMs. To back out, remount using the original IP once it is reachable again.

KB 323107 adds that if the server moves to a new IP, VMs show as inaccessible because they reference the old server's UUID, so unregister and re-register them on the new volume. If the server keeps the same IP, the volume becomes inaccessible and APD errors appear because the host does not see the backend change, and that article says to reboot the host. Disruptive step: a host reboot means entering maintenance mode first and moving or powering off its VMs.

Mounts not active after a shutdown

If esxcli storage nfs list shows false and false, and vmkping -I <vmk> -d -s 1472 <NFS target IP> is clean, run:

esxcfg-nas -r

KB 394144 documents this as the restore. In that case esxcli storage nfs add -H <host> -s <share> -v <volume> failed with "Volume with label : ##############-01 already exists." After the restore, the list shows true and true. KB 308081 gives the same command for ESXi 9.0 and earlier and esxcli storage nfs restore for ESXi 9.1 and later.

Port or export problems

Open ports 111 and 2049 end to end. On the host, esxcli network firewall ruleset list --ruleset-id nfsClient shows whether the NFS client rule set is enabled, and esxcli network firewall ruleset allowedip list --ruleset-id nfsClient lists the addresses it allows. Broadcom's ESXCLI firewall commands page describes allowedip list as "List the allowed IP addresses of the specified rule set." For NFS v3, ESXi manages this rule set itself. Broadcom's NFS Client rule set behaviour page says: "When you add or mount an NFS v3 datastore, ESXi checks the state of the NFS Client (nfsClient) firewall rule set." If the rule set is off, ESXi activates it, and "The IP address of the NFS server is added to the allowed list of outgoing IP addresses." Manual settings do not last: they "are overridden when the last NFS v3 datastore is unmounted." So the useful check is whether the NFS server's address is in the allowed list. NFS 4.1 uses a different rule set, nfs41client. On the array, confirm that the export allows the host with no_root_squash and read/write, that the server name has unique DNS records pointing at the right address, and that LDAP, if used for share permissions, is reachable (KB 308081). If the share is mounted over UDP, KB 323107 shows Server (...) does not support Mount Program (100005) Version (3) Protocol (TCP); use TCP.

Risk: a manual esxcli network firewall ruleset set --ruleset-id nfsClient --enabled true changes the host firewall and is a temporary test only, because ESXi overrides it when the last NFS v3 datastore is unmounted. Do not switch the rule set off to back out. --enabled false deactivates the rule set that every mounted NFS v3 datastore on the host depends on. A firewall change on the array or switch affects every host using it. Change one thing at a time and test again.

NFS 4.1 datastore missing after each reboot (ESXi 8.x)

KB 416172 describes the cause: the vmnic carrying NFS traffic is not fully initialised when ESXi tries the mount at boot, and NFS 4.1 does not retry on ESXi 8.x. Its log strings are socket disconnected, Failed to wait for the cluster to be located: Timeout, NFS41_FSMount failed: Timeout and Jumpstart plugin restore-nfs-volumes activation failed. The manual remount is:

esxcli storage nfs41 remove -v <Datastore_name>
esxcli storage nfs41 add -H <NFS_server_IPaddress> -s <Mountpoint_name> -v <Datastore_name>

The KB says an NFS 4.1 retry arrives in ESX 9.0, advises fixing the network delay on the NFS vmnic, and does not say whether the manual remount survives the next reboot, so check the datastore after each restart. Disruptive step: removing the datastore needs it free of running VMs.

NFS 4.1 "Permission denied"

KB 419497 (ESXi 8.0.x) describes a mount that fails with Mount failed: Unable to complete Sysinfo operation and Permission denied, with vmkernel.log lines NFS41FSGetRootFH:4480: Lookup ... failed for volume nfs41: Permission denied. The cause is that ESXi sends the mount from the management IP, while the array allows only the NFS VMkernel port's IP. Options in that KB: create a VMkernel port in the NFS server's subnet and add its IP to the export, or on ESXi 8.0 Update 3 and later use vmkportbind (KB 380337), or on earlier versions add a static route (KB 308786). Check the path with vmkping -I <vmkX> <NFS server ip address>.

Confirm the fix

  1. esxcli storage nfs list shows Accessible true and Mounted true for every datastore.
  2. vmkping with the full-size do-not-fragment test shows 0% loss, and nc -vz <array-IP> 2049 succeeds.
  3. The datastore shows as available in the vSphere Client on every host that mounts it.
  4. For the boot case, reboot one host in a maintenance window and check the list again.

What to watch afterwards

  • NFS latency. KB 410891 shows NFS41 volume #### performance has deteriorated messages and a high DAVG in esxtop ahead of the failure.
  • NIC CRC and carrier error counters on the NFS uplinks.
  • MTU consistency across host, switches and array.
  • Network or storage maintenance windows that restart the array's NFS service: check the mount state afterwards.

Risks and limits

  • Version scope is as listed in each KB. The firewall commands are from the vSphere 8.0 documentation; check the rule set name with esxcli network firewall ruleset list on your build.
  • Broadcom's own pages disagree on one detail: KB 308081 describes RPC status 13 as permission denied in one place and as a timeout in its log line. Read the actual text in your log.
  • esxcfg-nas -r and esxcli storage nfs restore are given by KB 394144 and KB 308081 for different versions. Use the one that matches your build.
  • Block-storage APD and PDL are a different fault with different commands.

NetLeverage runs its own hosting on VMware vSphere, vCenter and ESXi. Talk to an engineer.

Sources