Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

ESXi alarm: Network uplink redundancy lost on a virtual switch

Technical article
VMware

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

vCenter raises Network uplink redundancy lost (Broadcom's vSphere 8.0 networking guide prints it as Host name or IP Network uplink redundancy lost) when only one physical NIC on the host is connected to a standard or distributed switch. The other NIC is down or was never assigned to the switch. Find the vmnic, read the log lines that name it, and fix the link, the team size, the failure detection setting or the driver. Do not unplug the NIC that is still up.

What the alarm means

Broadcom's guide says: "Only one physical NIC on the host is connected to a certain standard or a distributed switch. The redundant physical NICs are either down or are not assigned to the switch." Its example: a host has vmnic0 and vmnic1 on vSwitch0, vmnic1 goes offline, and only vmnic0 is left, so "the uplink redundancy for vSwitch0 is lost on the host." Broadcom's fix is to connect at least one more physical NIC to the switch and reset the alarm to green.

Broadcom's knowledge base lists the names under which the alarm is seen: "Network uplink redundancy degraded," "Network uplink redundancy lost," "Uplink redundancy on DVPorts degraded" and "Lost Network Redundancy" (KB 426494), and "Lost uplink redundancy" (KB 378014).

The alarm is about the number of working uplinks. It does not say the network is down. Traffic carries on over the one remaining uplink. But a failure of that uplink is now an outage.

NetLeverage runs VMware vCenter, vSphere and ESXi.

Likely causes

Cause What you see How to confirm
A vmnic has link down One NIC shows Link Status Down. Log says Uplink: vmnic# is down esxcli network nic list, then vobd.log
Cable, SFP+ module, switch port or driver fault behind that link The link is down, or goes down repeatedly KB 426494 lists driver, firmware, SFP+ module, cable and physical switch port
Only one NIC is assigned to the switch Alarm from the day the switch was built esxcli network vswitch standard list shows one uplink
Beacon probing with fewer than three uplinks Failed criteria: 160 in the log, and connectivity trouble Network Failure Detection: beacon in the failover output
Outdated NIC driver Several hosts alarm together, vmnics flap esxcli network nic get -n vmnicX shows the driver and version
Alarm not cleared after a very short link drop NICs are up, alarm remains vobd.log shows link down and up within a second; ESXi build is older than the fix

Prove the cause before changing anything

All of these are read-only. Run them in an ESXi shell on the host named in the alarm.

1. Which NICs are up. KB 426494 shows this command listing Admin Status and Link Status for each vmnic:

esxcli network nic list

2. Which uplinks the switch has. KB 326175 gives these two commands:

esxcli network vswitch standard list
esxcli network vswitch standard policy failover get -v vSwitch0

Read the uplinks, and the Network Failure Detection field. In KB 311349's example output the value is beacon when beacon probing is set.

3. What the host logged. KB 419622 gives /var/run/log/vobd.log as the log with the uplink events:

grep -E 'uplink.transition|linkstate' /var/run/log/vobd.log

KB 426494 shows lines of this form: [vob.net.pg.uplink.transition.down] Uplink: vmnic# is down. Affected portgroup: <portgroup-name>. 1 uplinks up. Failed criteria: 128. The Broadcom KB says "Failed Criteria 128 is the driver reporting a link state down." KB 419622 shows the same event with Failed criteria: 160, with vob.net.vmnic.linkstate.down lines and an esx.clear.net.vmnic.linkstate.up line when the link returns.

4. The driver. KB 432259 gives this command; note the Driver, Firmware Version and Version fields:

esxcli network nic get -n vmnic1

A good result is two uplinks, both with Link Status Up, and no down events in the last day. A bad result is one uplink, a vmnic with Link Status Down, or repeated down and up lines.

Fix

Pick the cause you proved.

The link is down: cable, SFP+ module, switch port

KB 426494 says the cause "can be caused by unplugging the network cable or administratively shutting down the physical switchport", that a port may have been disabled by loop detection, and that an unintended outage "will likely be an issue with the driver, firmware, SFP+ module, cable, and/or switchport of the physical switch". Its resolution is to check the driver, try replacing the cable to the physical switchport, and contact the host and switch hardware vendors.

First save the current state: run esxcli network nic list and keep the output, and ask the network team to write down the state of the switch port for the down vmnic.

Disruptive step: the next work is on the cabling of a host that has one working uplink. If the wrong cable is pulled, the host loses its network. Work only on the down vmnic's cable and switch port. Leave the uplink that is up alone.

Replace the cable of the down vmnic (KB 426494), or ask the network team to enable its switch port. Then run esxcli network nic list again and wait for Link Status Up.

To undo it: refit the original cable, ask the network team to return the switch port to the state they wrote down, and compare esxcli network nic list with the saved output.

Beacon probing with fewer than three uplinks

KB 419622 gives the cause as beacon probing enabled with fewer than three physical uplinks, and the resolution as "set Network Failover Detection to Link Status Only." KB 311349 says three or more NICs connected to three separate physical switches must be assigned to the team to configure beacon probing successfully. The teaming guide says link status alone "does not detect" some configuration errors, such as a physical switch port blocked by spanning tree or set to the wrong VLAN.

First save the current setting:

esxcli network vswitch standard policy failover get -v vSwitch0

Risk: the next command changes how vSwitch0 detects failed uplinks, for the management network and every port group that inherits the switch policy. After it, a fault that leaves the link up, such as a blocked switch port, is no longer detected by beacons.

KB 311349 gives this command, and the check:

esxcli network vswitch standard policy failover set -f link -v vSwitch0
esxcli network vswitch standard policy failover get -v vSwitch0

To undo it: if the saved output showed Network Failure Detection: beacon, run esxcli network vswitch standard policy failover set -f beacon -v vSwitch0 and read the policy again to confirm it matches the saved output.

Only one NIC is assigned

The remedy is a second NIC on the switch. The guide "vSphere HA warning: host currently has no management network redundancy" gives the command with its save-first step and back-out. The second NIC's physical port must be in the same Layer 2 broadcast domain as the first.

Driver

KB 432259 finds an unstable or outdated NIC driver as the cause when several hosts alarm at once and vmnics flap. Its fix is a driver version validated in the VMware Compatibility Guide, installed with the host in maintenance mode, followed by a host reboot. That is a change to the host's software, and it is described here, not given as a step. Plan it in a maintenance window.

Alarm will not clear

KB 301538 describes an alarm that is not cleared when the link drops and returns within a second. It says ESXi 7.0 Update 3o and ESXi 8.0 Update 1 resolve it. Broadcom's alarm page says to reset the alarm to green after redundancy is restored.

Example (illustrative): the alarm fires for esx03. esxcli network nic list shows vmnic0 Link Status Up and vmnic1 Link Status Down. vSwitch0 lists both as uplinks. vobd.log contains [vob.net.pg.uplink.transition.down] Uplink: vmnic1 is down. Affected portgroup: Management Network. 1 uplinks up. Failed criteria: 128. Failed criteria 128 is a link-down report from the driver, so the network team checks the port. The port is shut down on the switch. The network team enables it, esxcli network nic list shows Link Status Up for vmnic1, and the administrator resets the alarm to green.

Confirm the fix

  1. esxcli network nic list shows Link Status Up for every vmnic of the switch.
  2. esxcli network vswitch standard policy failover get -v vSwitch0 shows the uplinks you expect, and the detection method you intended.
  3. grep of vobd.log shows no new uplink.transition.down lines after the fix time.
  4. The alarm on the host is green.

What to watch afterwards

  • Several hosts alarming together points to a shared cause such as a driver, firmware or a switch. A single host points to its own cable or port.
  • Repeated down and up lines for one vmnic are a flapping link. Record the times before changing hardware.
  • With a link aggregation group (LAG), Broadcom's KB 378014 says the alarms "are not triggered" when one uplink of the LAG is down, because the LAG still has an uplink up. It describes this as by design, and gives a custom alarm with three LACP triggers: esx.problem.net.lacp.lag.transition.down, esx.problem.net.lacp.uplink.fail.speed and esx.problem.net.lacp.uplink.transition.down.
  • The guide "How to design and troubleshoot VMware networking" covers uplink, VLAN and MTU checks beyond this alarm.

Risks and limits

  • These commands have not been run or lab-tested.
  • Broadcom's alarm page does not say which events trigger the alarm by default. Check your vCenter alarm definitions.
  • The commands here are for standard switches. For a distributed switch, read and change the uplinks in the vSphere Client. KB 301538 and KB 426494 show the matching dvport events.
  • Broadcom's KB 318898 lists the Failed Criteria codes: 32 is beacon probing and 128 is the link state reported by the driver. KB 419622 shows 160 only as a log example tied to beacon probing, and KB 318898 does not list 160 on its own.
  • A pulled or flapping uplink can carry storage and vMotion traffic as well as management. Plan physical work for a time when losing the remaining path is acceptable.

Sources