Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

VM on a distributed port group has a link but passes no traffic: tell a blocked port from a physical trunk or uplink fault

Technical article
VMware

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

A VM that is connected to a distributed port group but cannot talk to anything has one of two problems. Either the host has blocked the VM's virtual port, or the frames leave the host and the physical network drops them. A capture at the VM's switch port and a second capture at the uplink tell you which, and they take a few minutes.

Work from the host outwards. Read the VMkernel log for a blocked port first, then capture at the uplink. Only change the port group, the uplinks or the physical switch once you know where the frames stop.

What the error means

A vSphere Distributed Switch (vDS) is managed in vCenter, but each ESXi host enforces the configuration on its own virtual ports. A VM's network adapter connects to one of those ports. "Connected" in the VM settings only says the adapter is attached to a port. It does not say frames get through.

Frames can stop in three places:

  1. At the virtual port, inside the host. The host enforces the port group's security policy. Broadcom documents a case where a port is blocked even though the port group security settings are set to Accept. In that case, KB 317477 says the MAC Learning feature introduced in vCenter 8.0 was "not propagated properly to the host", so the host behaved as if MAC address changes were denied. It lists the fix as ESXi 8.0 Update 2b, build 23305546.
  2. On the uplink and the first physical switch port. The physical port is not carrying the port group's VLAN, or it treats that VLAN as untagged. KB 451775 describes this as VMs that talk to each other on the same host and port group but cannot "traverse the physical network", for "VMware vSphere ESXi 8.0 and higher".
  3. On one of several uplinks. One uplink is cabled or configured correctly and another is not. KB 438551 describes intermittent or total loss for VMs on a specific host, where traffic pinned to the faulty uplink fails.

A symptom shared by the last two is that the VM's MAC address never appears in the physical switch's MAC address table. KB 447264 lists this as "MAC addresses of the affected VMs are not populating in the upstream physical switch MAC address table".

Treat the version scope above as what Broadcom states, not as a guarantee for a build that is not named.

The subject here is dropped frames. A vDS health check warning about an uplink that a port group does not use is a separate fault with its own article, on the health-check VLAN and unused-uplink warning.

Likely causes

Cause What you see How to confirm
Virtual port blocked by the host (ESXi 8.0 before 8.0 Update 2b, MAC Learning disabled) VM has no traffic in either direction; VMkernel log names the VM and the port vmkernel.log contains Port is blocked; net-dvs -l shows MAC change denied
VLAN missing from the physical trunk VMs on the same host and port group talk; nothing leaves the host Capture at the uplink shows frames sent but none returned; physical port allowed list lacks the VLAN
Port group VLAN equals the physical native VLAN Untagged frames are dropped by the vSwitch Port group VLAN ID compared with the native VLAN on the physical port
One uplink wired or configured differently Failure only on some hosts or some VMs; moving the VM to another host restores traffic esxtop network view shows which uplink the VM uses; forcing the other uplink changes the result
Guest or VM adapter fault No frames at the VM's switch port at all Capture at the switch port shows nothing from the VM

Prove the cause before changing anything

Everything in this section is read-only apart from the packet captures, which only read traffic. Run it over SSH as root on the ESXi host that runs the affected VM. These commands have not been run or lab-tested.

Step 1: look for a blocked port

Search the VMkernel log for the two lines Broadcom documents in KB 317477:

grep -i "Port is blocked" /var/log/vmkernel.log
grep -i "disallowed by vswitch policy" /var/log/vmkernel.log

The lines read client <vmname> requested mac address change to <mac> on port 0x# #######, disallowed by vswitch policy and client <vmname> has policy violations on port 0x#######. Port is blocked, and Broadcom shows them beginning with etherswitch: L2Sec_EnforcePortCompliance:.

  • Bad result: a line naming your VM. The host has blocked the port, so the physical network is not the problem.
  • Good result: no such line. Move on to the capture.

You can also look at the port's security settings as the host sees them. Broadcom gives net-dvs -l. In the affected case it shows com.vmware.vswitch.port.security = deny promiscuous; deny mac change; deny forged frames and Allow MAC Change = False, even though vCenter shows the port group with Accept. Compare what the host shows with what vCenter shows. A difference is the fault described in that article.

Step 2: find the VM's switch port and uplink

List switch ports with net-stats -l. KB 393957 uses it to map a VM or VMkernel adapter to a switch port number. It then offers three ways to find the uplink a port uses:

  • run esxtop, press n, and read the switch port to uplink mapping;
  • run vsish, then cat /net/portsets/vSwitch_name/ports/switchport_number/teamUplink, using the vSwitch name and port number from net-stats -l;
  • run netdbg vswitch instance list.

KB 438551 uses the esxtop network view and its TEAM-PNIC column, which names the physical NIC in use.

Step 3: capture at the uplink

KB 447264 gives this command for a VM whose MAC address is missing upstream. Replace the uplink name and the MAC address:

pktcap-uw --uplink vmnic#### --capture UplinkSndKernel,UplinkRcvKernel -o - | tcpdump-uw -enr - | grep -i "VM-MAC-ADDRESS"

Start the capture, then make the VM try to reach something, such as its gateway. Read the result like this, using Broadcom's two scenarios:

What you see What it means Next step
Frames from the VM's MAC on the send side (Snd) but none returning (Rcv) Frames leave the host. The physical infrastructure drops or blocks the traffic or its return Check the physical trunk, VLAN and native VLAN
No frames from the VM's MAC on the send side Nothing leaves the host. Check the guest and whether a security policy such as MAC address changes or forged transmits is blocking the port Go back to Step 1; capture at the switch port

Step 4: capture at the VM's switch port

KB 393957 shows a capture at a switch port, filtered here for ICMP:

pktcap-uw --switchport <switchport> --dir 2 -o - | tcpdump-uw -enr - | grep -i icmp

Use the switch port number from net-stats -l. Broadcom's example does not say what --dir 2 selects, so run pktcap-uw -h on your build before relying on it. Broadcom's interpretation is that ICMP lines mean traffic reaches the physical NIC and the VM's switch port. With no output, traffic is not reaching the NIC.

For the no-output case, KB 393957 points to the NIC counters:

esxcli network nic stats get -n vmnic#

A healthy NIC shows no transmit drops or errors. If the NIC looks clean, Broadcom says to look upstream, at the top-of-rack switch or a router.

Step 5: check the VLAN settings on both sides

Following KB 451775, in the vSphere Client open the host, then the virtual switch settings, and find the port group the VM uses. Confirm two things:

  • the port group's VLAN ID is one the physical port allows;
  • the port group's VLAN ID is not the native VLAN of the physical port. Broadcom says a match "caus[es] the vSwitch to drop the untagged ingress packet".

The physical side has to be checked on the switch itself. Ask the network team for the trunk's allowed VLAN list and native VLAN on every port the host's uplinks connect to.

Example (illustrative): VM app01 on host esx01 sits on a distributed port group using VLAN 120 and cannot reach its gateway 10.0.20.1. grep -i "Port is blocked" /var/log/vmkernel.log returns nothing, so the port is not blocked. esxtop shows app01 using vmnic1. The capture on vmnic1 shows frames with app01's MAC address on the send side and none on the receive side, which is Broadcom's first scenario. The network team finds that the switch port for vmnic1 allows VLANs 100 and 110 only. VLAN 120 is added to the allowed list and the capture then shows return traffic.

Fix

Fix only the cause the checks identified.

Blocked port (Step 1 found the log lines)

  1. Check the vCenter and ESXi build. Broadcom lists the fix as ESXi 8.0 Update 2b, build 23305546, so a host on an earlier 8.0 build is affected by this fault.
  2. Update the host to a fixed build through your normal patching process. Disruptive step: patching normally needs the host in maintenance mode and a reboot, so VMs must be migrated or shut down first. Back out by returning to the previous image through your normal rollback process, if you hold one.
  3. The workaround Broadcom documents is to use a standard virtual switch. Disruptive step: moving a VM to a standard switch port group changes its network connection and can interrupt it. Back out by reconnecting the adapter to the distributed port group. Do this only when patching is not possible soon.

VLAN missing from the trunk, or native VLAN clash

  1. Have the network team add the VLAN to the allowed list on every physical port the host's uplinks use. Risk: a trunk change on a shared port affects every VLAN on it. Add the VLAN to the allowed list rather than replacing the list, and back out by removing only what you added.
  2. If the port group's VLAN is the physical native VLAN, either change the physical port's native VLAN or move the port group to a different VLAN. Disruptive step: both change tagging for every device on that VLAN. Plan a maintenance window and keep the old configuration to restore.
  3. If the fabric is HPE Synergy, KB 447264 says to check that the VLAN is mapped in HPE OneView in the Logical Interconnect Group and Uplink Set, and associated with the network set on the server profile.

One uplink wrong

KB 438551 tests the second uplink by forcing traffic onto it: set the identified vmnic to Unused or Down at the port group level. Disruptive step: this moves all traffic for that port group off the uplink and can drop connections for every VM using it. Back out by restoring the original teaming order. Use it as a test, then correct the physical switch port or cabling that is wrong, and restore the teaming policy.

Confirm the fix

  1. Repeat the capture from Step 3. You should see frames from the VM's MAC on both Snd and Rcv.
  2. Check the physical switch: the VM's MAC address should now appear in the MAC address table on the port the VM's uplink connects to.
  3. Ping the VM's default gateway from inside the guest.
  4. For a blocked-port fix, search vmkernel.log again for Port is blocked and look at its timestamps. Lines older than your change are expected. New lines are not.

What to watch afterwards

  • New VLANs. KB 447264 gives a newly provisioned VLAN missing from the physical trunk as the cause. Add the physical trunk change to your VLAN provisioning checklist.
  • Hosts patched at different times. The blocked-port fault depends on the ESXi build, so a cluster with mixed builds may show it on only some hosts.
  • Uplink wiring after hardware changes. A swapped or misconfigured uplink produces failures that follow the host, not the VM.

Risks and limits

  • KB 451775 says "VMware vSphere ESXi 8.0 and higher"; the others name ESXi or vSphere without other version numbers, apart from the fixed build in KB 317477.
  • The Broadcom articles show pktcap-uw by example. They do not explain --dir 2 or list every capture point, so check the options on your own build.
  • Capturing on a busy uplink produces a lot of output. Filter by MAC address as shown.
  • Broadcom's workaround for the blocked port, a standard switch, is a design change, not a repair.

NetLeverage runs its own hosting on VMware vSphere, vCenter and ESXi. Talk to an engineer.

Sources