Software iSCSI Paths Dead After a NIC Change: Check the 1:1 Uplink Policy on Bound Ports

An ESXi 8.0 host uses the software iSCSI adapter with port binding. Two or more VMkernel ports sit on one vSwitch, in the same subnet as the array's target portals. Someone changes the host's networking: a failed physical NIC is replaced, a new NIC is added to the switch, or the teaming order of a port group is edited. Afterwards some paths to an iSCSI device show Dead, or a rescan takes much longer than it used to.

Port binding depends on each bound VMkernel port having exactly one active physical adapter. A NIC added to the switch can quietly change that. The common reflex is to remove and re-add the binding or rescan until paths return. Each of those is a change and replaces the state you need to record. This guide shows how to record each bound port as it is now, compare it with Broadcom's documented rules, and hand a specific, dated finding to the network, host and array owners.

This guide is based on source research only. Nothing in it was run. Host, adapter, address and IQN values below are fictional.

Scope: same subnet only

This guide applies only when all of the following are recorded:

• every bound VMkernel port is on the same vSwitch;
• every bound VMkernel port and every target portal is in the same IP subnet;
• the network owner confirms that the ports and portals share one broadcast domain (VLAN), because a matching subnet mask alone does not show that.

If any target portal is in another subnet, or reaching it depends on a static route or a gateway, stop. That is a routed design, and this guide gives it no verdict: not supported, not unsupported, not unreachable. The vSphere 8.0 sources cited here do not agree on it:

A routed design needs a version-specific answer from Broadcom or the array vendor for that exact topology. Record the topology and hand these four references to the design owner.

What Broadcom documents for the same-subnet case

Two things follow for diagnosis:

• The uplink policy of each bound port is readable without changing anything. A port with no active uplink, two active uplinks or a standby uplink is non-compliant as read, whatever caused it.
• A warning sign is not, by itself, a specific defect. One listed cause, "The active physical adapter is changed", can apply to a port that still has exactly one active uplink.

Before you start

• Read access to the host in the vSphere Client: Configure > Storage > Storage Adapters (the software iSCSI adapter, its Network Port Binding tab and the View Details icon for each bound adapter, as in Review Port Binding Details on the ESXi Host), its Dynamic Discovery and Static Discovery tabs, and Configure > Storage > Storage Devices, Paths tab (View Storage Device Paths).
• Read-only ESXi Shell output, if the host owner provides it: esxcli iscsi session list (Review iSCSI Sessions) and esxcli network ip interface ipv4 address list for each VMkernel port's address, netmask and gateway (best-practices page). Both list state; neither changes it.
• The host's logs for the period since the change: vmkernel.log and syslog.log (KB 400677).
• The change record: what was changed, on which switch, port group, uplink or VMkernel port, by whom and when. Keep any retained export of the port group's teaming policy from before the change. Without one, the earlier policy is "unknown".
• The network owner's record of the VLAN and broadcast domain for the bound ports and the portals.
• The array's view: its target IQNs and portal addresses, and which initiator connections it shows, with the time it was read. KB 415729 shows the array can see a different picture from the host.

Do not remove or add a port binding, rescan, change an uplink order or a service on a VMkernel port, or reconnect storage while you collect this. A rescan changes discovery state (the discovery page above) and can take a long time in a broken topology (KB 317719). Every change replaces the state you are trying to record.

Step 1: record each bound port

Record the host, ESXi version and build, the software iSCSI adapter (for example vmhba64), and the time of observation with time zone. Then, for each VMkernel port on the Network Port Binding tab:

Then once for the adapter:

vmkping from a bound port is an active test, not part of this record. If an owner runs it under their own change process, record the command, time and result as a separate observation. A reply shows the packet got through at that moment; a failure does not by itself say why.

Step 2: match the record

Check every row and record each one that applies. Do not stop at the first match: an uplink fault and a Management-service fault can exist at the same time. A row whose evidence is missing is marked "candidate"; a row you could not check is marked "not checked".

Four paths from two bound ports and two portals is the documented normal case, not a duplicate. A session or path count that matches G2 shows that the expected number exists at that moment; it does not show that each pair carries I/O.

LACP or another link aggregation on the uplinks of a bound port is outside KB 317719's rules; record it if you see it.

Worked example (fictional)

Host esx-a03, ESXi 8.0, software iSCSI adapter vmhba64, observed 3 November 09:40 AEDT. Two bound ports, vmk3 and vmk4, on vSwitch2, port group VLAN 120, both in 192.0.2.0/24; the network owner's record puts both ports and the array portal in the VLAN 120 broadcast domain. Dynamic Discovery lists 192.0.2.50; Static Discovery lists one target, iqn.2001-04.example:fin-lun, at 192.0.2.50:3260, matching the array's list read at 09:50. The change record says that on 2 November at 14:10 a failed physical NIC was replaced and vmnic5 was added to vSwitch2. No teaming-policy export from before the change was kept.

Reading it:

• Scope conditions are met.
• vmk4 has two active uplinks as read at 09:40 on 3 November. That is non-compliant under the 8.0 troubleshooting page, and it is proven for the time of observation.
• vmk4's logins to 192.0.2.50 fail from 14:26 on 2 November, after the 14:10 change. The timing is consistent with the change, but B2 is unknown, so this record does not prove the policy was correct before it.
• One Dead path is recorded for naa.600a…01 only. Which bound port it uses is unknown, so the Dead path is not tied to vmk4 by this record, and other devices are not assessed.

The finding handed on: "vmk4 on esx-a03 has two active uplinks (vmnic3, vmnic5) as read at 09:40 AEDT on 3 November, which is outside the 1:1 policy. Its iSCSI logins have failed since 14:26 on 2 November, after vmnic5 was added at 14:10. The earlier policy is not recorded. Device naa.600a…01 has one Dead path whose port is not identified." Restoring a 1:1 policy is the network owner's change. Whether the Dead path returns afterwards is a separate check.

Contrasting records (fictional)

esx-b01 has two bound ports and two portals in one subnet and VLAN. Static Discovery lists one target at both portals. G shows four sessions and H shows four paths for each device: the documented normal case, not a duplicate. A storage event later turns all four paths Dead, with no warning sign, every B one active uplink and no login failures logged. No row except the last applies: no configuration-rule mismatch is established and the path loss stays unexplained. The device state is classified on its own evidence.

esx-c02 has its array portals renumbered into 198.51.100.0/24 while its bound ports stay in 192.0.2.0/24. The scope conditions are not met. This guide records the topology and hands the four scope references to the design owner without a verdict.

Troubleshooting

Acceptance checks

1. The scope conditions are recorded, including the owner's broadcast-domain confirmation; a record outside scope is handed on with the four references and no verdict.
2. For each bound port: A to E are recorded, with "unknown" or "not available" where a value could not be read.
3. F to J are recorded once for the adapter, with the host, ESXi version and build, and the time of observation; targets are listed by name and portal, not only by discovery server.
4. Every Step 2 row is marked as applies, candidate, not applicable or not checked; more than one finding can stand.
5. A policy finding states the observation time; a start time or cause is stated only with B2 or J.
6. No count is accepted as proof that every port-and-portal pair carries I/O.
7. No port binding, route, uplink, service, VLAN or storage setting was changed, and no rescan was run, while collecting the record.
8. Any fix is handed to its owner as a separate change, and path recovery is confirmed by its own check of the Paths tab and sessions.

Risk notes

• Removing or adding a port binding changes the host's paths. KB 415729 shows the path set changing after a binding was removed and the host rescanned.
• A rescan changes discovery state and can be slow. The discovery page shows targets returning to the Static Discovery list on rescan, and KB 317719 documents longer rescans in a wrong topology. Collect the record first.
• Adding Management to an iSCSI-bound VMkernel port is the unsupported configuration in KB 421153; the host can flap in vCenter.
• Link aggregation on iSCSI uplinks is outside KB 317719's rules.

These are source-documented behaviours. Expert review and a lab check are still needed before any change to a vSphere environment.

Learning notes

• A bound port needs exactly one active uplink. Adding a NIC to the switch can quietly break that, and the current policy is readable without changing anything.
• A warning sign tells you to look, not what is wrong.
• Discovery servers and target portals are different things; count sessions and paths against named port-and-portal pairs.
• Per-port login lines in syslog.log name which port fails, which the path list alone may not.
• Routed designs with port binding are where the sources disagree; they need their own answer, not this table.

References

Related articles

• Design and troubleshoot VMware networking, for VMkernel networking and frame-size tests.