vSAN Skyline Health shows the cluster partition check in red, an alarm says a host is partitioned, or virtual machines on one host have become inaccessible. Broadcom's knowledge base contains repair procedures that make a host leave and rejoin the cluster, rebuild unicast lists, move vSAN traffic to another adapter or reboot hosts. Each of those changes cluster membership or the storage network while data may already be at reduced availability.
This article is the step before anyone considers a repair: record what every host believes about membership, list every documented candidate cause the record supports, and hand that record to the cluster owner or Broadcom. It does not repair anything and does not choose a repair.
Scope
This covers vSAN 8.0 Original Storage Architecture (OSA) single-site standard clusters in unicast mode with IPv4 vSAN VMkernel adapters, where the cluster owner has confirmed the architecture.
If the cluster runs the Express Storage Architecture (ESA), or nobody can confirm which architecture it runs, complete Step 1 as a record and go straight to Step 4. The sources below are OSA cases or do not state an architecture, so they give no basis for a candidate cause on ESA.
Stretched and two-node clusters, witness hosts, vSAN Max and remote datastores, IPv6 and routed vSAN networks are out of scope.
What the health check is telling you
The vSAN 8.0 documentation describes the check this way: "This health check examines the cluster to see how many partitions exist. It displays an error if there is more than a single partition in the vSAN cluster."
So a red result means hosts disagree about who is in the cluster. It does not say which host is wrong, how many groups there are, or why. A single host that has lost its vSAN network forms a group of one; a cluster split by a version mismatch can form two groups of several hosts each. The membership record comes first because those situations need different owners.
Step 1: record membership on every host
Collect these from every host in the cluster, including hosts that look healthy. The commands display only. Record the host and the time of each capture, because membership can change while you collect.
Before you start, write down the expected cluster: each host's name, its vSAN node UUID and its vSAN VMkernel IP address, from the cluster owner's records.
From esxcli vsan cluster get (the command is used in KB 451794, vSAN 8.0 or later; the field names below are from the example outputs in KBs 326579, 388099 and 435368, which state no version, so record what your build actually prints):
| Field | What to record | Why |
|---|---|---|
| A | Sub-Cluster UUID | Hosts with the same value see themselves in the same group |
| B | Sub-Cluster Member UUIDs and Member HostNames, in full | Which hosts this host can see, by identity, not just by count |
| C | Sub-Cluster Member Count | Cross-check with B |
| D | Local Node State and Sub-Cluster Master UUID | Which host each group treats as its leader |
| E | Maintenance Mode State | A host in maintenance mode explains some changes |
| F | Architecture: OSA or ESA, as confirmed by the cluster owner | Decides whether Step 3 applies. KB 435368's example output includes the line "VSAN ESA Enabled: false"; that KB states no version |
Broadcom KB 326579 describes the partitioned case as one where "the member count is in reduced value when compared to actual member count value of the vSAN Cluster", and its example shows a host with "Sub-Cluster Member Count: 1". A count that matches is not enough on its own: compare field B with the expected host list.
From the other display commands:
| Field | Command | Source and version | What to record |
|---|---|---|---|
| G | esxcli vsan network list |
vSAN 8.0 TechDocs | Which VMkernel adapter carries vSAN traffic, or that none is listed |
| H | esxcli network ip interface list |
vSAN 8.0 TechDocs | MTU and switch of that adapter |
| I | esxcli network ip interface ipv4 get -i <vmk> |
vSAN 8.0 TechDocs | IP address and netmask of that adapter |
| J | esxcli vsan cluster unicastagent list |
KB 451794 (vSAN 8.0 or later) | Each entry's node UUID and IP address |
| K | ESXi version and build, from the vSphere Client host summary | — | Whether every host is on the same build |
The vSAN 8.0 page notes that esxcli vsan network list "continues to work even if vSAN has been turned off and the hosts no longer participate in vSAN", so an empty result is itself a finding. For field J, KB 451794 says each host's unicast list should hold "all other vSAN hosts, other than itself". KB 388590 adds: "The vSAN node should not have its IP address added to the unicastagent list in the data node." Compare each entry with the expected list by UUID and IP, not by number of entries.
Logs worth capturing
Copy these lines with their timestamps before anyone restarts a service or a host. The KBs that show them state no vSAN version, so a matching line is supporting evidence, and a missing line proves nothing.
/var/log/vmkernel.log: CMMDS membership lines. KB 326579 shows the form "Current membership uuid … has 1 members" when a host is on its own./var/run/log/vsansystem.log: membership updates with a node count. KB 435368 shows "nodeCount: 1"./var/run/log/vobd.log: KB 435368 shows "vSAN is no longer using vmknic vmk#. There are no vSAN vmknics remaining." when the vSAN tag has gone from the adapter.
Step 2: record the vSAN network path results
vmkping sends ICMP traffic on the production storage network. It is an active test, not a read. Use results the network or cluster owner already has. A new test is optional and needs its own authorisation from the cluster owner; this article does not ask for one.
For each result, record:
| Field | What to record |
|---|---|
| L | Source host and source VMkernel adapter |
| M | Destination host and destination vSAN IP |
| N | Time, packet count, ICMP payload size, whether don't-fragment (-d) was set |
| O | Packets lost |
The vSAN 8.0 page uses vmkping -I <vmk> <peer vSAN IP> and advises: "Consider adding -d to the vmkping command to test if packets can be sent without fragmentation." It also warns: "If jumbo frames are configured, this command does not report any issues if the jumbo frame MTU size is incorrect."
The -s value is the ICMP payload, not the packet. With the 20-byte IPv4 header and the 8-byte ICMP header, -s 1472 makes a 1,500-byte IPv4 packet and -s 8972 makes a 9,000-byte IPv4 packet. Neither number is an Ethernet frame size.
Record each direction separately. A result from esx-04 to esx-01 says nothing about esx-01 to esx-04, or about esx-04 to esx-02.
Step 3: list the candidate causes (confirmed OSA only)
Lay the fields out with one row per host, then group hosts by field A. Mark every row below that your record supports. More than one can be true at once: a host can lose its path during an upgrade that also mixed the builds. Each row names the evidence that would tell it apart from the others. None of the rows is a verdict, and none leads to a repair from this article.
| Record shows | Candidate cause | Source | Evidence that discriminates |
|---|---|---|---|
| Every host shows the same A, field B matches the expected identities on every host, all captured at compatible times, and the health check is now green | No partition seen at those times; the earlier red is unexplained | Labelled inference from the health check definition | Keep the logs and timestamps from the red period; recovery is not proof of cause |
| One host alone; G on that host lists no adapter | vSAN traffic tag missing from the adapter | KB 435368 (no version stated) | The vobd line above, and the owner's record of any adapter change at that time |
| One host alone; G lists an adapter; results from that host show loss at every size to named peers | Uplink, NIC, firewall or switch-side fault on that host's path | KB 326579 (no version stated) | Network owner's confirmation of link and switch-port state for that host's vSAN uplinks |
Small payloads pass and large ones fail with -d on named paths |
MTU mismatch or filtering of large packets on those paths | KB 444421 (vSAN 8.x OSA) | Network owner's confirmation of the configured MTU at each hop of those paths |
| One host alone after a driver or patch change; results to its peers pass | NIC firmware incompatible with the installed driver | KB 388099 (no version stated) | The host owner checks that NIC's driver and firmware versions against each other and the hardware vendor's compatibility statement. A passing ping plus a recent patch is not enough |
| Two or more groups, a host on a higher build added while builds were mixed | Sub-clusters formed around the new host | KB 451794 (vSAN 8.0 or later) | Field K per host and the time the host was added, from vCenter tasks |
| Groups split along 7.0 and 8.0 hosts during an upgrade | CMMDS TLS handshake failure between SHA-1 and SHA-256 hosts | KB 453752 (7.0 to 8.0 upgrades) | Field K per host; the cluster owner or Broadcom reads the certificate and CMMDS evidence that KB describes |
| Every host alone and results fail between all vSAN adapters | Network-wide fault or firewall issue on the vSAN uplinks | KB 388590 (vSAN 8.0.x, 7.0.x) | Network owner's view of the shared switches and firewalls |
| J lacks an expected peer, or holds the host's own address | Unicast list does not match the expected cluster | KB 451794, KB 388590 | A mismatch alone is a record, not a cause; note it with the time |
| None of the above, ESA, or architecture unconfirmed | Not covered here | — | Go to Step 4 |
Two sources change how network evidence is read. KB 388099 states "vmkping between the nodes may be successful." for a host partitioned by NIC firmware. KB 444421 records a case "confirming that small payloads passed while large payloads dropped". A passing ping does not close the network question, and a failing one does not rule out a version cause.
KB 453752 describes why version can matter: "the CMMDS daemon cannot establish a TLS handshake between peers." KB 451794 gives a prevention point that needs no command: "If you are actively doing an upgrade of the ESXi hosts in the vSAN cluster, finish the upgrades before adding a new host."
Step 4: hand the record over
Send the record to the cluster owner, and to Broadcom support where the owner opens a case. Include:
- the exact topology: hosts, expected UUIDs and vSAN IPs, the architecture (or that it is unconfirmed), and field K for every host;
- fields A–K per host with capture times, and fields L–O for every path result used;
- the log lines with timestamps;
- the candidate rows marked in Step 3, with their discriminating evidence, or a note that it has not been obtained yet;
- which VMs and services are affected, and what is known about their data protection and backups.
What this record does not authorise
The repair procedures in the sources include esxcli vsan cluster leave and join, esxcli vsan cluster unicastagent add and remove, re-tagging or adding vSAN traffic on another adapter (KB 388590 enables vSAN on the management network as a workaround), setting /VSAN/IgnoreClusterMemberListupdates, forced VM power-off and host reboots (KB 451794, KB 453752). None of them is a next step in this article, whichever row matched.
Each of them changes membership, the storage network or running workloads while objects may already be at reduced availability. Before any of them, the cluster owner or Broadcom needs: the exact topology and version, the state of the data and its protection, the workload impact, the prerequisites for the specific procedure, an abort and recovery plan, and validation by someone qualified in an isolated environment where the procedure allows it. Only a separately authorised change follows that review.
Worked example (fictional)
The values below are invented to show how the records are read. Cluster cl-prod runs vSAN 8.0 OSA, confirmed by the cluster owner, on ESXi 8.0 build 24000000 on all four hosts. vSAN runs on vmk2 with MTU 9000. A switch firmware update ran at 08:45 on 4 November; the partition check turned red at 09:10.
Expected cluster:
| Host | vSAN node UUID (last 4) | vSAN IP |
|---|---|---|
| esx-01 | a101 | 192.0.2.11 |
| esx-02 | a102 | 192.0.2.12 |
| esx-03 | a103 | 192.0.2.13 |
| esx-04 | a104 | 192.0.2.14 |
Membership and adapter records, captured 09:14–09:16:
| Host | A (last 4) | B: member UUIDs (last 4) | G | J: entries (UUID, IP) |
|---|---|---|---|---|
| esx-01 | 2f67 | a101, a102, a103 | vmk2 | a102 .12, a103 .13, a104 .14 |
| esx-02 | 2f67 | a101, a102, a103 | vmk2 | a101 .11, a103 .13, a104 .14 |
| esx-03 | 2f67 | a101, a102, a103 | vmk2 | a101 .11, a102 .12, a104 .14 |
| esx-04 | 7c0e | a104 | vmk2 | a101 .11, a102 .12, a103 .13 |
Path results supplied by the network owner from their own tests at 09:20, 5 packets each, -d set:
| From (vmk2) | To | -s 1472 loss |
-s 8972 loss |
|---|---|---|---|
| esx-04 | 192.0.2.11 | 0/5 | 5/5 |
| esx-04 | 192.0.2.12 | 0/5 | 5/5 |
| esx-04 | 192.0.2.13 | 0/5 | 5/5 |
| esx-01 | 192.0.2.14 | 0/5 | 5/5 |
What the record supports: esx-04 forms a group of one while the other three see each other by identity. Its adapter is tagged, its unicast list matches the expected peers, and all hosts are on one build, so the tag, unicast and version rows are not supported. On the four named paths, 1,500-byte packets pass and 9,000-byte packets fail. That supports the MTU or large-packet filtering candidate (KB 444421) on those paths. No result was supplied for paths between esx-01, esx-02 and esx-03, so nothing is said about them.
The handover to the cluster and network owners lists the four paths observed at 09:20, all with -d set: esx-04 vmk2 to 192.0.2.11, 192.0.2.12 and 192.0.2.13, and esx-01 vmk2 to 192.0.2.14. On each, -s 1472 lost 0/5 and -s 8972 lost 5/5. The paths from esx-02 and esx-03 to esx-04 were not tested, so their results are unknown. The 09:20 results are one sample: they do not show when the failures began or that they have continued since. MTU or large-packet filtering stays a candidate for the owners to confirm. The record does not ask for esx-04 to rejoin, and it does not show the switch update as the cause.
Acceptance checks
- The architecture is recorded as confirmed OSA, confirmed ESA or unconfirmed. Step 3 was used only for confirmed OSA.
- Fields A–K were recorded for every host, with capture times, and field B and field J were compared with the expected hosts by UUID and IP.
- Every path result used names its source, destination, direction, time, count, payload size and loss, and says whether it came from the owner or from a separately authorised test.
- Every matching Step 3 row is listed as a candidate with its discriminating evidence, or a note that it has not been obtained yet. An unresolved record is an acceptable result.
- Nothing was changed while reading. Because only reads were made, there is nothing to roll back.
- The record went to the cluster owner or Broadcom. Any membership, network or host change is a separate, authorised change with its own review.
Learning notes
- A partition alarm says hosts disagree about membership. It does not say which host is wrong or why.
- Compare members by identity. A correct count can hide the wrong members.
vmkping -ssets the ICMP payload. Add 28 bytes for the IPv4 packet.- A passing small ping does not clear the path, and a passing large ping does not clear the firmware or version candidates.
- Several candidates can be true together. Keep all of them until evidence separates them.
Related articles
- How to diagnose VMware and vSAN storage latency, for vSAN performance problems that are not a partition.
- How to design and troubleshoot VMware networking, for the VMkernel and physical network design behind the vSAN path.
- A healthy vSAN witness cannot replace missing VM data (proposed: https://netleverage.com/ffb9917-articles/vsan-witness-online-inaccessible-object-data-quorum.html; link active only once that article is published), for stretched-cluster quorum.
Sources
All retrieved 2026-11-03 UTC; dates are the page's own "Last Updated" or knowledge base date. Related-article pages were checked HTTP 200 on 2026-11-03 UTC.
- Broadcom TechDocs, vSAN 8.0, "Troubleshooting the vSAN Network", last updated 9 September 2026. https://techdocs.broadcom.com/us/en/vmware-cis/vsan/vsan/8-0/vsan-network-design/troubleshooting-the-vsan-network.html
- Broadcom KB 326579, "vSAN cluster partition after network uplink failover", 29 June 2026. https://knowledge.broadcom.com/external/article/326579/vsan-node-is-in-cluster-partitioned-stat.html
- Broadcom KB 435368, "vSAN cluster partition after vSAN untagged from vmkernel network adapter", 9 September 2026. https://knowledge.broadcom.com/external/article/435368/vsan-cluster-partition-after-vsan-untagg.html
- Broadcom KB 451794, "After adding a new host to the vSAN cluster and removing another, we are now encountering a 'Partitioned Cluster' error.", 31 August 2026. https://knowledge.broadcom.com/external/article/451794/after-adding-a-new-host-to-the-vsan-clus.html
- Broadcom KB 453752, "vSAN Cluster Partition and Inaccessible VMs After ESXi Upgrade from 7.0 to 8.0 vSAN", 26 August 2026. https://knowledge.broadcom.com/external/article/453752/vsan-cluster-partition-and-inaccessible.html
- Broadcom KB 444421, "vSAN Objects Inaccessible due to MTU Mismatch causing Cluster Partition", 6 August 2026. https://knowledge.broadcom.com/external/article/444421/vsan-objects-inaccessible-due-to-mtu-mis.html
- Broadcom KB 388590, "How to fix vSAN Cluster partition in case of ongoing network issues on vSAN vmkernel uplink", 6 March 2025. https://knowledge.broadcom.com/external/article/388590/how-to-fix-vsan-cluster-partition-in-cas.html
- Broadcom KB 388099, "vSAN Node network partitioned", 18 September 2026. https://knowledge.broadcom.com/external/article/388099/vsan-node-network-partitioned.html