Australian cloud for business software — built and run in Sydney since 2003

NetLeverage
NetLeverage

A healthy vSAN witness cannot replace missing VM data

Technical article
VMware

A connected witness does not prove that a vSAN stretched-cluster object has enough usable data to remain accessible. The witness stores metadata that helps vSAN decide availability; it does not hold another copy of the VM's user data. Diagnose the affected object's surviving components and failure sequence alongside witness connectivity. A17-P1, A17-S2

This explanation follows the current official stretched-cluster guide, which explicitly targets vSAN 9.1 in VMware Cloud Foundation 9.1 and primarily discusses ESA. It is an inspection-only article, not a VCF deployment or recovery procedure. No site failure, object repair or takeover was executed.

The witness answers a different question

In a stretched cluster, the preferred and non-preferred sites hold the data-side components. The witness site supplies a separate location for witness metadata. vSAN uses component information to decide whether and where an object can remain available, including during isolation between data sites. This helps prevent independent updates to competing copies, commonly called split brain. A17-P1, pages 4–6

The witness therefore participates in the availability decision. It is not a third data site from which an entire VMDK can be reconstructed after both data copies are lost. The Stretched Cluster Guide states that the witness stores only a small amount of metadata and is outside the data path. A17-P1 The witness-appliance KB is corroboration only. A17-S2

Keep these questions separate: is the witness reachable, are sufficient data components accessible, and can the object establish the required quorum? A green host connection answers only part of the first question.

Start with an object, not a cluster average

Prerequisites are the cluster's actual version and architecture, the affected VM/object identity, its storage policy, existing component-health evidence and a timestamped site-connectivity history. Use authorised inventory and support captures; do not change object metadata or inject a network failure.

Record which data site and hosts hold the affected components, their reported states and when those states changed. Keep the object-level record: another VM being available does not prove that this VM's objects have the same placement or failure history.

Then record witness connectivity from the relevant data-side perspective. Management access from vCenter to the witness is useful evidence, but it is not a complete account of every communication path involved in the storage decision.

If the available report contains only a cluster headline, retain that as incomplete evidence. Avoid inventing a component majority from host counts; the actual object layout and policy matter.

Worked interpretation

Consider this synthetic incident:

Observation What it proves
Witness appears connected in inventory Its management connection is observed as up
One VM object is inaccessible That object's availability requirements are not met
Other VMs remain accessible Their own objects still have usable availability paths
Data-side component evidence is missing The affected object's cause remains unresolved

The correct next step is to obtain the missing component and failure-sequence evidence. Replacing the witness merely because one object is inaccessible does not follow from this table.

If existing evidence instead shows both data sites' required components are unavailable, an online witness cannot supply the missing user data. Conversely, when data remains on a site but quorum is lost, “inaccessible” is not sufficient proof that every byte has been destroyed. The availability mechanism may be preventing unsafe independent updates.

Use the vSAN Stretched Cluster Guide as the primary authority for the witness’s small metadata-only role outside the data path; KB 407619 is corroboration, not a relocation procedure offered here. The guide dates Adaptive Quorum Control to vSAN 7 U3, while its manual site-takeover capability is described for VCF 9.1. Those dates do not validate a 9.1 recovery procedure on an older cluster. Consult the guide’s Appendix: Stretched Cluster Failure Handling Scenarios when comparing failures before and after 7.0 U3; sequence and simultaneous failures matter.

Failure order changes the explanation

The guide describes Adaptive Quorum Control, introduced in vSAN 7 U3, adjusting votes after a data-site outage so that eligible objects can tolerate a subsequent witness outage. Adjustment is per object and takes time. The guide explicitly distinguishes that sequence from a simultaneous loss of a data site and witness. A17-P1, pages 32–33

For this 9.1-scoped diagnostic, preserve the order and timestamps. “Site A and witness failed” omits information that may explain why objects behaved differently. Do not assume that the adjustment completed merely because a short interval elapsed.

The guide also describes 9.1-specific recovery capabilities under particular maintenance conditions. Those are not general solutions to simultaneous double failure and are outside this article. Do not generalise them backwards to 8.0 or use them as permission to force ownership during an uncertain partition.

Acceptance and safe limits

The diagnostic result should identify the object, policy, observed component states, witness evidence and event sequence. It should explain whether missing data availability, quorum loss or incomplete evidence prevents a conclusion.

After a separately approved recovery, proposed acceptance includes accessible objects, the required protection state and application-owner verification. A witness returning Connected or a VM appearing powered on is not enough to establish either data consistency or restored resilience. No acceptance test was run here.

Read-only investigation has no rollback. Witness replacement, site takeover and object recovery require a qualified procedure that accounts for every potentially surviving data copy. Preserve the evidence and involve the platform support owner rather than modifying votes or deleting objects.

Learning notes

  • Witness metadata helps decide availability; it cannot replace missing user data.
  • Object-level evidence is more precise than a cluster headline.
  • The order of site and witness failures can change the outcome.

References

Related articles