Stretched clusters: witness placement, split brain and what each failure really does

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

A stretched cluster needs a product-specific way to decide which site may write after a failure and to prevent competing writers. Witnesses, arbiters and fencing perform different parts of that decision. Place the required arbiter somewhere that fails independently of both sites and has its own path to each. Then test four separate events, because a site loss, a link loss, a witness loss and a double failure do different things in each product.

Names and addresses are examples. Tables that compare products are drawn from each vendor's documents, and cells worked out by vote arithmetic are marked.

Interpretations are editorial; no lab testing was performed.

Why it matters

From inside one site, a dead partner site and a cut link look identical. NetApp says so directly: MetroCluster configurations typically cannot differentiate between failures and disasters without witness software.

If each side assumes the other is dead and carries on, both accept writes to what should be one copy of the data. That is split brain. Microsoft describes quorum as the mechanism that prevents it. Proxmox's HA chapter gives the concrete harm: a resource running twice once it is recovered on another node.

Two equally weighted isolated sites cannot both retain a majority under ordinary majority voting. Unequal voting can favour one site, but does not make either site independently survivable; examine the product's arbitration and fencing rules.

How it works

Three separate jobs

Designs go wrong when these are treated as one thing.

• Quorum decides which group of members may continue.
• The witness adds a vote or an arbiter so a tie between two equal sites can be broken. It normally holds no workload data. vSAN's witness stores only metadata.
• Fencing makes the losing side stop. A correct vote achieves nothing if the losing hosts keep writing.

The vote arithmetic

Take two nodes per site and one witness. That is five votes, so three are needed.

The last row is why a failed witness is an urgent repair, not a warning to clear next week.

Windows softens that row but does not remove it. Dynamic quorum recalculates the majority among survivors, and an even node count without a witness has one node's vote zeroed. Microsoft's table for four nodes without a witness gives two simultaneous failures a fifty percent chance. By our reading, a two-against-two split is that case: one side survives, and without a Preferred Site the node whose vote was zeroed decides which side.

You can bias that outcome. Microsoft's Failover Clustering team described in 2019, for Windows Server 2016, how a Preferred Site makes dynamic quorum drop votes from the other site first, so that in an even split of two datacentres with no witness the Preferred Site "is automatically elected to win". Microsoft's stretch cluster guide shows the setting, (Get-Cluster).PreferredSite. We did not find the quorum statement on the current Learn pages we opened, so test it on your Windows version.

What corosync's third vote does

On Proxmox VE, the third vote is a QDevice talking to corosync-qnetd on an external host. Its default ffsplit algorithm gives one vote to the partition with the most active nodes.

In an even site split, the order of tie-breaks is:

1. Heuristics score: connected nodes plus nodes passing heuristics, minus nodes failing them.
2. The partition with the most clients connected to qnetd.
3. The tie_breaker setting. The default, lowest, picks the lowest node ID.

The same page documents a further option, keep_active_partition_tie_breaker, on by default. When a tie happens, QNetd "will prefer partition with members of previously active (quorate) partition". The page does not say where this sits against the list above.

So an equal split may be decided by node numbering or by which partition was active before, and a repeat test may not match the first. If one site must win, configure it deliberately with a heuristics test or a tie_breaker node ID, then prove it over repeated runs. This man page is the undated Debian testing copy, so check the page for the corosync-qdevice version on your Proxmox release.

Where the witness must live

Vendor rules first.

Three details are easy to miss.

Microsoft does not support DFS or replicated storage under a file share witness, because of the risk of partition in time and data loss. A witness share copied between your two sites is the design this forbids.

vSAN allows a witness to run on another stretched cluster only if that cluster does not occupy the same sites. Two clusters witnessing each other across the same two rooms would fail together.

NetApp requires that a failure in one site must not interrupt IP connectivity between the other two. The other vendors do not all spell this out, but the logic is general.

Our recommended checks, which are editorial rather than vendor text:

• Path. Trace the route from each site to the witness. If site B reaches it through site A, losing site A leaves site B with no witness and no majority.
• Carrier and power. A third building on the same carrier ring or power feed may not be a third failure domain.
• Hosting. A witness VM inside a data site adds no independence. Losing that site removes two votes.
• Shared third site. One location may host many witnesses, which vSAN explicitly permits. An outage there degrades all those clusters at once, so plan the repair capacity accordingly.

What each failure does

Cells marked arithmetic are worked out from votes because the opened documents do not list that case.

vSAN's link-loss behaviour means healthy VMs on the non-preferred site are stopped and restarted. VMs whose storage policy places data only on the non-preferred site keep running there, held by DRS rules. Which site you call preferred is therefore a real workload decision.

MetroCluster's link-loss behaviour is the opposite instinct. Nothing moves, but the two sides stop mirroring. During that window a site outage can make unmirrored writes unavailable; their permanent loss depends on whether that site's data can be recovered.

NetApp's conditions for Mediator-assisted switchover apply here. By default it needs SyncMirror and DR mirroring of each node's nonvolatile cache to be operating and synchronised at the time of failure, no surviving node in takeover state, and all nodes at one site failed. Our inference is that automatic switchover should not be expected during a mirroring gap, so that event becomes a human forced-switchover decision. NetApp documents options that change this, including automatic forced switchover from ONTAP 9.12.1. Versions differ too: from 9.13.1 environmental shutdowns can trigger it, and the failure-domain default is auso-on-dr-group, where 9.12.1 and earlier used auso-on-cluster-disaster.

Double failures

Losing a data site and the witness at the same moment leaves vSAN objects without quorum. For all versions up to vSAN in VCF 9.1, recovering from the remaining site means contacting Global Support.

Sequence matters. vSAN's Adaptive Quorum Control adjusts votes after a data site outage so that a later witness loss can be tolerated. VMware says the adjustment takes seconds to minutes, per object, and does not help a simultaneous double failure.

The layers must agree

A hypervisor cluster over independent storage has two quorum systems, and they can reach different answers.

On Proxmox with Ceph, corosync and the Ceph Monitors each decide separately. A QDevice tie-break by lowest node ID has no knowledge of which zone Ceph's tiebreaker favoured. The result can be guests allowed to run in the site where storage has stopped. This is our inference from the two documents: neither page covers the combination, and no opened Proxmox page describes a supported stretched design. No document we opened describes a way to align the two decisions. Placing both third votes in one site gives them the same reachability, but they can still choose different sides, and it concentrates two arbiters in one place. Treat this as an open risk: test a link cut with both layers watched, and decide in advance what you will do if they disagree.

Proxmox also requires under 5 ms between all nodes for corosync, is sensitive to jitter, and advises against sharing that network with storage.

Windows has the same issue one level up. A SQL Server availability group on the cluster inherits its quorum. Microsoft's voting guidance says nodes should not vote without justification, that primary replicas and automatic failover owners should vote, that secondary disaster recovery nodes generally should not, and that votes should be odd in number and re-assessed after failover.

Fencing across sites

Quorum only says who should stop. These are the mechanisms that make it happen.

Proxmox nodes self-fence with a watchdog, using softdog if no hardware watchdog is configured. Corosync on an LACP bond needs care: the default LACP rate gives a 90-second failover, longer than the roughly one minute before HA nodes fence. Proxmox recommends bond-lacp-rate fast on both the node and the switch.

Pacemaker's documentation says a fence device must not share power with its target, so on-board IPMI should not be the only device. It also says devices must "not rely on the same network as the cluster itself", so a fencing path over the inter-site link fails during the event it exists for. A pcmk_delay_max setting "is sometimes used in two-node clusters" so the two sides do not fence each other at the same time, and pcmk_delay_base can do the same.

Where nodes must fence across sites, we prefer a mechanism that lets the isolated node remove itself.

vSAN stretched clusters need host isolation detection that works per site. VMware specifies das.isolationaddress0 and das.isolationaddress1 set to addresses on the vSAN network in each site, with das.usedefaultisolationaddress set to false.

What to choose and why

Latency budget. vSAN specifies 5 ms round trip or less between data sites and 10 Gbps minimum. Storage Replica recommends an average of 5 ms or less for synchronous replication. A synchronous write is acknowledged only after the remote copy is, so every write pays at least one round trip. Convert that to your database's commit time before signing off the sites. Measure the link rather than estimating from distance.

Capacity for the bad day. VMware recommends HA admission control at 50% for both CPU and memory. If you run each site above half, a site loss cannot restart everything. Ceph stretch mode keeps four copies, so usable capacity is a quarter of raw before free-space margins. It is also replicated only, needs five Monitors in a 2+2+1 layout, and is recommended only on SSD OSDs.

Choosing a pattern.

A stretched cluster also replaces neither backup nor point-in-time recovery. Synchronous replication copies corruption and deletion just as faithfully.

Testing it

Test witness loss first, then link loss, then site loss, one at a time. Let everything resynchronise fully between tests.

Warning. A link-cut test can deliberately stop or reset hosts on one side. Use a non-production cluster or an agreed outage window, and have a way to reach both sites that does not depend on the link being cut.

Read state rather than assuming it:

pvecm status                  # Proxmox: votes, and per-node QDevice flags
Get-ClusterQuorum | Format-List *   # Windows: quorum resource
Get-ClusterFaultDomain              # Windows: site and node layout
ceph -s                             # Ceph: health, including stretch warnings
metrocluster show                   # ONTAP: configuration and state
metrocluster operation show         # ONTAP: last operation and healing

In pvecm status, A or NA shows whether qnetd is reachable, and V or NV whether the QDevice will cast a vote for that node.

Failback needs the same discipline as failover:

• vSAN waits 60 minutes by default before starting to resynchronise some absent components, and moving VMs back with DRS can take time depending on how much changed.
• Ceph enters recovery stretch mode and requires OSDs only from the zone that stayed up until placement groups are healthy.
• MetroCluster healing can take hours. On MetroCluster IP the disaster-site nodes must be booted before healing. Automatic healing can fail, in which case you run the metrocluster heal commands yourself.

Time the resync in your test. It measures how long full cross-site protection remains absent. Local copy counts and protection during that interval differ by product; Ceph stretch mode retains two copies in the surviving zone.

Risks and limits

Quorum overrides require a recovery decision. Forcing a quorum, disaster switchover or stretch-mode transition can permit writes without the normal protection. Physical confirmation that another site is down is only one prerequisite. The recovery plan must establish authoritative data, exclude every competing writer, confirm available replicas and use the supported procedure for the exact product version. These overrides are not diagnostic steps.

two_node: 1 on corosync sets quorum to 1 artificially and turns on wait_for_all. It keeps a two-node cluster running but does not supply a third opinion, so fencing has to do all the work.

Ceph device classes. A CRUSH rule with a device class is not supported in stretch mode. The development documentation explains why it is dangerous: when a data centre is lost, every placement group using that rule goes inactive, and restoring the data centre does not clear it. Use a rule with no device class from the start.

Ceph version differences. The automatic tiebreaker form of ceph mon enable_stretch_mode appears on the Umbrella page but not the Tentacle page, and exiting stretch mode needs 18.2.8 or later. Read the page for your exact version.

Sources

Opened 2026-11-10.

• vSAN Stretched Cluster Guide, VCF 9.1
• Understanding cluster and pool quorum, Microsoft
• Deploy a quorum witness, Microsoft
• Storage Replica overview, Microsoft
• Stretch cluster replication using shared storage, Microsoft
• Fault domain awareness, Microsoft
• WSFC quorum modes and voting configuration, Microsoft
• Cluster Manager, Proxmox VE
• High Availability, Proxmox VE
• corosync-qdevice(8)
• votequorum(5)
• Stretch mode, Ceph Umbrella
• Stretch mode, Ceph development documentation
• Fencing, Pacemaker 3.0 Explained
• MetroCluster data protection and disaster recovery, NetApp
• Mediator requirements, NetApp
• Considerations for ONTAP Mediator, NetApp
• How ONTAP Mediator supports automatic unplanned switchover, NetApp
• Site-aware failover clusters in Windows Server 2016, Microsoft Failover Clustering blog

This article was originally published on 10 November 2026.