Proxmox HA with Ceph: quorum, fencing and why nodes reboot

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

A Proxmox node may reset deliberately after losing quorum while holding active HA services. If its watchdog expires, self-fencing removes the node so another node can safely start those guests. Confirm that chain in the logs before attributing an unexplained reset to HA. The cure is rarely to turn fencing off. It is to find what broke Corosync, and to size votes, monitors and replicas so that one failure cannot take all three down.

Node names and addresses are examples.

Interpretations are editorial; no lab testing was performed.

Why it matters

A hyperconverged Proxmox cluster runs three separate agreement systems on the same servers:

• Corosync decides which nodes form the cluster.
• Ceph monitors decide what the storage map is.
• Ceph placement groups decide whether a given piece of data has enough copies to accept I/O.

Each has its own arithmetic, and a cluster can pass one test and fail another.

HA here means restart, not continuous running. The HA chapter gives typical detection and failover times of about two minutes. A guest on a failed node boots again elsewhere from whatever its disk last held.

How it works

The three agreements

Proxmox's default replicated pool is size 3 with min_size 2. Proxmox says plainly not to set min_size to 1, because writes accepted on a single replica can lead to data loss, incomplete placement groups or unfound objects.

Vote arithmetic explains the node counts. The figures below are calculated from the 50%-plus-one rule with one vote per node.

Four nodes tolerate no more node failures than three. A clean two-and-two split leaves neither side with a majority.

The HA daemons and locks

Each node runs two daemons. pve-ha-lrm, the local resource manager, runs what it is told on that node. pve-ha-crm, the cluster resource manager, makes the cluster-wide decisions, and only the node holding the manager lock is active as master.

The locks live in pmxcfs, the Proxmox cluster filesystem mounted at /etc/pve. This design is what links quorum to fencing:

1. pmxcfs goes read-only when a node loses quorum.
2. A node that cannot refresh its lock loses the right to run HA services.
3. The LRM source updates the watchdog only after it obtains its lock, so no lock means no watchdog update.

What the reset actually is

A watchdog is a timer that resets the server unless software keeps refreshing it. Proxmox's chapter also lists external power switches and network isolation as fencing methods, but builds its HA on watchdog self-fencing. It multiplexes one watchdog between its HA daemons with a small service, watchdog-mux. Reading its source shows two layers:

• The device timer is set to 10 seconds, and watchdog-mux keeps refreshing it.
• Each client, meaning an LRM or CRM, has a 60-second limit. A warning is written at 50 seconds.
• Past 60 seconds, watchdog-mux stops refreshing the device.

A client that disconnects without the clean close sequence also stops the refreshing, with no 60-second wait. Proxmox warns that killing pve-ha-crm, pve-ha-lrm or watchdog-mux can cause an immediate reboot or reset.

Quorum loss alone does not always reset a node. The HA chapter sets out the conditions:

• A node resets after 60 seconds if it has active services, or if the LRM or CRM process is killed or not scheduled.
• A node with an active CRM and an idle LRM does not reset on quorum loss, because everything that CRM writes sits on the read-only filesystem.
• With no service configured for more than 15 minutes, the CRM returns to idle and closes the watchdog.

The failure timeline

This sequence is assembled from the documentation and the source. It is expected behaviour, not a measurement. The isolated node's times count from its last successful watchdog update. The majority's 60 seconds is a different clock: it runs from when the master last saw the node online, which on our reading of the code is about when the new membership forms, so fence comes later by roughly the token and consensus time.

Our reading of the design is that the ordering is the safety argument: the lock cannot normally be taken until after the watchdog should have fired. That is why the reset is not optional. The HA chapter describes the alternative: if every network except storage fails and an isolated node's VM keeps writing while a second copy starts elsewhere, double writers can destroy all the VM's data.

Corosync timers matter because they consume this margin. Proxmox gives the formula and thresholds: token = 3000 + (nodes − 2) × token_coefficient milliseconds, and consensus = 1.2 × token. The sum is the minimum time to form a new membership. Proxmox calls lowering the coefficient "a suggested optimization" above 30 seconds, recommended above 40 and strongly recommended above 45.

These are our calculations from the formula. At 650 ms the 28-node row is past the 40-second level and the 32-node row past 45. The 5-node, 650 ms figure matches Proxmox's own example. New clusters since 9.2 set 125 ms. A cluster with no explicit setting runs 650 ms, so an older cluster should be checked rather than assumed.

What to choose and why

The choices in this section are our recommendations. The linked limits are vendor facts.

Size votes and monitors together

Three nodes is the minimum Proxmox gives for reliable quorum and for hyperconverged Ceph.

Work through failure combinations before go-live. The outcomes below are derived from the rules above, with one vote per node, size 3, min_size 2, a host failure domain, and on five nodes three monitors.

The last three rows are the ones people miss. A five-node cluster does not automatically ride through any two node failures. Corosync votes do, but storage may not. "Expected to stall" is our reasoning: we found no Ceph statement on what happens to I/O already in flight when monitors lose quorum, only that a client needs a current map from a monitor.

Spread monitors so a likely double failure, such as a shared rack or power feed, cannot take two of three.

Two nodes, even nodes and the QDevice

A QDevice is an external vote served by corosync-qnetd on a third machine. Proxmox recommends it for two-node clusters and supports it for even counts. It discourages it for odd counts, where the QDevice gets N−1 votes and its own failure means no node may fail.

# On the external host
apt install corosync-qnetd
# On every cluster node
apt install corosync-qdevice
# On one cluster node, with all nodes online
pvecm qdevice setup 192.0.2.50
pvecm status

Setup needs key-based root access to the external host, or root login with a password allowed temporarily, which is a security decision to make on purpose. Run the daemon as an unprivileged user, as the package does. Remove the QDevice before adding or removing a node.

The vote host can sit outside the cluster LAN, because it uses TCP and is not bound by Corosync's latency needs. Port 5403 must be reachable. A QDevice supplies a vote only. It adds no Ceph monitor or replica, so it does not make two-node Ceph sound.

Corosync's two_node: 1 option sets quorum to 1 artificially and turns on wait_for_all. It is not the path Proxmox documents for two nodes. We would not use it with watchdog self-fencing: with quorum at 1, our reasoning is that both sides of a network split can stay quorate, so nothing stops either side.

Give Corosync its own network

Proxmox's requirements are specific:

• Latency under 5 ms between all nodes. Above about 10 ms with more than three nodes, stable operation becomes rather unlikely.
• A dedicated physical NIC. 1 Gbit is enough in most situations, because the need is low jitter rather than bandwidth.
• A second link on a different physical network, up to 8 links in total.
• Do not share a network with storage, except as a low-priority fallback.

Ceph makes this a real threat. Proxmox warns that recovery traffic can interfere with Corosync and cause loss of quorum. So one failed node starts recovery, recovery swamps a shared link, and Corosync then loses a second node.

Bonds need care, and a separate second link does more than tuning one. Proxmox recommends at least one dedicated physical NIC for the primary link, with bonds only "as additional links". Its failure scenario is a bonded interface that stops passing packets but keeps link state up, with no other Corosync link available. Load-balancing modes can then leave nodes able to reach only different subsets of each other. Even nodes with healthy bonds may fence, and in the worst case the whole cluster fences itself.

LACP at the default slow rate takes 90 seconds to drop such a member, which is longer than the roughly one minute a node with HA resources survives. Set bond-lacp-rate fast on both node and switch for about 3 seconds. Avoid balance-rr, balance-xor, balance-tlb and balance-alb for Corosync. active-backup does not cause the split, but Proxmox warns the bond "may not switch over to the backup link", so the node can still lose the cluster and fence.

Use a real watchdog if the board has one

Without configuration Proxmox uses the Linux softdog. Proxmox says it is not independent of the server's hardware and has lower reliability than a hardware watchdog. Our inference is that this matters most for a kernel that has stopped running, since the safety timeline above assumes the reset happens.

Hardware watchdog modules are blocked by default, and Proxmox calls them a loaded gun if not correctly initialised. To use one, set it in /etc/default/pve-ha-manager, for example WATCHDOG_MODULE=iTCO_wdt, and test on a node with no active services.

Plan the maintenance modes

Update packages one node at a time. An LRM restart needs an active master CRM to acknowledge it, and without one the update can run long enough to reach a watchdog reset.

Once a host has been away for the 600-second default, monitors mark its OSDs out and data starts to move where there is room to place it. For a reboot that may run past that, flag only the host with ceph osd set-group noout HOSTBUCKET, then clear it with ceph osd unset-group noout HOSTBUCKET. Ceph warns against treating the cluster-wide ceph osd set noout as advice and says flagging only affected OSDs "is a safer approach". Proxmox uses the global flag for shutting down the whole cluster, a different case. While a flag is set, Ceph will not mark those OSDs out, so a disk that really fails in the window is not re-replicated. In our view a reboot well under 600 seconds may need no flag at all.

How to prove why a node reset

Start with the reset itself. On the node that reset, read the previous boot:

journalctl -b -1 -u watchdog-mux
journalctl -b -1 -u pve-ha-lrm
journalctl -b -1 -u corosync

Then on the current master, read journalctl -u pve-ha-crm for the same period. watchdog-mux forces a journal sync before reset, but the last lines can still be lost, and -b -1 needs a journal retained across reboots.

Then check current state:

ha-manager status                    # quorum, master, fencing, each LRM
pvecm status                         # Expected votes, Total votes, Quorum, Flags
corosync-cfgtool -s                  # link status on this node
corosync-cfgtool -n                  # nodes and their links, with MTU
corosync-cmapctl | grep -Ew 'runtime.config.totem.token|runtime.config.totem.consensus'
ceph -s
ceph health detail
ceph quorum_status
ceph osd pool get POOLNAME min_size

The Ceph commands are in ceph(8). The ha-manager status fencing line reads armed, standby, disarming or disarmed.

If guests stall while Corosync stays quorate, look at OSD state and the storage network, not at fencing.

Find the cause in this order:

1. Compare reset times across nodes. Several nodes within seconds suggests a shared switch, bond or quorum event, not one server.
2. Check whether the Corosync link shares a path with Ceph or backup traffic, and whether Ceph recovery was running at that moment.
3. Check for bond faults, with the LACP rate on both ends.
4. Add up the token and consensus values against the 30, 40 and 45-second levels.
5. Check for package updates, service restarts or memory pressure at that time.

Risks and limits

pvecm expected 1 is dangerous during an outage. Proxmox documents it for changing configuration on a node with no quorum, for those who understand what they are doing. It makes that node quorate on its own. If the other nodes are alive but unreachable, our reading is that you have removed the protection quorum gave you and could run a guest twice. Do not use a quorum override as a diagnostic step. A supported recovery plan must establish authoritative configuration, exclude competing writers and specify how normal quorum will be restored; powered-off peers alone do not complete that plan.

Recovery concentrates load. Leave room for one node's guests, and for Ceph recovery on top. On three hosts with host failure domain there is no spare host to rebuild the third copy, so data stays degraded until the node returns.

Source code changes. Timer values, log strings and states come from pve-ha-manager at HEAD on 2026-11-10 and may differ in your version.

Sources

Opened 2026-11-10.

• Proxmox VE: High Availability (9.2.13)
• Proxmox VE: Cluster Manager (9.2.13)
• Proxmox VE: Deploy Hyper-Converged Ceph Cluster (9.2.13)
• Proxmox VE: Cluster File System
• ha-manager(1)
• pve-ha-manager: watchdog-mux.c
• pve-ha-manager: LRM.pm
• pve-ha-manager: NodeStatus.pm
• pve-ha-manager: Env/PVE2.pm
• pve-ha-manager: Manager.pm
• votequorum(5)
• corosync.conf(5)
• corosync-cfgtool(8)
• Ceph: Monitor configuration reference
• Ceph: Pools
• Ceph: Monitoring OSDs and PGs
• Ceph: Troubleshooting OSDs
• Ceph: Architecture
• ceph(8)
• journalctl(1)

This article was originally published on 10 November 2026.