About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.
Galera Cluster gives MariaDB synchronous multi-node replication. Data-bearing nodes maintain replicated data and eligible nodes can accept writes. Continued service depends on a Primary Component, healthy storage and node readiness; apply queues can differ, and an arbitrator holds no database copy. For many teams it is the main high-availability layer under their applications.
That resilience has a boundary. When every node stops, through a power failure, a hypervisor or storage outage, a network event that isolates all nodes, or a planned shutdown that went out of order, recovery depends on saved component state and which members return. Automatic component recovery may succeed under the conditions discussed below. When manual bootstrap is necessary, a qualified operator must establish the authoritative history before any node becomes a new primary.
This guide explains how that decision works, as the MariaDB documentation describes it, and what to protect before anyone starts anything.
Why it matters
The quorum reset page is direct about the stakes. Before bootstrapping, "you must identify the most advanced node in the cluster. This is the node whose local database committed the last transaction. Starting the cluster from any other node can result in data loss."
The pressure in a full outage pushes the other way. Applications are down, people are waiting, and the quickest-looking action is to start whichever node is easiest to reach. If that node was behind, the others will synchronise to it when they rejoin, and the transactions only they held are overwritten.
A second risk is starting two separate clusters. If a cluster is split by a network fault and someone forces each half to run, the result is two independent databases accepting different writes. Reconciling them afterwards is slow, manual work.
How it works
Quorum and the Primary Component
Galera decides which nodes may serve data by quorum. The quorum page explains that "Quorum is achieved when more than 50% of the total nodes in the last known membership are in communication." This node-count explanation assumes the ordinary equal-vote configuration. Weighted or customised quorum needs its own analysis. The group that has quorum is the Primary Component. "Nodes not in this group switch to a non-primary state, halting queries and entering a read-only 'safe mode' to prevent data discrepancies."
The quoted "safe mode" is not a promise of normal read access: non-primary nodes commonly reject queries. The purpose is to avoid split-brain, "which occur when network partitions lead to parts of the cluster operating independently and accepting writes."
Node counts matter. "In a 3-node cluster, a majority is 2. The cluster can tolerate the failure of 1 node and remain operational." In a two-node cluster, "if one node fails, the remaining node represents only 50% of the cluster, which is not a majority, and it will lose Quorum. This is why a 2-node cluster has no fault tolerance without an external voting member." The documented answer for even-numbered clusters is the Galera Arbitrator, garbd, "a lightweight, stateless daemon that acts as a voting member in the cluster without being a full database node."
Single-node failures recover themselves
The Primary Component recovery page is reassuring about the common case: "If one node in a cluster with three or more members fails, the rest of the cluster maintains Quorum and continues to operate. When the failed node comes back online, it will automatically connect to the cluster and initiate a State Transfer to synchronize its data."
A node that fails while receiving a full state transfer is also simple to handle: "To recover, simply restart the MariaDB service on the failed joiner node."
The difficult case is when no Primary Component remains.
Recognising a full outage
The quorum reset page describes the symptoms. Nodes that have lost the Primary Component "may return an Unknown command error for many queries". You can confirm it by checking the wsrep_cluster_status status variable on every node: "If none of your nodes return a value of Primary, you must manually intervene to reset the Quorum and bootstrap a new Primary Component."
The quorum page lists the variables worth checking in any health review: wsrep_cluster_status (healthy value Primary), wsrep_cluster_size (matches the expected total), and wsrep_cluster_state_uuid and wsrep_cluster_conf_id, which should be the same on all nodes.
If nodes are still running but some report Primary and others do not, this is not a full outage. Fix the network or the failed nodes and let them rejoin. Bootstrapping in that situation risks creating a second cluster.
grastate.dat and the safe-to-bootstrap flag
Each node keeps a small state file, grastate.dat, in its data directory. The quorum reset page shows an example containing the cluster uuid, a seqno and a safe_to_bootstrap value.
The safe-to-bootstrap feature exists "to facilitate a safe restart and prevent an administrator from choosing the wrong node". The page explains: "When a cluster is shut down gracefully, the last node to be stopped will be the most up-to-date. Galera tracks this and marks only that last node as safe to bootstrap from by setting a flag in its state file. If you attempt to bootstrap from a node marked as unsafe, Galera will refuse and show a message in the logs."
After a crash, the flag cannot help: "In the case of a sudden, simultaneous crash, all nodes will be considered unsafe, requiring manual intervention."
So the first question in any full outage is how the cluster stopped.
After an orderly shutdown
For a planned shutdown, the documentation says "you only need to follow the recommendation of the 'Safe-to-Bootstrap' feature." Inspect grastate.dat on each node, find the one with safe_to_bootstrap: 1, and use it. The quorum page puts it plainly: "You should always look for and bootstrap from this node."
After a crash
After a hard crash, "all nodes will likely have safe_to_bootstrap: 0", and the most advanced node must be found by comparison.
The documented method is to run the server on each node with the --wsrep-recover option. The quorum reset page explains that this "will read the InnoDB storage engine logs and report the last known transaction position in the MariaDB error log." The log line looks like the documentation's example: WSREP: Recovered position: 37bb872a-ad73-11e6-819f-f3b71d9c5ada:345628. You then "compare the sequence number (the number after the colon) from all nodes. The node with the highest sequence number is the most advanced."
The quorum page also mentions the seqno recorded in grastate.dat as a way to compare nodes. After an unclean stop, the value in the file may not reflect the last committed transaction, which is why the crash procedure relies on --wsrep-recover reading the storage engine logs. A disagreement needs investigation; retain the recovered position and its logs as evidence, including any storage or recovery errors, before selecting an authoritative node.
Two checks are worth adding to the comparison:
- Compare the UUID part too. The position is the cluster UUID, a colon, then the sequence number. Nodes reporting a different UUID are not on the same history, and comparing their numbers means nothing. Find out why before going further.
- Write down every node's result with the time it was read, and have a second person check the comparison. It takes a minute and prevents the most expensive mistake in the procedure.
The documentation permits an explicit safety-flag override in its qualified bootstrap procedure. This changes a protection mechanism; it does not validate the node's data. Preserve each node and obtain a reviewed decision on history, health and fencing before any such change. An unreachable member leaves uncertainty about whether the accessible nodes contain the latest committed transactions.
Bootstrapping the new Primary Component
The documentation describes two methods.
Automatic bootstrap is "recommended if the mysqld process is still running on the most advanced node. It is non-destructive and can preserve the GCache, increasing the chance of a fast Incremental State Transfer (IST) for the other nodes." It uses the documented provider bootstrap mechanism on the selected running node. The term "non-destructive" does not make an incorrectly selected history safe; it still changes which component can accept writes. The page notes a version difference: if the wsrep_provider plugin is enabled, the documented provider option statement fails and a separate global variable is used instead.
Manual bootstrap "involves a full shutdown and a special startup of the most advanced node". It requires the distribution-specific bootstrap startup path for the selected node. Service-manager automation and storage access must be controlled so another component cannot independently become writable.
Then, "after the first node is successfully running and has formed the new Primary Component, start the MariaDB service normally on all of the other nodes. They will detect the existing Primary Component, connect to it, and automatically initiate a State Transfer to synchronize their data and rejoin the cluster."
That last step is where a wrong choice becomes permanent. The rejoining nodes take their data from the bootstrapped node. If they held transactions it did not, those are replaced.
Automatic recovery with pc.recovery
The Primary Component recovery page describes a feature that can avoid manual work: "Modern versions of Galera Cluster enable the pc.recovery parameter by default." Nodes that were part of the last Primary Component save it to a file called gvwstate.dat, and "if the entire cluster goes down, it can automatically recover its state once all the nodes from that last saved component achieve connectivity with each other."
The file "is created in the data directory of a node when it is part of a Primary Component and is deleted upon graceful shutdown." In other words, it is mainly there for crashes.
The condition matters: all the nodes from the last saved component must come back and reach each other. If one node's disk is damaged or one host does not return, automatic recovery will not complete, and the manual procedure applies.
The documentation also describes editing gvwstate.dat to force a chosen set of nodes into a new component, and warns: "Avoid manually editing the gvwstate.dat file unless absolutely necessary. Doing so may cause data inconsistency or prevent the cluster from starting." Treat this as a last resort under expert guidance.
Split-brain: bootstrap one side only
When a network fault splits a cluster so that no side has a majority, the quorum page says that "both halves of the cluster will fail to achieve a majority, and all nodes will enter a non-Primary state." If recovery is needed before the network is fixed, the decision must include fencing every competing component and identifying the authoritative history; an inaccessible side must not be assumed stopped. The page's warning is the most important sentence in this subject: "Never execute the bootstrap command on both sides of a partition, as this will create two independent, active clusters with diverging data."
What to protect before starting anything
The documented procedure assumes the data on each node is intact. A full outage caused by power or storage problems is exactly when that assumption is weakest. These steps are our recommendations:
- Stop and look before starting. Make sure nothing is restarting nodes automatically, such as a service manager or an orchestration tool, while you investigate.
- Copy each node's data directory, or snapshot its disk, before any bootstrap. If the wrong node is chosen, or a node turns out to be damaged, these copies are what let you recover the transactions that would otherwise be overwritten.
- Know where your last good backup is. A cluster is not a backup: a destructive statement replicates to every node. If the outage followed a bad change, recovery may need a backup and point-in-time replay rather than a bootstrap.
- Check the storage and the error logs on each node for corruption messages. If the most advanced node has damaged storage, do not silently substitute a healthy older node. Escalate the preservation and recovery decision, record any accepted loss and establish whether backup plus logs can recover more complete data.
What to choose and why
These are our recommendations, based on the documentation above.
Design
- Use an odd number of voting members. Three database nodes, or two plus an arbitrator, so that one failure does not stop the cluster.
- Place nodes in separate failure domains: different hosts, power feeds and, where latency allows, different sites, with the arbitrator in a third location.
- Keep regular physical or logical backups of at least one node, and test restoring them. Replication protects against node loss, not against mistakes.
Runbook
- Write the full-outage procedure down for your cluster: node names, data directory paths, service names, how to run the position check, who decides, and who checks.
- Record how each shutdown happened. For planned maintenance, stop nodes one at a time and note which was last; that node will carry the safe-to-bootstrap flag.
- Rehearse a full-outage recovery on a test cluster at least once a year, including a simulated crash where every node shows
safe_to_bootstrap: 0.
On the day
- Confirm that no node reports
Primary. - Establish whether the stop was orderly or a crash.
- Protect each node's data before changing anything.
- Choose the node: the safe-to-bootstrap flag after an orderly stop, the highest recovered position with a matching UUID after a crash. Have the choice checked.
- Bootstrap that one node only, and confirm it reports
Primary. - Rejoin members under the reviewed procedure, recording state-transfer completion and failures.
- Check the application data, and keep the pre-bootstrap copies until you are sure nothing was lost.
Risks and limits
- Bootstrapping the wrong node can lose committed transactions once the others resynchronise from it.
- Bootstrapping both sides of a partition creates two diverging clusters.
- Editing
grastate.datorgvwstate.datoverrides Galera's safety checks; leave these changes to the qualified, version-specific recovery procedure after preserving the original state. - Damaged storage changes the procedure; recovery may need backups rather than a bootstrap.
- Distribution differences. Percona XtraDB Cluster and other Galera builds may use different service and script names. This article follows the MariaDB documentation.
- Nothing here was run. The procedure is described from the documentation for planning.
Sources
- MariaDB, Understanding Quorum, Monitoring, and Recovery: https://mariadb.com/docs/galera-cluster/high-availability/understanding-quorum-monitoring-and-recovery
- MariaDB, Recovering a Primary Component: https://mariadb.com/docs/galera-cluster/high-availability/recovering-a-primary-component
- MariaDB, Resetting the Quorum (Cluster Bootstrap): https://mariadb.com/docs/galera-cluster/high-availability/resetting-the-quorum-cluster-bootstrap/
This article was originally published on .