Australian cloud for business software — built and run in Sydney since 2003

+61 2 8282 5656
support@netleverage.com
Talk to an engineer
Talk to an engineer
NetLeverage
NetLeverage

Linux software RAID after a server fails: assembling mdadm arrays on replacement hardware

Technical article
Backup & Recovery

Originally published:

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

When a physical Linux server fails and its disks survive, an mdadm software RAID array can normally be assembled on another machine, because the description of the array lives on the member disks rather than in the failed server. The mdadm manual says that, for assembly, mdadm needs to know "the identity of the array, and the number of component devices", and that only devices whose md superblock "contains the right identity will be considered for any array". A recovery can become harder after a decision made too early: letting the array start read-write before you know whether every member is current, letting a degraded and dirty RAID 5 or RAID 6 rebuild over the only good copy, or reaching for options that rewrite metadata.

The first decision is whether the disks are stable enough to examine. Failing media or uncertain membership may require specialist imaging before assembly. For healthy, understood members, read-only examination and controlled access can preserve options while important data is copied. Resynchronisation is a later decision, not an automatic next step.

Why it matters

Software RAID moves the knowledge of the array out of a controller and into the disks themselves. That is a real recovery advantage: native Linux md metadata is not tied to one controller model. A replacement still needs compatible disk presentation, sector sizes and Linux support; external firmware-assisted formats need separate checks. The md(4) manual describes the metadata, called the superblock, as the record of "the structure and state of the array", which "allows the array to be reliably re-assembled after a shutdown".

The same design means the disks also carry the evidence of what happened before the failure. If one member dropped out days before the server died, its metadata is older than the others. If the server lost power mid-write, the array is marked as not cleanly stopped. A replacement machine that assembles and starts everything automatically can act on that evidence before anyone has looked at it.

RAID is also not a backup. It protects against a disk failing while the system runs; it faithfully mirrors deletion, corruption and ransomware encryption. In a bare-metal recovery, an intact array is a gift that saves a long restore, but your plan still needs a separate copy of the data in case the array cannot be trusted.

How it works

What the superblock records

Each member disk or partition carries md metadata. The mdadm manual lists the formats. The default for new arrays is version 1.2. Version 1 superblocks "can easily be moved between hosts with different endian-ness". The sub-versions differ only in where the metadata sits: at the end of the device for 1.0, at the start for 1.1, and 4K from the start for 1.2.

The older 0.90 format is still found on long-lived systems. The manual notes it limits arrays to 28 component devices and component devices of RAID levels 1 and above to 2 terabytes, and that there can be "confusion about whether the superblock applies to a whole device or just the last partition". The md(4) page adds that 0.90 stores multi-byte fields in a processor-dependent way. That rarely matters when moving between two x86 servers, but it is a reason to know which format you have before you plan.

Two other formats deserve attention on physical servers. Intel Matrix Storage Manager (IMSM) metadata is used by firmware-assisted RAID on some Intel platforms and is managed as a container, and DDF is a SNIA standard format that the manual describes as "deprecated" with "no active development". If the failed server used firmware RAID with IMSM metadata, the replacement's firmware and option ROM become part of the story. mdadm --detail-platform reports "the platform's RAID capabilities (firmware / hardware topology)", so you can check the replacement before you move the disks.

Identity: UUID, name and home host

Every array has a 128-bit UUID and, with version 1 metadata, a name. mdadm also records a home host, the machine the array considers its home. The manual explains that for version 1 superblocks the home host "will be prefixed to the array name", while for 0.90 part of a hash of the host name is stored in the UUID.

This is why a recovered array often turns up under an unexpected device name on new hardware. Under auto-assembly, the mdadm manual says, "only arrays tagged for the given homehost will be allowed to use 'local' names", meaning names "not ending in '_' followed by a digit string". Arrays that do not obviously belong to the current host "are given names that are expected not to conflict with anything local". A replacement server has a different host name, or the same name set later than the boot that assembled the disks, so it treats the old arrays as foreign.

Filesystem UUID references can reduce dependence on an md device name, but every boot, monitoring and application reference still needs checking. Duplicate restored filesystem UUIDs also need isolation. A name change can break access if /etc/fstab, a boot loader entry or a script refers to /dev/md0 or a similar fixed name.

Configuration: mdadm.conf and the initramfs

The mdadm.conf(5) page describes ARRAY lines that identify actual arrays, normally by uuid=, and states that "if multiple identities are given, then a component device must match ALL identities to be considered a match". It also describes the HOMEHOST line, whose default <system> uses the machine's host name, and the special value <ignore>, which disables the home-host check during auto-assembly.

On many distributions, arrays needed during boot are assembled from an early boot environment, the initramfs, which carries its own copy of the configuration. The dracut.cmdline(7) page documents boot options such as rd.md.conf=0, which ignores "mdadm.conf included in initramfs", and rd.md.uuid=, which activates only the arrays with the given UUIDs. The practical point is that there can be two copies of mdadm.conf, the one in /etc and the one inside the initramfs, and a recovery that fixes only one of them can boot differently from what you expect.

Event counts and why "out-of-date" matters

The metadata on each member includes an event counter that advances as the array state changes. When one member dropped out earlier, its count stops while the others continue. During assembly, mdadm uses this to decide which devices are current. The manual's description of --re-add shows the same mechanism working in your favour: based on the event count, recovery "may only require sections that are flagged by a write-intent bitmap to be recovered or may not require any recovery at all".

This is the piece of evidence you most need before starting an array on new hardware. Matching counts support the assessment that members reached the same recorded state; they do not prove clean shutdown or identical, correct data. If one is behind, that member holds older data, and treating it as current would bring stale blocks back into service. mdadm --examine prints the metadata on a component device, which the manual contrasts with --detail, which "applies to a whole array which is currently active".

Dirty, degraded and the RAID 5 write hole

The kernel RAID arrays guide explains why the kernel is cautious: "If a raid5 or raid6 array is both dirty and degraded, it could have undetectable data corruption." Dirty means the parity cannot be trusted because the array was not shut down cleanly. Degraded means some data blocks are missing and have to be reconstructed from that parity. For this reason md "will normally refuse" to start such an array.

The md(4) page describes the boot-time override, md_mod.start_dirty_degraded=1, and states plainly that "there is a real (though small) risk of data corruption in this situation". This override belongs only in a qualified recovery plan after preserving members, not in routine inspection. The same page describes md_mod.start_ro=1, which starts arrays in a "soft read-only" mode: nothing is written and no resync or recovery starts until the first write request.

What to choose and why

Before you move the disks

Record what you can about the failed server while it is still a known quantity, or from your documentation if it cannot boot: the RAID level, member count and order, the metadata format, the array UUIDs, and whether any member was already reported as failed. A copy of /etc/mdadm.conf or /etc/mdadm/mdadm.conf and the output of mdadm --detail from normal operations, kept outside the server, makes this a reading exercise rather than archaeology. The manual's suggested prototype config, generated with mdadm --detail --scan, "should be reviewed before being used as it may contain unwanted detail", which is good advice for any configuration you plan to restore.

Label disks with their bay and serial number. Version 1 metadata does not depend on physical order, but people do, and a disk put into the wrong server is a common way to start the wrong rebuild.

On the replacement server: inspect first

Before attaching the old disks, establish how the rescue environment controls automatic assembly, filesystem mounting and service startup. Rescue media alone does not guarantee this. On healthy media, the following documented inspection interfaces provide evidence; none was executed for this article:

cat /proc/mdstat
mdadm --examine /dev/sdX1      # repeat for each member
mdadm --examine --scan

Compare the array UUID, RAID level, device role and event count for every member. If your rescue system has already auto-assembled the array, it may be running read-auto. The manual says foreign arrays found during auto-assembly "are started 'read-auto' so that nothing is written to any device until the array is written to". This is not enforced read-only protection: a write request can leave that state, including requests caused by mounting or automatic services.

Assemble read-only when the members agree

Before assembly, compare UUIDs, member roles, RAID level, layout, expected member count, event counts, health and any interrupted reshape. Agreement on UUID and event count alone is insufficient. The mdadm manual documents --readonly, which starts the array "read only rather than read-write as normal. No writes will be allowed to the array, and no resync, recovery, or reshape will be started."

mdadm --assemble --readonly /dev/md/recovered /dev/sdX1 /dev/sdY1 /dev/sdZ1

A qualified recovery plan can then consider filesystem access and copying important data to independent storage. Read-only assembly still changes host state and does not prove the filesystem or application data is consistent. If the filesystem replays a journal on mount, check that filesystem's documentation for its read-only and no-replay mount options before mounting, so the mount itself does not try to write.

When a member is missing

If a member failed with the server and you are left with a degraded array, mdadm will not start it from an incomplete list unless asked. The manual: "if --scan is not given and not all expected drives were listed, then the array is not started (to guard against usage errors)", and --run insists that it start, "as may work for RAID1, 4, 5, 6, or 10". A mirror can remain readable only if its surviving copies cover the required data; RAID 10 depends on which members failed, not just how many. A RAID 5 with one missing member has no further redundancy, so an unreadable block on another member can make the affected data unrecoverable from that array. Neither case is automatically low risk. Media health determines whether specialist imaging or a controlled copy is appropriate before rebuilding.

When members disagree: what --force means

--force, according to the mdadm manual, assembles "even if the metadata on some devices appears to be out-of-date". When mdadm cannot find enough working devices but finds some recorded as failed, "it will mark those devices as working so that the array can be started". The manual's own warning is direct: "An array which requires --force to be started may contain data corruption. Use it carefully."

Forcing can change recorded member state and must be treated as a metadata-changing recovery action, not reversible inspection. If you face that choice, take a sector-level image of each member first, ideally to separate storage, and work on copies. A disk that already has read errors should be imaged with a tool designed for failing media before anything else touches it.

What to avoid: re-creating the array

Old forum advice sometimes suggests running mdadm --create with --assume-clean over the existing members, in the original order, to rebuild lost metadata. The manual explains why that option exists: it tells mdadm "that the array pre-existed and is known to be clean", which "can be useful when trying to recover from a major failure". It also warns: "Use this only if you really know what you are doing."

Re-creating writes new metadata. The metadata format, data offset, chunk size, layout and device order all have to match the original exactly, and the values mdadm chooses by default are not guaranteed to be the ones the original array was created with. A mismatch produces an array that assembles and holds scrambled data, and a resync can then make that permanent. This is a specialist operation on images of the disks, never a step in a routine recovery runbook.

Bring the system back to its own configuration

Once the data is safe and the array is healthy, decide what the array should be called on its new home. Update the ARRAY lines in mdadm.conf so the identities match, make sure /etc/fstab refers to filesystems by UUID rather than md device names, and regenerate the initramfs so its embedded copy agrees with /etc. Only after preservation, select the appropriate consistency or reconstruction operation for the diagnosed state, with logged progress and alerts. A check, resync and repair have different effects; none substitutes for validating copied application data. The kernel guide describes the sync_action file, whose values include resync, recover, check and repair, and mismatch_cnt, the count of sectors that were, or would have been, rewritten during a check or repair.

A non-zero mismatch_cnt after an unclean shutdown is not automatically a disaster. The kernel guide warns that the count can exceed the number of real errors because "most raid levels work in units of pages rather than sectors". It is still a number worth recording and explaining before you declare the recovery finished.

Risks and limits

  • Automatic assembly on the wrong host. Rescue and installer images often assemble any arrays they find. Booting the replacement into a full operating system with an old array attached can start services on it. Control activation before attaching vulnerable members and confirm the resulting state before filesystem access.
  • Firmware RAID changes the rules. IMSM and DDF arrays depend on platform support and container metadata. A replacement from a different vendor may present the disks differently or not support the container at all. Check --detail-platform and your vendor's documentation before relying on a move.
  • Old metadata formats. 0.90 arrays have device and size limits and can be confused with partition boundaries. The kernel's own in-kernel auto-detection works only with 0.90 and partition type FD, and the manual says it "is not recommended for new installations".
  • Clustered RAID. The manual warns that --zero-superblock with clustered RAID needs care, because the array may be assembled on another node. Clustered md is outside the scope of this article.
  • Encryption and LVM. Many arrays carry LUKS or LVM layers above them. Assembling the array is only the first step; the layers above have their own recovery requirements, which belong in separate procedures.
  • This article is not a tested procedure. The commands shown are taken from the documentation for inspection and read-only assembly. Your distribution's packaging, initramfs tooling and defaults may differ, so rehearse on non-production disks or images before you depend on any step.

A short recovery checklist

  1. Keep the array's UUID, level, member list, metadata format and mdadm.conf outside the server.
  2. Control automatic activation in the rescue environment before attaching old disks.
  3. --examine every member and compare UUID, role and event count.
  4. Assess the complete member state and filesystem requirements before read-only access.
  5. If members disagree or one has read errors, image the disks before considering --force.
  6. Never re-create an array over its members as a recovery step outside specialist work on images.
  7. Reconcile boot configuration, then validate data and plan any required consistency work separately.

Sources

This article was originally published on .