Australian cloud for business software — built and run in Sydney since 2003

+61 2 8282 5656
support@netleverage.com
Talk to an engineer
Talk to an engineer
NetLeverage
NetLeverage

Importing a ZFS pool on replacement hardware: hostid, cache files and feature flag compatibility

Technical article
Backup & Recovery

Originally published:

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

ZFS stores a pool's configuration on the pool's own disks. That makes pools portable: the zpool-export manual says exported devices "can be moved between systems (even those of different endianness) and imported as long as a sufficient number of devices are present".

Portability is not the same as automatic. When a server fails and its disks are moved to replacement hardware, the new host may see the pool as possibly in use elsewhere, may not import it at boot, or may not be able to open it at all because the pool uses features the installed software does not support. Each of these has a documented cause. Knowing them before the day you need them turns a risky forced import into a planned one.

Why it matters

A ZFS pool frequently holds everything that matters on a storage server: file shares, virtual machine disks, database files and the snapshots that serve as short-term restore points. When the server around it fails, the quickest recovery is often to move the disks, or a set of replicated disks, to other hardware.

The commands for this are short. The decisions are not. Forcing an import of a pool that really is in use elsewhere can corrupt it. Upgrading a pool on the new host can make it impossible to move back. Choosing recovery options from the import manual without reading their warnings can discard recent data.

How it works

Exported, active and "potentially active" pools

A cleanly exported pool is marked as no longer in use by any host. The export manual states: "All devices are marked as exported". Before exporting, "all datasets within the pool are unmounted".

A server that fails does not get to export its pools. The zpool-import manual explains the consequence: "If a device is removed from a system without running zpool export first, the device appears as potentially active. It cannot be determined if this was a failed export, or whether the device is really in use from another host. To import a pool in this state, the -f option is required."

That sentence is the heart of the matter. ZFS cannot tell, from the disks alone, whether the old server is dead or merely disconnected. The -f (force) option tells ZFS that a person has checked. The check is the important part: confirm that the old server is powered off, or at least can no longer reach those disks, before forcing an import elsewhere. With shared storage, such as a SAN or a shared disk shelf, this is not a formality.

Multihost protection

For configurations where two hosts genuinely can see the same disks, OpenZFS provides an extra safeguard. The zpoolprops manual describes the multihost property: it "controls whether a pool activity check should be performed during zpool import. When a pool is determined to be active it cannot be imported, even with the -f option." It is "intended to be used in failover configurations where multiple hosts have access to a pool on shared storage."

Two cautions from the same text matter. First, "multihost provides protection on import only. It does not protect against an individual device being used in multiple pools." Second, "in order to enable this property each host must set a unique hostid." The default is off.

The host identifier

The host identifier, or hostid, is how OpenZFS on Linux records which machine last used a pool. The spl(4) manual describes spl_hostid as a value that "can be used to uniquely identify a system", disabled by default (zero) and enabled "by placing a unique non-zero value in /etc/hostid". The zgenhostid manual describes the tool that "creates /etc/hostid file and stores the host ID in it", and stresses that "this value must be unique among your systems".

For recovery this has two sides:

  • A replacement server has a different hostid. That is correct and is part of how ZFS notices that the pool was last used elsewhere. Do not copy the old server's hostid onto new hardware merely to avoid an import warning, especially while the old server might still be running.
  • Restoring a root filesystem copies its hostid. If a server's operating system is restored onto new hardware from a backup or image, it carries the old /etc/hostid. Where that host replaces the old one, that may be intended. Where both might run, as in a test restore alongside production, they now share an identity, which defeats the purpose of multihost protection.

Some boot environments read the hostid early, from a copy inside the initramfs, so changing /etc/hostid alone may not change what the boot-time import sees. Check your distribution's ZFS boot documentation. (our reading; not verified against a specific distribution here)

The cache file

The OpenZFS FAQ explains that "whenever a pool is imported on the system it will be added to the /etc/zfs/zpool.cache file. This file stores pool configuration information, such as the device names and pool state." The zpoolprops manual adds that "all pools in this cache are automatically imported when the system boots".

A replacement server, or a freshly installed one, has no record of the pool, so cache-based boot import needs appropriate configuration. Distribution services may instead scan devices, and the cachefile property can disable caching; absence of a cache alone does not prove import is impossible. A server restored from a root filesystem backup brings back an old cache file that may describe device names that do not exist on the new hardware. The FAQ notes that for non-persistent names such as /dev/sdX, adding or removing hardware "can easily cause the names to change", and "you would then need to remove the zpool.cache file and re-import the pool using the new names." (our reading of how this applies to restores)

The FAQ discusses persistent names for most systems: /dev/disk/by-id/ for small pools and /dev/disk/by-vdev/ for large configurations. When importing on new hardware, searching a persistent directory, as in the FAQ's example zpool import -d /dev/disk/by-id, controls the discovery paths used. The no-pool-name form lists candidates; an actual import and the cachefile configuration determine whether configuration is cached. The FAQ also notes that since OpenZFS 2.2 a vdev's path can be changed without exporting, with the warning that zpool set path "does not check whether the path exists or identifies the same device".

Feature flags: whether the new software can open the pool

Modern pools no longer use a single version number. The zpool-features manual says pool formats "are specified via 'features' which replace the old on-disk format numbers (the last supported on-disk format number is 28)". Each feature is in one of three states:

  • active: its on-disk changes are in effect, and "support for this feature is required to import the pool in read-write mode". If it is not read-only compatible, support is also needed to import the pool read-only.
  • enabled: marked for use but not yet in effect; "the pool can still be imported by software that does not support this feature, but changes may be made to the on-disk format at any time".
  • disabled: not in use. The manual is direct: "Features cannot be disabled once they have been enabled."

The manual also defines read-only compatibility: "If all unsupported features on a pool are read-only compatible, the pool can be imported in read-only mode by setting the readonly property during import."

The recovery consequence is that the replacement host's ZFS version matters. A rescue system, an older distribution release or another operating system's ZFS implementation may support fewer features than the software that created and upgraded the pool. If the pool has an active feature the rescue software lacks, it may not open the pool read-write, and possibly not at all.

Upgrades are one-way

The zpool-upgrade manual explains that zpool upgrade "enables all supported features on the given pool", and warns that a pool upgraded from a legacy version "will no longer be accessible on systems that do not support feature flags". Pools reported as not having all features enabled "can continue to be used". Combined with the rule that enabled features cannot be disabled, this makes upgrading a pool on a replacement server a decision about where the pool can go next.

Compatibility feature sets

OpenZFS provides a way to limit a pool to a known feature set. The compatibility property, described in the zpoolprops and zpool-features manuals, can be off (all features may be enabled), legacy (no features), or a list of files naming allowed features, read from /etc/zfs/compatibility.d or /usr/share/zfs/compatibility.d. "Only features present in all files are enabled." By convention, the files in /usr/share/zfs/compatibility.d "are provided by the distribution, and include feature sets supported by important versions of popular distributions". The OpenZFS source tree, at access, includes sets such as openzfs-2.1-linux, openzfs-2.2, openzfs-2.4, grub2-2.12 and compat-2018 through compat-2021 sets.

For recovery planning, this is the mechanism for keeping a pool importable by the software you would use in an emergency. If your rescue media or standby host runs an older OpenZFS release, a suitable compatibility set constrains future feature enablement. It does not disable features already enabled or active, so compare the existing pool state with the recovery host before relying on it.

Encrypted datasets

Native ZFS encryption adds one more step. The import manual's -l option "will request encryption keys for all encrypted datasets it attempts to mount"; "without this flag encrypted datasets will be left unavailable until the keys are loaded". Keys stored as files on the old server's root filesystem, or entered by a person, must be available on the new host. Plan where keys and passphrases live so that a server failure does not also take the keys with it.

What to choose and why

Before anything fails

  • Record pool details outside the server. Keep zpool status with full device paths, the pool's GUID, zpool get all for properties including compatibility, the list of active features, the OpenZFS version and the hostid.
  • Use persistent device names. Persistent names reduce device-order dependence, but the recovery host must still expose the correct disks and enough healthy members.
  • Decide the oldest software that must be able to import the pool, such as your rescue media or a standby host, and set compatibility to match. Avoid casual zpool upgrade runs.
  • Plan encryption keys so that they can reach a replacement host.

On the day

  1. Confirm the old host cannot use the disks. Power it off, or physically or logically disconnect the storage from it. Only then consider a forced import.
  2. List before importing. zpool import with no pool name lists pools available for import and their health without importing anything.
  3. Prefer a read-only, unmounted first import when the pool's state is uncertain. The import manual documents -N ("import the pool without mounting any file systems") and the readonly=on property set at import time, and -R to use an alternate root, which also defaults the cache file to none. That prevents this import from populating the ordinary cache, but separate boot-time scanning services still need checking. These controls separate import from dataset mounting and restrict pool writes; they do not prove the data is healthy or justify reading failing disks before preservation.
  4. Leave the pool's features alone until the recovery is finished and you know where the pool will live.

Read the recovery options' warnings before using them

The import manual documents options for damaged pools, and each comes with a cost stated plainly:

  • -F attempts recovery "by discarding the last few transactions", and "the data from the discarded transactions is irretrievably lost". -n with -F checks whether recovery would work "but does not actually perform the pool recovery".
  • -X and -T roll back further and carry the warning that they "can be extremely hazardous to the health of your pool and should only be used as a last resort".
  • -m allows import with a missing log device; "recent transactions can be lost".
  • --rewind-to-checkpoint discards everything after a checkpoint; "there is no way to undo the rewind".

These are not first steps. If a pool will not import normally, stop, record the exact messages and get expert help before choosing one.

Risks and limits

  • Implementation differences. This article uses OpenZFS documentation. Oracle Solaris ZFS has diverged and uses different feature support; pools may not move between them freely.
  • Boot pools. Pools that hold the operating system also depend on the boot loader's ZFS support, which the grub2 compatibility sets address. Booting from ZFS on new hardware is a separate topic.
  • Shared storage. Multihost needs unique hostids and adds periodic writes. It protects import only.
  • Nothing here was run. Options and properties are quoted from manual pages for planning. Rehearse imports on replicated or non-production pools.
  • Replication is not covered. Moving data with zfs send and zfs receive is the usual alternative to moving disks and deserves its own plan.

Sources

This article was originally published on .