About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.
The Logical Volume Manager (LVM) sits between disks and filesystems. Physical volumes (PVs) are pooled into a volume group (VG), and logical volumes (LVs) are carved from that pool. The mapping is recorded in VG metadata areas on participating physical volumes. A PV can be configured without a metadata area, so do not assume every member contains a complete copy. When that metadata is unavailable, damaged or inconsistent with the disks in front of it, the filesystems inside the logical volumes can be perfectly intact and still out of reach.
Three features decide whether a host uses an LVM volume group after a restore or a hardware change: the metadata itself, with its automatic backup and archive copies; the devices file, which limits which disks LVM looks at; and the system ID, which restricts a VG to one host. This guide explains each, using the current LVM2 manual pages, and turns them into recovery planning.
Why it matters
LVM problems during recovery can appear as failures in higher layers. They look like a missing root filesystem at boot, a data volume that will not mount, or an empty listing where a volume group should be. The tempting responses, such as recreating the volume group, re-initialising a disk or forcing an import, are exactly the ones that can turn a recoverable situation into data loss.
Knowing what LVM keeps, and where, changes those responses. Most of the information needed to reason about a damaged or unfamiliar VG is already written down, either on the disks themselves or in text files that LVM maintains automatically. The work is to make sure those copies exist outside the server and to use read-only inspection before anything is written.
How it works
Where the metadata lives
The pvck manual describes the on-disk layout. A label header and PV header share "a 512 byte sector, usually the second sector of the device". A metadata area header follows at "offset 4096 bytes", and "a second mda_header can optionally exist near the end of the device". The metadata itself is text, stored in "an area (about 1MiB by default) immediately following the mda_header sector".
Two consequences follow. First, the VG description is human-readable text, not an opaque binary structure. Second, the metadata area holds more than the current version: pvck can list "all versions of VG metadata found in the metadata area", and has a search mode that looks in "common locations so metadata can be found if headers are damaged". A damaged header does not necessarily mean the description of the volume group is gone.
Automatic backups and archives
LVM also writes copies of the metadata to text files on the host. The example lvm.conf explains the terms: "a 'backup' is a copy of the metadata for the current system, and an 'archive' contains old metadata configurations." Both are on by default, with comments that read "Think very hard before turning this off!" The example configuration retains archives until they exceed both the minimum count of 10 (retain_min = 10) and minimum age of 30 days (retain_days = 30), and twice repeats the same instruction: "Remember to back up this directory regularly!"
The vgcfgbackup manual says that in a default installation each VG is backed up "into a separate file bearing the name of the VG in the directory /etc/lvm/backup". It adds, in capitals: "This DOES NOT back up the data content of LVs." It also suggests: "It may also be useful to regularly back up the files in /etc/lvm."
That last point matters more than it first appears. The backup and archive files live on the same host whose storage may have failed. If the root filesystem is itself on the damaged VG, the copies are inside the thing you are trying to recover. A backup of /etc/lvm, kept elsewhere, is a small, cheap safeguard.
Restoring metadata: what it does and does not do
The vgcfgrestore manual states that it "restores the metadata of a VG from a text back up file produced by vgcfgbackup", and that it "writes VG metadata onto the devices specified in back up file". With no file given, "the most recent one is used". The --list option shows "the available back up and archive files of a VG", which is read-only and a sensible first step.
The manual also carries a direct warning: "When a VG contains thin pools, changes to thin metadata cannot be reverted, and data loss may occur if thin metadata has changed. The force option is required to restore in this case." Restoring LVM's own description of a thin pool does not roll back the thin pool's internal mapping of which blocks belong to which thin volume. The two can disagree. (our reading of the warning)
For a lost PV, the manual describes recovering its recorded UUID and size, deliberately recreating a compatible PV identity, and restoring the matching VG description. These are metadata-writing operations requiring an expert recovery plan and preserved copies, not a general repair recipe. This recreates the structure. It does not recreate the data that was on the missing disk. Any logical volume with extents on that PV will have a hole where those extents were, unless the LV was mirrored or RAID-protected across PVs.
That distinction is the most important one in LVM recovery. Restoring metadata is the right tool when the data is intact and the description is wrong or missing. It is the wrong tool when the data itself is gone, because it will make the volume group look whole while part of it is empty.
The devices file: which disks LVM will look at
Current LVM releases can restrict themselves to a list of devices. The lvmdevices manual describes "the LVM devices file" as "the list of devices that lvm commands will use", located at /etc/lvm/devices/system.devices. "Removing a device from the devices file will prevent lvm from seeing or using that device."
Each entry records a device ID, made of a type and a value. The manual lists the types in order of preference: sys_wwid, "the wwid reported by the wwid sysfs file", is "the first choice"; a serial number is used "if no wwid is available"; and the device name is used "if no other type applies". There are also types for multipath, encrypted, software RAID and loop devices.
This is good protection in normal operation: it stops LVM on one host from using disks that belong to something else. It also has a predictable effect on recovery. A replacement disk, a different storage controller or a restore onto new hardware presents devices with different world-wide names or serial numbers. If the restored root filesystem carries the old devices file, its entries describe hardware that is no longer there. The lvmdevices listing marks this case: "A line begins with 'Device none' if no device on the system matches the device ID." (our reading of how this applies to restores)
The manual also documents how the feature is turned off or bypassed. If system.devices does not exist, "the devices file feature is disabled"; use_devicesfile=0 in lvm.conf disables it; and a single command can be told to ignore the file with --devicesfile "" or pointed at named devices with --devices. For investigation, a per-command override is less intrusive than editing the restored system's configuration. Updating the file properly, with lvmdevices or vgimportdevices, is then a deliberate step for the rebuilt system.
System IDs and foreign volume groups
The lvmsystemid manual explains a second gate: "The lvm(8) system ID restricts Volume Group (VG) access to one host." A VG's system ID "identifies one host as the VG owner", and "LVM on other hosts will ignore it". The ID can be custom, or assigned "automatically by LVM using some unique identifier already available on the host, e.g., machine-id or uname".
The manual names the result: "A foreign VG is a VG seen by a host with an unmatching system ID". If "the host has no system ID, and the VG does, the VG is foreign and LVM will ignore it." A VG with no system ID is unrestricted.
On recovery hardware, this can make a correctly restored data VG invisible to normal commands, by design. The manual warns that changes to system IDs "can be made in limited circumstances (see vgexport and vgimport)" and that "improper changes can result in a host losing access to its VG, or a VG being accidentally damaged by access from an unintended host." The safe reading is to treat "foreign" as information first: it tells you which host last owned the VG.
Duplicated physical volumes
Storage snapshots, cloned disks and some restore methods produce disks that are byte-identical copies, including LVM's identifiers. The vgimportclone manual addresses this: "A duplicated VG cannot be used until it is made to coexist with the original VG." vgimportclone "renames the VG associated with the specified PVs and changes the associated VG and PV UUIDs". If a recovery plan involves attaching a restored copy of a disk to a host that still sees the original, plan for this from the start rather than discovering duplicate-PV warnings during the incident.
What to choose and why
Keep the descriptions outside the server
Include /etc/lvm (backups, archives, configuration and the devices file) in whatever backs up the host's configuration, and keep a copy off the server. Alongside it, keep the plain-language picture: the output of pvs, vgs and lvs with UUIDs, the system ID if one is set, and which physical disks (by serial or world-wide name) belong to which VG. Keep enough history to identify the configuration belonging to each backup date; a single current description may not match an older restored image.
Inspect before writing
In an incident, collect the available metadata archives, on-disk headers, device identities and devices-file entries. The manuals distinguish inspection modes from repair modes. Account for automatic activation and local state updates in the chosen environment before calling a tool harmless. Compare the on-disk metadata with the external recovery record before deciding what should change.
Avoid pvcreate, vgcreate and forced imports until you know exactly what they will write, and to which device. Initialising a disk that held a PV replaces the label and metadata header that pvck could otherwise have read.
Choose the restore type to match what LVM needs
- Image restore to identical or larger disks. LVM's on-disk metadata comes back with the image. Expect the devices file to need updating on new hardware, and check for duplicates if the original disks are still attached.
- File-level restore onto newly created LVs. You are building a new LVM layout. Recreate volumes deliberately from your recorded layout, then restore files. Do not restore the old devices file or backup files over the new ones without understanding that they describe the old disks.
- Lost physical volume. First decide whether the data on it is recoverable elsewhere, for example from a mirror or RAID LV, or from backup. Recreating identities and restoring VG metadata repairs structure, not content.
Treat thin pools as their own system
Thin provisioning keeps a separate metadata store inside the pool. The lvmthin manual notes that "thin pool metadata is compact, so even small areas of damage or corruption can result in significant data loss", that thin_check runs at activation and leaves a damaged pool inactive, and that "simple repair commands are not always successful". It also advises: "Report the details of damaged thin metadata to get the best advice on recovery." Plan restores of thin-provisioned systems at the filesystem or image level, and do not expect a VG metadata restore to repair a thin pool.
Risks and limits
- Version differences. The devices file and some pvck search options are relatively recent. Long-term-support distributions may ship older LVM releases with different defaults. Check the manual pages installed on your systems.
- Clusters and shared storage. Shared VGs using lvmlockd follow different rules from local VGs with a system ID. This article covers local VGs only.
- Nothing here was run. The commands are named from the manual pages for orientation. Commands that write metadata (pvcreate, vgcfgrestore, vgimportclone, lvconvert --repair) should be rehearsed on non-production copies, ideally on a cloned disk rather than the original.
- Metadata cannot replace data. No LVM command restores data that was on a failed disk. That comes from redundancy or backups.
Sources
- pvck(8), LVM TOOLS 2.03.42: https://man7.org/linux/man-pages/man8/pvck.8.html
- vgcfgbackup(8): https://man7.org/linux/man-pages/man8/vgcfgbackup.8.html
- vgcfgrestore(8): https://man7.org/linux/man-pages/man8/vgcfgrestore.8.html
- lvm2 example configuration (example.conf.in, main branch): https://raw.githubusercontent.com/lvmteam/lvm2/main/conf/example.conf.in
- lvmdevices(8): https://man7.org/linux/man-pages/man8/lvmdevices.8.html
- lvmsystemid(7): https://man7.org/linux/man-pages/man7/lvmsystemid.7.html
- vgimportclone(8): https://man7.org/linux/man-pages/man8/vgimportclone.8.html
- lvmthin(7): https://man7.org/linux/man-pages/man7/lvmthin.7.html
This article was originally published on .