Australian cloud for business software — built and run in Sydney since 2003

NetLeverage
NetLeverage

Linux VM in Emergency Mode After a Disk Change: fstab Checks

Technical article
Troubleshooting

Direct answer: On a systemd-based Linux VM, local file systems listed in /etc/fstab become mount units that local-fs.target requires, unless the entry carries nofail, noauto or its own x-systemd.wanted-by= or x-systemd.required-by= dependencies. If one of those required entries names a device that no longer exists or no longer matches (a removed disk, a device name that moved, a file system re-created with a new UUID), the mount fails, local-fs.target is not reached and boot stops at the emergency shell. Find the failing entry from the installed system's own journal and findmnt --verify, compare its identifier with the disks that are present, and confirm the identifier is unique before any change. Editing fstab, changing disk attachments and adding nofail are changes with their own decision; nofail states that the server may run without that data.

Why it matters

The VM looks dead from the network: no SSH, no services, no monitoring agent. The console shows an emergency shell. This can follow routine work: a data disk removed or added, a disk restored onto a new virtual disk, or a controller change that reorders device names. The risk is in the shortcut. Commenting out lines or adding nofail everywhere brings the server up without the data it expects, and applications can then write into the empty mount point on the root file system.

How it works

fstab becomes systemd units

The systemd-fstab-generator manual states that it "translates /etc/fstab ... into native systemd units early at boot and when configuration of the system manager is reloaded." The systemd.mount manual describes the default dependencies: local mount units get an After= dependency on local-fs-pre.target and a Before= dependency on local-fs.target "unless one or more mount options among nofail, x-systemd.wanted-by=, and x-systemd.required-by= is set." With noauto, "the mount unit will not be added as a dependency for local-fs.target or remote-fs.target", so it is not mounted at boot unless another unit pulls it in. Network file systems are tied to remote-fs.target instead, and network block storage such as iSCSI needs _netdev so systemd recognises it as a network mount.

Why the result is an emergency shell

systemd.special describes emergency.target as the most minimal way to get an interactive shell, used "when a file system check on a required file system fails and boot-up cannot continue." A required mount whose device never appears, or which fails to mount, leaves the boot in the same position.

Device names, UUIDs, labels and clones

fstab(5) recommends LABEL= or UUID= instead of a device name, because device names follow hardware detection order and, in the manual's words, "can change when other disks are added or removed." PARTUUID= and PARTLABEL= are also supported for GPT partitions. This applies to block-device entries. Network file systems, bind mounts, pseudo file systems such as tmpfs, and swap files are named differently and are not candidates for a UUID.

Two kinds of disk change break boot:

  • Device name entries (/dev/sdb1) follow detection order. After a disk change the name can refer to a different disk or to nothing.
  • UUID or label entries point at a specific file system. If the file system is re-created, or a disk is replaced by an empty one, the identifier no longer exists. Whether a restore keeps the UUID depends on whether it copies the disk image or re-creates the file system and copies files; check rather than assume.

A cloned or restored disk attached alongside its original can carry the same UUID or label. The findmnt manual calls a duplicated tag "an unusual situation" and offers a SOURCES column that lists every device where a tag from fstab is detected. A matching UUID therefore does not prove the entry selects the intended disk.

What nofail and device-timeout change

The systemd.mount manual: "With nofail, this mount will be only wanted, not required, by local-fs.target or remote-fs.target. Moreover, the mount unit is not ordered before these target units. This means that the boot will continue without waiting for the mount unit and regardless whether the mount point can be mounted successfully." So a nofail file system is not guaranteed to be mounted when services start, and a missing nofail disk does not fail the boot; monitoring has to catch it.

x-systemd.device-timeout= is separate. It configures "how long systemd should wait for a device to show up before giving up on an entry from /etc/fstab", and only works in /etc/fstab. It bounds the wait for the device and anything that depends on it; it is not what lets a nofail boot continue.

Read-only checks

These assume the installed system's own emergency shell, where /etc/fstab and the journal belong to the failing system. None were run for this article.

If you are in separate rescue media instead, the commands below read the rescue system's own files. findmnt --verify defaults to /etc/fstab of the running environment; it accepts --tab-file to verify another file, but finding and mounting the installed root (read-only) and reading its journal is distribution-specific. Use your distribution's rescue documentation for that step and treat every mount in it as a change to record.

  1. Which unit failed: read the current boot's journal for failed mount units, failed systemd-fsck units and device timeouts. Note the mount point named.
  2. The fstab entries: findmnt --verify --verbose The findmnt manual describes --verify as checking /etc/fstab "parsability and usability". Note every warning or error.
  3. What the system can see: lsblk --fs The blkid manual recommends lsblk --fs for an overview of file systems. If you use blkid, run it as root; for unprivileged users it may return cached, unverified information.
  4. Uniqueness: for each fstab tag, list every device that carries it (findmnt --fstab -o TARGET,SOURCE,SOURCES per the findmnt manual's SOURCES column). More than one device for a tag means the tag is ambiguous; an attached clone or copy is one possible cause, and independently created filesystems can also share a label. Stop and identify which disk is intended before any change.
  5. Compare: for each failing entry, check whether its identifier appears in the lsblk output, and whether it points at the disk you expect by size, file system type and VM disk position.
  6. VM side: in vCenter, compare the VM's disks and their order with the record from before the change.

Candidate causes (hypotheses to test)

Hypothesis Evidence that supports it Where a change would be made
Data disk removed, fstab entry kept Entry's UUID absent from lsblk --fs fstab, after a decision about the data
Device name entry now points elsewhere or nowhere /dev/sdX entry; disk order changed fstab
File system re-created by restore or reformat Old UUID absent; a new UUID on the expected disk fstab, after confirming the disk's contents
Ambiguous tag: a clone or copy with the same UUID or label, or separate filesystems sharing a label SOURCES lists more than one device VM disk attachment, decided with the data owner
Disk replaced by an empty disk No file system on the expected disk Restore the data first
Network or iSCSI file system treated as local Mount fails before the network is up fstab (_netdev)
File system check failure systemd-fsck unit failed File system repair, separate decision
Syntax error in fstab findmnt --verify error fstab

Decision guidance

Every item here is a change. Each needs console access, a change record and, where data is involved, the application owner's agreement. In the emergency shell the root file system may be mounted read-only; remounting it read-write is itself a change. Copying /etc/fstab before editing is also a write; record where the copy went.

Finding Decision
The data should exist and its disk is not attached, or the wrong disk is attached Correct the VM's disk attachment first; do not edit fstab to match the wrong disk
A duplicate tag is present Decide with the data owner which disk stays attached; do not reformat or change identifiers as part of diagnosis
A device-name entry on a block device Replace it with the UUID= or LABEL= of the intended, uniquely identified file system
A disk deliberately retired Remove its entry and record that the data is no longer expected
A file system the server can genuinely run without nofail is acceptable, with monitoring for the absent mount and the reason recorded
Network block storage _netdev, per systemd.mount
A failed file system check File system repair is a separate decision with its own backup

After any change, findmnt --verify again, then a reboot in an agreed window, confirming every intended file system is mounted and services start.

What not to do

  • Do not reboot repeatedly before reading the journal and verify output.
  • Do not add nofail to every entry. Boot then continues whether or not the data is present.
  • Do not comment out entries without recording which file systems are now absent.
  • Do not take the first UUID that matches; check for duplicates first.
  • Do not run a file system repair as a first response to a missing device.
  • Do not switch to /dev/sdX names to match what is there now.
  • Do not reformat a disk that does not mount until you know what was on it.

Limitations

  • No command was run for this article. Lab validation is outstanding.
  • Root file system and initramfs failures are covered in the disk controller article.
  • LVM, software RAID, encrypted volumes and multipath add their own identifiers and are not covered.
  • Rescue-media procedures and the exact emergency shell text are distribution-specific and not sourced here.
  • The default device wait time is not stated in the fetched pages, so no number is given.

Acceptance checklist

  • [ ] The failing entry was identified from the installed system's journal and fstab, not the rescue system's.
  • [ ] Every block-device fstab entry uses UUID=, LABEL=, PARTUUID= or PARTLABEL=; network, bind, pseudo and swap-file entries are checked on their own terms.
  • [ ] No fstab tag is carried by more than one attached device.
  • [ ] findmnt --verify reports no errors.
  • [ ] Each nofail entry is recorded with the reason the server can run without it, and monitoring alerts when it is absent.
  • [ ] Network-backed block storage carries _netdev.
  • [ ] After a reboot, every intended file system is mounted and services are healthy.

Sources

Displayed dates as shown when accessed on 30 October 2026.

  • Linux manual pages, fstab(5). util-linux upstream 2026-08-03; HTML rendering 2026-09-09.
  • systemd manual, systemd.mount. Not displayed (systemd 262).
  • systemd manual, systemd-fstab-generator. Not displayed (systemd 262).
  • systemd manual, systemd.special. Not displayed (systemd 262).
  • Linux manual pages, blkid(8). util-linux upstream 2026-08-03; HTML rendering 2026-09-09.
  • Linux manual pages, findmnt(8). util-linux upstream 2026-08-03; HTML rendering 2026-09-09.

Related articles

Learning notes

  1. Local fstab entries are required for local-fs.target unless nofail, noauto or explicit x-systemd dependencies say otherwise.
  2. Device names follow detection order; use UUID or LABEL for block devices.
  3. A clone can carry the same UUID, and separate filesystems can share a label; check uniqueness before trusting a match.
  4. nofail means boot continues regardless of the mount; x-systemd.device-timeout= only bounds the device wait.
  5. In rescue media, the default commands read the rescue system, not the failed one.