Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

"UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY" at boot: what to do first on an ext4 server

Technical article
Troubleshooting

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

The boot message "UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY. (i.e., without -a or -p options)" comes from e2fsck, the checker for ext2, ext3 and ext4 filesystems. The boot scripts run it in its automatic mode, and in that mode it stops if it finds a problem that might need an administrator's judgement. Safe repairs may already have been made before it stopped, and the server will not finish booting. Take the copy before anything further.

Do not answer "yes" to everything on the first try. Make a copy of the disk first, run the checker read-only to see the scale of the damage, and only then repair, with an undo file so the repair can be reversed.

What the message means

The e2fsck source shows exactly when the message is printed. A function called preenhalt returns immediately unless e2fsck was started in preen mode (-p, or its older alias -a). In preen mode it prints "UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY." with the device name in front and "(i.e., without -a or -p options)" on the next line. It then sets the error flag in the filesystem superblock and exits with a code named FSCK_UNCORRECTED, which the manuals list as 4, "errors left uncorrected".

The e2fsck manual describes preen mode. The -p option "will cause e2fsck to automatically fix any file system problems that can be safely fixed without human intervention". If it finds a problem "which may require the system administrator to take additional corrective action, e2fsck will print a description of the problem and then exit with the value 4 logically or'ed into the exit code". The manual adds that this option "is normally used by the system's boot scripts".

What happens next at boot

The systemd-fsck manual explains the consequence. For the root and /usr filesystems, systemd-fsck-root.service and systemd-fsck-usr.service "will activate reboot.target if fsck(8) returns the 'System should reboot' condition, or emergency.target if fsck returns the 'Filesystem errors left uncorrected' condition". For other filesystems listed in /etc/fstab without nofail or noauto, a failed check leads local-fs.target to activate emergency.target as well. That is why you land at an emergency prompt.

The exit codes

The fsck and e2fsck manuals give the same table: the exit status is the sum of 0 (no errors), 1 (errors corrected), 2 (errors corrected, system should be rebooted), 4 (errors left uncorrected), 8 (operational error), 16 (usage error), 32 (cancelled) and 128 (shared library error). The boot-time stop corresponds to a 4.

Likely causes

Cause What you see How to confirm
Unclean shutdown plus real damage Power loss, crash or forced stop of the VM before the message Kernel and journal lines from the previous boot; an e2fsck report listing specific inodes or blocks
Storage layer errors I/O errors in the kernel log before the failure; other disks or the datastore also in trouble journalctl kernel messages; storage or hypervisor logs
Wrong device or stale /etc/fstab entry The device named in the message is not the disk you expect lsblk -f and blkid against the name in the message
Filesystem damaged by a bad copy or restore Problem appears right after a clone, snapshot revert or restore History of the last change to the disk
Not an ext2/3/4 filesystem Different message at boot lsblk -f shows xfs or another type; use that type's own tool

Prove the cause before changing anything

  1. Note the device named in the message (the text before "UNEXPECTED INCONSISTENCY").
  2. At the emergency prompt, identify the device. The lsblk manual says -f outputs "info about filesystems" including type, label, UUID and mount points, and the same manual says the "authoritative information about filesystems and raids is provided by the blkid(8) command":
lsblk -f

Make sure the filesystem type is ext2, ext3 or ext4, and that it is not mounted (the MOUNTPOINTS column is empty for that device). The e2fsck manual says "in general it is not safe to run e2fsck on mounted file systems".

  1. Read the checker's earlier output in the journal. The journalctl manual says -b shows messages from a specific boot and -u shows messages for a unit:
journalctl -b -u systemd-fsck-root.service

Use systemd-fsck@<device unit>.service for a filesystem that is not the root. If the journal is not available in the emergency shell, read what is on the console.

  1. Look at the superblock summary, which does not change anything. The dumpe2fs manual says -h shows only superblock information:
dumpe2fs -h /dev/sda2

Use your own device name. The manual warns that on a mounted filesystem the information "may be old or inconsistent", which is another reason to check an unmounted device.

  1. Before any repair, make a copy. See "A failing disk in a Linux server: image it before you repair it" for imaging, and for a virtual machine take a snapshot or copy of the virtual disk at the hypervisor with the VM stopped.

    Disruptive step: stopping the VM powers it off, so everything that runs on it is down until you start it again. The copy is a read of the disk and does not change it. To back out, power the VM on.

  2. Run the checker read-only. The e2fsck manual says -n opens the filesystem read-only and assumes an answer of no to every question, and -f forces checking "even if the file system seems clean":

e2fsck -f -n /dev/sda2

If the journal needs recovery, e2fsck in this mode skips it. The e2fsck source prints "Warning: skipping journal recovery because doing a read-only filesystem check." Errors reported by -n on a filesystem whose journal has not been replayed may therefore be errors that a replay would have cleared, so read the report with that warning in mind.

Read the report. A short list of minor items is a different situation from thousands of lines about the inode table, which points to a failing disk or storage layer. Do not continue with repair if the report suggests hardware faults; follow the imaging steps first.

Example (illustrative): db01 stops at boot with "/dev/sdb1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY." and an emergency prompt. lsblk -f shows /dev/sdb1 as ext4, unmounted. The administrator stops the VM and takes a copy of the virtual disk at the hypervisor. The kernel log from the previous boot shows no I/O errors, only a hard power-off. e2fsck -f -n /dev/sdb1 lists 11 lines about orphaned inodes and a wrong free-block count, and exits with status 4. That is a small, explainable list after an unclean stop, so the administrator goes ahead with an interactive repair.

Fix

Repair interactively, with an undo file

The message itself tells you to run e2fsck without -a or -p. In interactive mode the manual says e2fsck "will ask the user to fix each problem found in the file system", with 'y' to fix, 'n' to leave it, 'a' to fix this one and all later ones, and Enter for the default.

Use an undo file so a bad repair can be reversed. The manual says -z undo_file writes the old contents of each block before it is overwritten, and that the file "can be used with e2undo(8) to restore the old contents of the file system should something go wrong". It adds a warning: "The undo file cannot be used to recover from a power or system crash." The manual does not say where to keep the file. The advice here is to keep it on a different filesystem from the one being repaired. In an emergency shell the root filesystem may be read-only or may be the damaged one, so mount a writable filesystem from another disk first, or name an undo path on it.

Disruptive step: this command writes changes to the filesystem.

e2fsck -f -z /root/db01-sdb1.e2undo /dev/sdb1

To back out, with the filesystem still unmounted, replay the undo file. The e2undo manual describes replaying an undo log for an ext2/ext3/ext4 filesystem, and offers -n for a dry run that does not write blocks:

e2undo -n /root/db01-sdb1.e2undo /dev/sdb1

Disruptive step: the replay itself rewrites filesystem blocks, so it changes the device again. The e2undo manual says it checks the filesystem superblock against the undo log and refuses to apply a log that does not match; its -f option disables that safety check, and it should not be used without a reason you can state. The undo file cannot help after a power or system crash during the repair. Back out of the replay by restoring the copy you made before starting.

e2undo /root/db01-sdb1.e2undo /dev/sdb1

Other backouts: restore from the copy you made before starting.

Answering yes to everything

The manual describes -y as: "Assume an answer of 'yes' to all questions; allows e2fsck to be used non-interactively." The same repair applies, but you do not see or choose each fix.

Disruptive step: -y writes every fix the checker proposes to the filesystem, including ones you would have declined. Use it only after the read-only report is small and understood, and keep the undo file (-z, as above). To back out, replay the undo file with e2undo, or restore the copy; the limits above apply.

Alternative superblock

If the report says the primary superblock is damaged, the manual's -b superblock option uses a backup. It says the backup locations depend on block size, blocks per group and features such as sparse_super, and that mke2fs -n can print where they are.

Disruptive step: repairing from a backup superblock writes to the filesystem; the e2fsck manual says it "will make sure that the primary superblock is updated appropriately". Use it with the undo file (-z) and only after the copy exists. To back out, replay the undo file with e2undo, or restore the copy. This is an expert step.

Make the next boot ask for the right thing

The systemd-fsck manual documents kernel command line options fsck.mode= (auto, force, skip) and fsck.repair= (preen, yes, no). The default for fsck.repair= is preen, and yes answers yes to all questions. Do not change these to avoid the message. A filesystem that preen mode refuses to repair needs a person to look at it.

Confirm the fix

  1. Re-run e2fsck -f -n /dev/sdb1. The exit status should be 0, which the manual defines as no errors.
  2. Reboot as you would after any change. The system should reach its default target without the emergency prompt.
  3. Check the data you care about: for a database, run its own consistency check; for files, compare with a recent backup or compare file counts on key directories.
  4. Read journalctl -b from the new boot for new kernel storage errors.

What to watch afterwards

  • Look for repeats. A second "UNEXPECTED INCONSISTENCY" within weeks points to a storage or hypervisor fault, not bad luck.
  • Watch the storage layer: datastore latency or errors, disk SMART data, snapshot chains on the virtual machine.
  • Make sure a recent, tested backup exists before the next maintenance on the disk.
  • Fix the shutdown path that caused it: a VM powered off instead of shut down, or a host losing power.

Risks and limits

  • These commands have not been run or lab-tested. They come from the manual pages and source listed below.
  • The message wording comes from the e2fsck source (e2fsprogs main branch). Distribution releases may format the line slightly differently.
  • The undo file protects against a wrong repair. The manual states it does not protect against a power or system crash during the repair. Keep a separate copy.
  • Repairs change the filesystem. A repair can remove or relocate damaged structures, and the manuals do not promise that every file survives. Keep the earlier copy until you have checked the data.
  • This covers ext2, ext3 and ext4. XFS uses xfs_repair and has its own messages.
  • A large report or storage errors mean the disk or the layer below it needs attention first.

Sources

  • e2fsprogs source: e2fsck/util.c — e2fsprogs project, main branch, no date shown, accessed 18 November 2026
  • e2fsprogs source: e2fsck/unix.c — e2fsprogs project, main branch, no date shown, accessed 18 November 2026
  • e2fsck(8) — e2fsprogs project, snapshot 2026-08-03, accessed 18 November 2026
  • e2undo(8) — e2fsprogs project, snapshot 2026-08-03, accessed 18 November 2026
  • dumpe2fs(8) — e2fsprogs project, snapshot 2026-08-03, accessed 18 November 2026
  • fsck(8) — util-linux project, snapshot 2026-08-04, accessed 18 November 2026
  • lsblk(8) — util-linux project, snapshot 2026-08-04, accessed 18 November 2026
  • blkid(8) — util-linux project, snapshot 2026-08-04, accessed 18 November 2026
  • systemd-fsck@.service(8) — systemd project, systemd 262, no date shown, accessed 18 November 2026
  • journalctl(1) — systemd project, systemd 262, no date shown, accessed 18 November 2026