Australian cloud for business software — built and run in Sydney since 2003

+61 2 8282 5656
support@netleverage.com
Talk to an engineer
Talk to an engineer
NetLeverage
NetLeverage

A failing disk in a Linux server: image it before you repair it

Technical article
Backup & Recovery

Originally published:

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

A disk can fail gradually or disappear without useful warning. It returns read errors for some areas, slows down, logs errors in the kernel journal, and the filesystem on it may go read-only or refuse to mount. The natural reaction is to run fsck or xfs_repair and get the server back.

On a failing device, that order can turn a partial loss into a larger one. A filesystem repair writes to the disk, and writes based on unreadable metadata can make the damage permanent. Additional reads can stress unstable media or consume the time remaining before it becomes inaccessible. The preservation goal is to stop avoidable writes and decide whether specialist handling or controlled imaging is appropriate. Physically unstable media and irreplaceable data can justify professional recovery before further power cycles or reads. When software imaging is appropriate, preserve what can be read and assess repairs on working copies.

This guide covers that order for Linux servers. It assumes there is no current backup that makes the disk's contents expendable. If there is one, restoring from it is usually faster and safer than recovering a failing disk.

Why it matters

The cost of the wrong order is that it cannot be undone. A repair tool that "fixes" a directory by discarding entries it could not read, or that zeroes a journal, makes decisions about your data based on what a failing disk happened to return at that moment. An image taken first lets you try a repair, look at the result, and try again differently if the first attempt was wrong. Without a preserved copy, later attempts inherit every change made by earlier ones.

The manuals for the common repair tools reflect this. The xfs_repair manual warns of the -L option: "When using this option the filesystem will likely appear to be corrupt, and can cause the loss of user files and/or data." On replaying a dirty log by mounting, it advises: "Please make sure that the machine's hardware is reliable before replaying to avoid compounding the problems."

How it works

Step 1: stop writing

If the filesystem is still mounted read-write, every write is a risk: new data may land on failing areas, and the filesystem's own housekeeping keeps writing. Where possible, stop the services using the disk and unmount it, or remount it read-only. Stopping services, unmounting and a clean shutdown can themselves flush writes. Choose the transition with someone qualified to assess the failure; do not repeatedly reboot or cycle power on mechanically unstable media.

Two Linux mechanisms help keep a device read-only. The blockdev manual describes --setro, with an important limit: "The currently active access to the device may not be affected by the change. For example, a filesystem already mounted in read-write mode will not be affected. The change applies after remount." The losetup manual offers -r or --read-only to "set up a read-only loop device", which is useful later when working with image files.

A note of care: some rescue environments and desktop systems automatically mount, check or activate storage they find, such as RAID arrays and LVM volume groups. Know what your rescue media does on boot before attaching a failing disk to it.

Step 2: record the evidence

Before copying, record what the system is saying. The kernel log shows which device and which sectors returned errors. The smartctl manual describes the drive's own view:

  • -H prints the device's health status. "If the device reports failing health status, this means either that the device has already failed, or that it is predicting its own failure within the next 24 hours. If this happens, use the '-x' option to get more information, and get your data off the disk and to someplace safe as soon as you can."
  • -x "prints all SMART and non-SMART information about the device", including error logs and self-test logs.
  • -l error prints the drive's error log; for ATA, "SMART disks maintain a log of the most recent five non-trivial errors."

Saving this output to separate storage serves two purposes: it supports a warranty or support claim, and it records the drive's state before you added any load to it.

Two cautions. A drive that reports healthy can still be failing; read errors in the kernel log are evidence in their own right. And smartctl -t starts a self-test, which reads the whole surface in its long form. On a disk that is already failing, that is extra wear you may not want before the data is copied. (our reading; the smartctl manual describes the tests but does not advise on this trade-off)

Step 3: copy with a tool built for bad media

Ordinary copy tools stop at the first read error, or retry each bad area many times before moving on. GNU ddrescue is designed for the opposite: "It copies data from one file or block device (hard disc, cdrom, etc) to another, trying to rescue the good parts first in case of read errors."

Rescuing the good parts first matters because a failing disk may not last. Getting most of the readable data off quickly, then returning to the difficult areas, gives the best result if the drive dies partway through.

The ddrescue man page's central advice concerns the mapfile:

"Always use a mapfile unless you know you won't need it. Without a mapfile, ddrescue can't resume a rescue, only reinitiate it. Be careful to not specify by mistake an old mapfile from an unrelated rescue."

The mapfile records which areas have been copied, which failed and which are untried. It allows a rescue to be stopped and resumed, which matters when a drive overheats, disconnects or needs to rest, and it shows how much of the disk was recovered. In versions before 1.20 it was called the logfile.

Other cautions from the same page:

  • "If you reboot, check the device names before restarting ddrescue." After a reboot, /dev/sdb may be a different disk. Copying in the wrong direction would overwrite the data you are trying to save. Check serial numbers, not just names.
  • "Don't use options '-F' or '-G' without reading the manual first." Fill mode and generate mode change how the output or mapfile is treated.
  • -f (--force) is required to "overwrite output device or partition". Treat its use as a deliberate decision about which device is the destination.

Practical choices (our recommendations):

  • Copy to healthy storage with more free space than the failing disk's full size. Imaging to a file on another filesystem is often easier to manage than imaging to a raw disk, and keeps the original layout intact.
  • Image the whole disk, not one partition, unless space forces a choice. The partition table, boot areas and other partitions are part of what you may need.
  • Keep the mapfile with the image and on separate storage from the failing disk.
  • Do not run ddrescue's retry passes indefinitely on a disk that is clearly deteriorating. The manual's -r option sets the number of retry passes (default 0). Every pass adds load and may reduce the time available for useful recovery. Decide in advance when a partial image is good enough to work with.

If the drive cannot be read at all, has physical symptoms such as clicking, or holds data whose value justifies it, stop and use a professional data recovery service. Software imaging is not a substitute for a clean-room recovery when the mechanics have failed.

Step 4: work on the copy

Once an image exists, everything risky happens on a copy of the image, never on the original disk and preferably not on the only image. An image file can be attached as a read-only loop device to inspect it, and with a working copy, as a writable loop device to repair it.

Before repairing, check in no-change mode:

  • The e2fsck manual describes -n: "Open the file system read-only, and assume an answer of 'no' to all questions." It also says plainly that "in general it is not safe to run e2fsck on mounted file systems", and that if e2fsck asks whether to check a mounted filesystem, "the only correct answer is 'no'".
  • The xfs_repair manual describes -n: "No modify mode. Specifies that xfs_repair should not modify the filesystem but should only scan the filesystem and indicate what repairs would have been made." It also requires the filesystem to be unmounted, "otherwise, the resulting filesystem may be inconsistent or corrupt."

For XFS, the manual explains a common trap. If the log is dirty, xfs_repair "cannot replay a dirty log and will exit with a status code of 2". The log "can be replayed by mounting and immediately unmounting the filesystem on the same class of machine that crashed". Only if mounting fails does the manual describe erasing the log with -L, and it warns: "All metadata updates in progress at the time of the crash will be lost, which may cause significant filesystem damage. This should only be used as a last resort." Working on a copy makes that last resort survivable: if the result is poor, you still have the image.

Metadata images for expert help

When specialist help is needed, the filesystem tools can produce metadata-only images. The e2image manual says it saves "critical ext2, ext3, or ext4 file system metadata", which "can assist an expert in recovering catastrophically corrupted file systems". The xfs_metadump manual describes a tool that copies XFS metadata and by default "obfuscates most file... names and extended attribute names to allow the dumps to be sent without revealing confidential information", with stated exceptions: short names, names in the log, and log recovery that "may expose clear-text metadata". Metadata can still reveal sensitive information, so decide what may be shared under your privacy obligations.

What to choose and why

  • Restore from backup if you can. A known-good backup beats a recovered image of a failing disk.
  • If you must recover: stop writes, record SMART and kernel evidence, image with a mapfile to healthy storage, and repair only a copy of the image.
  • Mirrored disks change the picture. If the failing disk is one member of a healthy RAID 1 or similar, the array may already hold a good copy. Replacing the failed member under the array's normal procedure may be more appropriate than imaging. Check the array's state before deciding, and image the remaining members first if their health is in doubt.
  • Encrypted disks are imaged as they are, encrypted. The image needs the same key material to open, so the image still needs usable LUKS header information and a working unlock method.

Risks and limits

  • Device mix-ups can destroy the source. The destination of a copy is overwritten. Identify disks by serial number and size before every command.
  • Solid-state drives behave differently from spinning disks: they may fail suddenly, become read-only or vanish from the bus. The principle of copying first still applies.
  • The full GNU ddrescue manual could not be independently opened; it returned an access error. It describes the rescue algorithm and advanced options in detail. This article therefore stays at preservation and planning level. It is not a validated imaging recipe, including for option combinations that look simple.
  • Nothing here was run. All commands are quoted from manual pages for planning.
  • Legal and privacy duties continue. An image contains everything the disk held. Store, transfer and destroy it under the same controls as the original.

Sources

This article was originally published on .