Australian cloud for business software — built and run in Sydney since 2003

NetLeverage
NetLeverage

A Linux Filesystem Became Read-Only: Preserve the First Error

Technical article
Troubleshooting

When a Linux filesystem unexpectedly becomes read-only, preserve the earliest relevant storage and filesystem errors before attempting recovery. The read-only state may be a protective response to an earlier fault. Forcing writes back on can hide the sequence without correcting the cause.1

First establish what actually failed: a mount marked read-only, a filesystem that has shut down, an underlying device problem, or an application denied access for another reason. These can produce similar user-facing symptoms but require different investigations.

This guide provides an evidence and decision workflow, not an executable repair procedure.

Start with the exact failure

Record the application's original error, affected path, operation and time. “Unable to save” is less useful than the underlying error indicating a read-only filesystem, an I/O failure or a permission problem. Preserve the original wording even if the support summary uses plain language.

Identify the mount containing that path from the application's own execution context. Container mount namespaces and bind mounts can make a host-wide assumption misleading. The findmnt manual describes finding the filesystem for a target path and selecting output columns; use an approved local reporting method appropriate to the installed version.3

Capture the source device, filesystem type, target and current options. Current options describe the present state; they do not prove how or why it changed. An intentionally read-only mount must be distinguished from one that changed during service.

Also record the filesystem and device identifiers used in the earlier messages. A plausible error from another mounted device is not evidence for the failing application. Matching time, path and device is part of diagnosis.

ext4 and XFS do not fail in identical ways

The ext4 documentation describes three error policies: errors=remount-ro, errors=continue and errors=panic. The first remounts read-only on a filesystem error; the others continue or halt the machine. Effective policy matters when interpreting the observed response.1

Do not generalise that every Linux filesystem remounts read-only after corruption. XFS can shut down after detecting corruption, with applications receiving I/O failures. A mount listing alone is therefore insufficient to establish that XFS remains usable. Preserve the XFS messages and investigate that filesystem's specific recovery requirements.2

This distinction changes the question asked of support. “Which error caused ext4 to apply its configured policy?” is different from “Why did XFS shut down?” Neither question is answered by changing application permissions.

An administrator should also separate filesystem damage from an intentional configuration change. If a deployment changed mount policy, evidence may point to that change instead of a storage incident. Do not diagnose corruption solely from the word “read-only”.

Preserve the first error, not only the final symptom

Build a short timeline from the earliest relevant available message through the application failure. Useful records include device I/O errors, transport timeouts, filesystem messages, mount changes and application exceptions. Keep the original timestamps and timezone information.

As an editorial investigation method, work backwards from the first known failure while maintaining device identity. An ext4 message about an aborted journal may be downstream of a storage error. An application log repeated thousands of times may contribute less than one earlier kernel message.

Use already available central logging or an unaffected destination to preserve evidence. Avoid filling the affected filesystem with diagnostic output. Record gaps explicitly: if earlier messages are unavailable, the first retained message is not necessarily the first event.

A reboot changes the running state and can remove useful context. Before proposing one, establish what evidence is already preserved, whether the root filesystem is involved, and whether console or recovery access is available. These are operational decision questions, not permission to reboot.

Read-only does not always mean zero writes

The ext4 documentation warns that mounting a filesystem read-only can still replay its journal and therefore write to the partition. It also documents suppression of journal loading, with a warning that skipping replay after an unclean unmount can leave inconsistencies.1

This matters when someone proposes mounting the original evidence device “just to look”. A forensic or recovery-preservation requirement needs a procedure that accounts for journal behaviour, not merely the ordinary read-only label.

Do not turn the documented options into a universal mount recipe. Whether replay should occur depends on the recovery objective, the condition of the device and the copy on which work is performed. A qualified recovery owner should resolve those choices before modifying the source.

The same principle applies to snapshots and clones: establish what they preserve and where they reside before assuming they provide an independent recovery copy. This article makes no guarantee about any particular storage platform's snapshot protection.

Repair is not the same as restoring application data

Filesystem checkers aim to restore metadata consistency. They cannot repair faulty hardware or determine whether application records have their intended meaning, and damaged structures may be discarded during repair. Red Hat also notes that fsck.xfs exits successfully without performing the substantive XFS check.2

The documented no-modify check is xfs_repair -n on the identified, unmounted filesystem. Its prerequisites and dirty-log handling still need specialist review: any mounting to replay a log is a separate write-capable action, not part of a supposedly read-only check. A dry run neither repairs the filesystem nor validates application data.2

A zero exit code from the wrong tool is therefore not an acceptance test. Nor is successful mounting proof that a database, repository or document collection is complete.

Decide the recovery objective before choosing repair: preserve evidence, extract important data, restore service from a known recovery point, or assess a recoverable copy. These objectives can lead to different sequences and different acceptable risks.

Confirm the storage fault has been addressed before relying on further writes. Where hardware or lower-layer I/O remains unreliable, a filesystem-only intervention has not resolved the initiating cause. Record which layer was repaired and which layers still need verification.

Separate the diagnostic and recovery decisions

A useful incident handoff contains the affected service and path, verified mount and device identity, exact first retained error, effective filesystem policy, relevant recent changes, evidence gaps and recovery options already available.

The recovery owner then needs an approved maintenance plan, data-preservation decision, filesystem-specific prerequisites and an application acceptance test. This article supplies no remount, log-zeroing or filesystem-repair commands because those operations can change recoverable state.

For a hypothetical case, suppose an application starts reporting read-only errors after a storage timeout, and ext4 subsequently records a protective remount. The immediate task is to preserve that sequence and investigate the storage timeout. Reapplying read-write mount options would not establish that the storage path is reliable.

If, instead, an intentionally read-only bind mount appeared in a deployment change, the evidence points to configuration review. The symptom alone does not choose between these cases.

Verify service recovery at more than one level

After an authorised intervention, check the filesystem's expected state, review new storage and filesystem errors, and perform the application's approved integrity and functional checks. Compare those results with the stated recovery objective.

Use a defined observation window suitable for the fault. “No new errors in the retained observation period” is a defensible statement; “the disk will never fail again” is not. Retain any uncertainty about missing logs or unverified data.

The final record should explain the initiating fault when known, the protective response, the layer corrected and the evidence supporting resumed service. If the initiating cause is still unknown, say so even when service has returned.

Learning notes

  • Read-only status is a symptom that needs device and timeline evidence.
  • ext4 protective remounts and XFS shutdowns are different mechanisms.
  • A read-only ext4 mount can still involve journal writes.
  • Filesystem consistency and application-data correctness require separate checks.

Acceptance checklist

  • [ ] The actual mount/filesystem state and earliest storage errors are preserved.
  • [ ] The initiating device/filesystem fault is distinguished from its protective response.
  • [ ] The approved intervention records write-capable steps and remaining data risks.
  • [ ] Filesystem state, new errors and application integrity/function are checked over a defined interval.

References

  1. ext4 General Information — The Linux Kernel documentation. Accessed 2026-11-01.
  2. RHEL 9: Checking and repairing a file system. Accessed 2026-11-01.
  3. findmnt(8) - Linux manual page. Accessed 2026-11-01.

Related articles