linux troubleshooting

Linux Read-Only File System: Safe Troubleshooting

Linux Read-Only File System: Safe Troubleshooting

When Linux returns an error like this:

touch: cannot touch '/srv/app/test.txt': Read-only file system

do not start by forcing the disk back to read-write mode. First identify which filesystem contains the path, confirm how it is mounted, check the kernel logs, and protect any important data.

A filesystem can be read-only because it was intentionally mounted that way, because the kernel detected storage or filesystem errors, or because you are inside a container or managed environment where that path is supposed to be immutable. The safe fix depends on the cause.

The fast helpdesk checklist

Use this read-only diagnostic sequence before changing anything:

findmnt -T /srv/app/test.txt
findmnt -no SOURCE,TARGET,FSTYPE,OPTIONS -T /srv/app/test.txt
df -hT /srv/app/test.txt
journalctl -k -b --no-pager | tail -100
lsblk -f

Replace /srv/app/test.txt with the path that failed.

The important questions are:

  1. Which mounted filesystem owns the path?
  2. Do its mount options include ro?
  3. Did the kernel report I/O, filesystem, or device errors?
  4. Is the storage local, networked, virtual, or container-managed?
  5. Is the system still able to read important files?

If the logs show repeated I/O errors, filesystem corruption, or a disappearing device, stop treating this as a permissions ticket. Preserve data and escalate according to your storage or incident procedure.

Read-only is not the same as permission denied

These errors point to different layers:

Permission denied
Read-only file system
No space left on device

Permission denied usually means the user, group, mode bits, ACL, SELinux policy, or another access-control layer rejected the operation. Changing ownership or using sudo may be relevant after you understand the policy.

Read-only file system means the mounted filesystem will not accept writes. Running the same command with sudo normally changes nothing because even root cannot write through a read-only mount.

No space left on device means the filesystem may be out of blocks or inodes. Use the workflow in How to Find What Is Eating Disk Space on Linux for that problem.

Do not run chmod 777 against a read-only filesystem. It will fail, and if it somehow succeeds after another change, you have created a new permissions problem without fixing the storage problem.

Step 1: Identify the affected filesystem

A path is not always on the disk you expect. /srv/app could be a separate partition, an NFS share, a bind mount, a container volume, or the root filesystem.

Ask Linux which mount owns the path:

findmnt -T /srv/app/test.txt

Example:

TARGET   SOURCE      FSTYPE OPTIONS
/srv/app /dev/vdb1   ext4   ro,relatime

The ro option means read-only. A healthy read-write mount normally includes rw instead.

For concise output that is easy to paste into a ticket:

findmnt -no SOURCE,TARGET,FSTYPE,OPTIONS -T /srv/app/test.txt

You might see:

/dev/vdb1 /srv/app ext4 ro,relatime

Also check capacity and filesystem type:

df -hT /srv/app/test.txt

Then inspect block devices and UUIDs:

lsblk -f

These commands help you distinguish the root filesystem from an attached data disk and prevent you from troubleshooting the wrong device.

Step 2: Confirm whether read-only was intentional

Some filesystems should be read-only. Common examples include:

  • Installation media and many ISO images
  • Recovery partitions
  • Immutable container layers
  • Read-only Kubernetes ConfigMap or Secret mounts
  • Replica or snapshot volumes
  • Network shares exported without write access
  • Hardened application paths
  • Filesystems deliberately mounted with ro in /etc/fstab

Check the current mount options with findmnt, then inspect persistent configuration when appropriate:

findmnt --verify
cat /etc/fstab

Reading /etc/fstab is safe. Editing it during an incident is a separate change and should happen only after you confirm the intended source, target, filesystem type, and options.

A line like this explicitly requests a read-only mount:

UUID=1111-2222  /srv/archive  ext4  ro,nofail  0  2

Do not change ro to rw merely because a user expected to save a file there. Confirm whether the design or the ticket is wrong.

Step 3: Check kernel and filesystem messages

When Linux detects filesystem or storage trouble, it may remount a filesystem read-only to limit further damage. That behavior is inconvenient, but it can keep an ugly problem from becoming an unrecoverable one.

Check kernel messages from the current boot:

journalctl -k -b --no-pager | tail -100

If journalctl is unavailable, try:

dmesg --level=err,warn | tail -100

Depending on the filesystem and device, warning signs may include:

I/O error
Buffer I/O error
EXT4-fs error
XFS metadata corruption detected
Remounting filesystem read-only
blk_update_request
rejecting I/O to offline device

The exact wording varies. Capture the full timestamped messages rather than searching for one phrase and ignoring the surrounding events.

For more log-reading basics, see How to Check Linux Logs: A Beginner Help Desk Guide.

Step 4: Check the device and storage layer

A filesystem error may be the symptom, not the root cause. Before scheduling repair, identify what sits underneath the mount.

lsblk -o NAME,SIZE,FSTYPE,FSVER,MOUNTPOINTS,RO,TYPE

The RO column can reveal whether the block device itself is read-only. A value of 1 deserves attention.

For a local SATA or NVMe device, your organization may use SMART or NVMe health tools. Typical commands are:

sudo smartctl -a /dev/sdb
sudo nvme smart-log /dev/nvme0

Do not guess the device name. Map the failed path to its source with findmnt, then map partitions to parent disks with lsblk.

In cloud environments, also check the provider console and volume events. A virtual disk can look present inside Linux while the underlying platform reports attachment, throttling, snapshot, or hardware issues.

For NFS, SMB, iSCSI, SAN, or clustered storage, involve the appropriate storage owner. A client-side remount cannot fix a server export that is read-only or a backend volume that has failed.

Step 5: Protect data before repair

If the filesystem still allows reads, preserve important changing data before experimenting with repairs. Follow the organization’s backup and incident procedure rather than improvising a copy destination on the same failing disk.

Useful evidence to capture includes:

findmnt -T /affected/path
lsblk -f
journalctl -k -b --no-pager | tail -200
df -hT /affected/path

Record:

  • The failed path and exact command
  • First observed time
  • Mount source, target, type, and options
  • Relevant kernel messages
  • Recent reboot, storage, patching, or power events
  • Whether reads still work
  • Whether the application is still serving traffic
  • Backup or snapshot status

If this is a production root filesystem, a database volume, or a disk with no verified backup, escalation is usually smarter than creative command-line heroics.

Should you remount it read-write?

Linux supports remounting a filesystem, and you will often see advice like:

sudo mount -o remount,rw /mountpoint

That command is appropriate only when all of these are true:

  • The filesystem was read-only for an understood, non-corruption reason
  • The underlying device is healthy and writable
  • The platform permits the change
  • The mount is supposed to be read-write
  • Your change procedure allows it

It is not a safe first response when the kernel remounted a filesystem after errors. A successful remount does not prove the disk or filesystem is healthy. It may simply allow more writes into a damaged situation.

If you are correcting an intentional mount option in a lab, use the exact mountpoint reported by findmnt, verify with findmnt again, and make any persistent /etc/fstab change separately and carefully.

When and how fsck fits

fsck checks and repairs supported filesystems, but it is not a universal “fix disk” button.

Never run a repair-mode filesystem check on a mounted read-write filesystem. For many filesystems, the safe process requires downtime, unmounting the volume, booting into rescue mode, or attaching the disk to another system.

First identify the filesystem type:

findmnt -no FSTYPE,SOURCE,TARGET -T /affected/path

Then use the filesystem’s documented tool and procedure:

  • ext2/ext3/ext4 commonly use fsck.ext4 or e2fsck
  • XFS commonly uses xfs_repair, not a normal repair pass from generic fsck
  • Btrfs has its own inspection and recovery tools, with important cautions
  • Network filesystems must be repaired at the server or storage layer

A read-only check may be possible for some tools, but even diagnostics can stress failing hardware. If the disk reports I/O errors, consider imaging or replacing it before repair attempts.

For a managed server, the right ticket update is often: “Filesystem remounted read-only after errors; data protected; downtime required for offline check,” not “ran fsck until the warning disappeared.”

Containers, WSL, and cloud images

The same error can mean something different outside a normal Linux server.

Containers

Container image layers are commonly read-only. Write application data to a declared writable volume or temporary directory. If a Kubernetes pod mounts a ConfigMap or Secret, that mount is expected to be read-only; update the source object instead of editing files inside the pod.

Check mounts from inside the container:

findmnt

Then inspect the container or pod definition from the management plane.

WSL

In WSL, Windows drives and virtual Linux disks have different storage paths. If a Windows volume is offline, protected, or unhealthy, Linux commands inside WSL may not be able to make it writable. Check both the Linux mount and Windows storage state before changing WSL configuration.

Cloud and immutable systems

Some cloud images and appliance-like distributions intentionally protect the root filesystem. Configuration may belong in cloud-init, an image pipeline, a management API, or a separate data volume. If the instance is designed to be replaced, fix the source image or deployment definition instead of hand-editing the running copy.

Common beginner mistakes

Using sudo repeatedly

Root privileges do not override a read-only mount. Repeating the command with sudo adds noise without adding evidence.

Treating it as chmod trouble

Permissions and mount state are different layers. Check findmnt before changing modes, owners, ACLs, or security policy.

Forcing a read-write remount after I/O errors

The kernel may have gone read-only to protect the filesystem. Check logs and device health first.

Running fsck on the mounted root filesystem

Repair normally requires the filesystem to be unmounted or the machine to be in a supported rescue environment. Schedule downtime and follow the filesystem-specific procedure.

Editing /etc/fstab before checking the live mount

Persistent configuration may not explain a runtime remount caused by errors. Capture current mount options and kernel logs before changing boot configuration.

Assuming the disk itself is healthy

A filesystem repair cannot fix failing storage hardware. Check the device, controller, cable, virtual volume, or network-storage backend.

Rebooting without a plan

A reboot may trigger an automatic check, fail to mount the filesystem, or leave the server in emergency mode. Confirm console access, backups, recovery steps, and the expected boot behavior first.

A safe ticket workflow

  1. Reproduce or capture the exact write error and path.
  2. Use findmnt -T to identify the owning filesystem.
  3. Record source, target, type, and mount options.
  4. Check journalctl -k -b for filesystem and device warnings.
  5. Use lsblk -f and platform tools to identify the storage layer.
  6. Confirm whether read-only behavior is intentional.
  7. Protect data and verify backups or snapshots.
  8. Escalate hardware, storage, or production-root failures.
  9. Schedule an offline filesystem check when the evidence supports it.
  10. Remount or edit persistent configuration only after the cause is understood.
  11. Verify application health and writable paths after recovery.
  12. Document the cause, repair, validation, and follow-up monitoring.

This sequence is slower than pasting the first remount command from a forum. It is also faster than explaining why the database volume became worse after the “fix.”

Practice Linux storage troubleshooting

Read-only filesystem tickets combine navigation, mounts, logs, block devices, and change control. Those skills get easier when you practice the diagnostic commands before a production disk starts complaining.

Practice Linux commands in Shell Samurai. It gives you a safe place to build command-line confidence without using a customer server as the lab.

FAQ

Why did Linux suddenly make my filesystem read-only?

Linux may remount a filesystem read-only after detecting filesystem corruption, I/O errors, or storage trouble. It can also be intentionally mounted read-only through /etc/fstab, container configuration, a snapshot, an appliance design, or a network-storage export.

Can I fix a read-only filesystem with chmod or sudo?

Usually no. chmod, chown, and sudo address permissions and privileges. A read-only mount rejects writes at the filesystem layer, including metadata changes.

Is it safe to run mount -o remount,rw?

Only when you understand why the filesystem is read-only, know the storage is healthy, and confirm the mount is intended to accept writes. Do not force it after filesystem or I/O errors without protecting data and following a repair plan.

Can I run fsck while the filesystem is mounted?

Do not run a repair-mode check against a mounted read-write filesystem. Many repairs require the volume to be unmounted, the machine to be in rescue mode, or the disk to be attached elsewhere. Follow the procedure for the actual filesystem type.

How do I find which filesystem contains a path?

Run findmnt -T /path/to/file. It reports the owning mount, source device, filesystem type, and options. Use findmnt -no SOURCE,TARGET,FSTYPE,OPTIONS -T /path/to/file for concise output.

What should I include in the ticket?

Include the exact error and path, mount source and target, filesystem type and options, relevant kernel messages, device or platform health, backup status, application impact, and the recovery steps that were approved and verified.

Practice This in a Real Terminal

Shell Samurai gives you safe Linux missions so the commands actually stick. Chapter 1 is free; the full practice path is a one-time purchase, not another subscription.