Our filesystem went read-only for 19 days. Nobody saw the warning.

One of our servers fired a kernel filesystem-error alert. Nine days later, the OS remounted the filesystem read-only. Ten days after the first alert, a service on the same server entered a failed state. All three alerts sat unacknowledged for between 11 and 19 days. No alert channel had been configured for that server, so every notification went nowhere.

What the alerts actually said

The first alert read: "Kernel reported filesystem errors -- schedule an fsck and check the underlying disk; back up first." It opened on 2026-09-10 and was notified once. It stayed open for about 11 days.

8.6 days after the first alert, a second one opened: "A filesystem was remounted read-only after errors -- a read-only remount means the disk or filesystem hit errors. Check SMART health, run fsck on the affected mount, and expect a reboot." That alert ran for about 19 days before it closed. Also notified once, never acknowledged.

10.8 days after the first alert, a third opened: "Service apport-autoreport is down." That one ran for about 17 days. Same story: one notification, no acknowledgement.

We are not claiming these three alerts are causally linked. What we can say is that they appeared on the same server in that order, and none of them reached a human.

Why a read-only remount is a serious signal

When the Linux kernel detects filesystem errors it cannot recover from, it remounts the affected filesystem read-only as a last resort to stop further corruption. At that point, any process trying to write -- a database, a web server, a log daemon -- either fails silently or crashes. Applications may appear to be running while actually writing nothing.

The kernel logs the event in the system journal. It also shows up in `dmesg`. The problem is that neither of those surfaces alerts you unless something is watching.

A read-only remount does not mean your data is safe. It means the kernel stopped accepting writes before things got worse. The underlying disk problem is still there.

How to check your own servers right now

Run these commands on any Linux server you care about. The first block checks for a current read-only mount; the second checks the kernel ring buffer for recent filesystem errors; the third runs a SMART health check on the underlying block device.

# 1. Find any filesystem currently mounted read-only
cat /proc/mounts | awk '$4 ~ /(^|,)ro(,|$)/ {print $2, $3, $4}'

# 2. Check the kernel log for filesystem errors in the last 500 lines
dmesg | tail -500 | grep -iE 'error|remount|read-only|EXT4-fs|XFS|I/O error'

# 3. Check SMART health on your primary disk (replace sda with your device)
sudo smartctl -H /dev/sda
# "SMART overall-health self-assessment test result: PASSED" is what you want
# "FAILED" means the drive is reporting imminent failure

# 4. Look for recent journal-level filesystem errors
journalctl -p err -b | grep -iE 'filesystem|disk|I/O|remount|fsck'

# 5. Check for any failed mount units
systemctl --failed --type=mount

If `dmesg` shows lines like `EXT4-fs error (device sda1)` or `XFS: corruption of in-memory data detected`, stop writes to that filesystem if you can, take a backup immediately, and schedule an fsck on the unmounted device. Do not run fsck on a mounted filesystem.

What a SMART check actually tells you

SMART (Self-Monitoring, Analysis and Reporting Technology) is built into every modern hard drive and SSD. The tool `smartctl` reads it. Install it with `sudo apt install smartmontools` or `sudo yum install smartmontools`.

The attributes that matter most for predicting failure are: `Reallocated_Sector_Ct` (non-zero means the drive has already remapped bad sectors), `Current_Pending_Sector` (sectors the drive has flagged as unstable), and `Offline_Uncorrectable` (sectors the drive could not correct during offline scanning). Any non-zero value in those three columns deserves attention.

# Full attribute list; look for non-zero RAW_VALUE in columns 5, 197, 198
sudo smartctl -A /dev/sda | grep -E 'Reallocated|Pending|Uncorrectable'

# Example output you do NOT want to see:
#   5  Reallocated_Sector_Ct   0x0033  100  100  036  Pre-fail  Always  -  17
# 197  Current_Pending_Sector  0x0012  100  100  000  Old_age   Always  -  4
# 198  Offline_Uncorrectable   0x0010  100  100  000  Old_age   Offline -  1

The real mistake: no alert channel on that server

All three alerts were generated and each was notified once. The first alert's detail even said "back up first". But we had not configured an alert channel for that server, so each notification had nowhere to go. Zero channels were configured, and no person acknowledged any of the three alerts.

We have written about this failure mode before in "Our alerts went nowhere for 59 days. Here is what we changed." This is a different server, a different set of alerts, and the same root cause: alert channel not configured.

The lesson is not about disk health specifically. It is that any server without a configured alert destination is a server you are monitoring in name only. Go through your fleet and verify that every server has at least one channel -- email is enough -- and that a test alert actually arrives.

Monitoring without a delivery channel is a log you never read. The alert count does not matter; the delivery count does.

How Opservo surfaces this

Opservo watches for kernel filesystem errors and read-only remounts as health-category alerts, separate from lower-priority recommendations. When a server has no alert channel configured, the dashboard flags it explicitly so you can see the gap before an incident reveals it. The health score docks points for both the filesystem condition and the missing notification path, so a server in this state cannot quietly score well.

Try Opservo free on 2 servers — one command installs a read-only agent and your first findings arrive in five minutes: start free.