The short version
A production gateway showed a bad sector on /dev/sda and very high power-on hours. I planned the drive replacement before it failed, and kept an unrelated kernel fault separate so one fix would not be blamed for the other.
Overview
A production Linux gateway was running on an old disk. Routine checks of the drive and the kernel log showed two separate problems.
Problem statement
Would the disk fail, and how soon? A gateway that loses its disk takes every connected user offline.
Investigation
sudo smartctl -a /dev/sda
sudo dmesg | grep -i dmar- SMART showed a bad sector and a very high runtime. The attributes to watch are
Reallocated_Sector_Ct,Current_Pending_SectorandPower_On_Hours. - The kernel log also showed a DMAR interrupt remapping fault. I treated that as a separate problem.
Root cause
The drive was worn: a bad sector on a disk with very high power-on hours. The DMAR message was a different issue and I tracked it independently.
Resolution
Forcing the drive to reallocate the bad sector (for example with hdparm) buys time but does not make a failing drive healthy. The real fix is a replacement, scheduled while the system still works.
Verification
This case study is documented up to the plan. After any reallocation or replacement I would re-read SMART and run a long self-test to confirm the result.
Lessons learned
- Treat SMART warnings as a deadline, not as noise.
- Do not let two unrelated symptoms share one explanation.
- A replacement plan written before the failure is cheaper than an emergency restore after it.
Skills demonstrated
Confidentiality: all identifying information, credentials, customer data, internal addresses and proprietary configurations have been removed or replaced with illustrative examples.