An eight-disk HP RAID 5 server with two hot spares, holding around 2 TB of business data for a hundred staff in a factory. It froze, would not reboot, and showed one failed disk. The hot spare never engaged. So the client selected a spare and rebuilt. The rebuild ran to 100% and declared victory — and the server still would not start. By the time it reached us, three disks were failing, not one.
An eight-disk RAID 5 server with two hot spares, holding roughly 2 TB of file data for a factory of about a hundred staff. It froze, was rebooted by hand, and then refused to come back up. In the RAID setup the client found a failed disk — and, worse, a hot spare that had not done the one job it exists to do. So he picked a spare himself and started a rebuild.
The rebuild ran to 100% and reported success. The server still would not boot. Their IT maintenance company looked at it, could not recover it, and sent it to us — which was the right call, and the point at which the damage stopped getting worse.
All ten disks came in. Diagnosis found that the array was in a far worse state than the console had admitted: since the original failure, two more disks had degraded. Three disks, in total, were failing. A RAID 5 tolerates one. The ‘successful’ rebuild had been reconstructing parity from disks that were themselves dying — writing bad maths across the array and calling it a repair.
Three of the disks were physically damaged badly enough that they could not simply be imaged as they were. Those went under our laminar-flow hood for platter work — the platters transferred into matched donor bodies, aligned, and brought back to a state where they could be read once. This is the most delicate work we do: a platter turning at 7,200 rpm treats one dust particle like gravel on a motorway, and alignment errors are unforgiving. It is also the whole reason the donor shelves exist.
With every disk readable, all ten were imaged individually on the DeepSpar imager, weak areas last, nothing written back. The array was then rebuilt virtually from the images — stripe size, disk order and parity rotation derived from the data, and the corrupted stripes left behind by the failed rebuild recoded from surviving parity. The server hardware was never trusted with any of it.
The file system mounted from the reconstructed array and the factory’s data was extracted, verified and returned. The whole RAID was processed within 48 hours of the disks arriving — because a hundred people were standing still, and that is what the emergency track is for.
The lesson here is not that hot spares are useless — it is that a controller only knows what its disks tell it, and dying disks lie. A rebuild reported as complete can be complete nonsense if the disks feeding it were failing while it ran.
If a server will not boot after a disk failure: stop. Do not rebuild, do not initialise, do not let the controller try again. Pull the disks, keep them in bay order, label them, and send every one — failed ones included. The free diagnostic costs nothing, and if the business is standing still, the emergency track exists for exactly this.