We recently helped a Dublin medical practice recover data from their failed Buffalo Terastation RAID 5 NAS device.
It was being used as a file server between four workstations in their office. One month previously, they had noticed that something might be awry when their NAS started to display error codes its LCD. But as their NAS continued to function, they did not give the error codes much heed. Then last week, their NAS became inaccessible altogether.
Diagnosis
We opened up their Terastation device. Inside we found 4 X 1TB Western Digital WD10EFRX-68PJCN0 NASware “Red” disks. We removed them from their bays and attached them to our recovery system. We discovered they were configured in RAID 5 (using a block size of 256KB) and the array was using the XFS file system. We performed diagnosis on each disk. Disk “0” had a problem with head “3” and disk “3” had extensive bad sectors. Disks “1” and “2” tested fine.
Data recovery solution
We would work with disks “1”,”2” and “3” to rebuild the array. We imaged all of these disks so we would be working with copies and not the originals. We now had the challenge of rebuilding the XFS volume. This took a few hours to manually reconstruct with the aid of a HEX editor. Eventually we had a mountable volume again but still no files were visible. This is quite normal after a reconstruction of an XFS array as there is a high probability that some allocation group header structures will still be “out of place” or some missing inode clusters may still exist. These needed to be rebuilt or corrected using XFS repair commands.
Files found
Once the volume’s files appeared, it was now time to extract them onto a 4TB external USB drive. Most of the files were .DICOM files (x-ray images, all of which were intact) which the client was very pleased to see again.
Beware of “warning fatigue”
Reflecting on this case, the client made two mistakes which eventually led to a data loss situation. Firstly, they had mistaken a file server for a back-up device. They believed that because it was a NAS device, it could not fail on them. This is a common misconception that users have about NAS devices. While they certainly do have a lower probability of suddenly and catastrophically failing, they can still fail, albeit slowly and with more warning than a standalone disk. But, in this case, they did not heed to the warning, namely the “I12” error message which they saw appear on the Buffalo’s LCD”. This is understandable. The average user is getting bombarded with warning messages about everything from their media player software being out-of-date to low-ink warning messages from their printer. When it comes to really important messages from, for example, back-up hardware or software, these can get “downgraded” in a user’s mind to “just another error message”. System administrators or IT support technicians have an important role to play in delineating to users the importance of acting on any messages relating to data backup. Hardware manufacturers also have a role in making their error messages on their devices less cryptic and more comprehensible for your “average” user. For example, Buffalo’s “I12” code to signify it’s RAID is operating in degraded mode means very little to the average user. A more succinct message such as “array failing” might spur a whole lot more users into action and help them avoid the costs of data recovery!