disks on fimm failed

2 out of 3 raid arrays on fimm has failed, so /home/fimm and /work* is gone at the moment. This is a major fault, and can take some time before it's fixed.

Will update this entry when I know more.


Wed Mar 30 04:32 Multiple disks and raid-controllers failed on two separate storage units.

10:15 Started restore of /home/fimm from backup, just in case we're unable to recover the filesystems on disk.

10:35 Got confirmation from Nexsan support.

13:20 Chatted with Nexsan-support. They'll call me back ASAP.

15:43 Called Nexsan up again.. Where's the support??

16:23 Got procedure to reset drives from serial menu. This seems to make the system functional again. Haven't tested accessing the volumes from linux yet.

18:39 Try accessing the volumes from linux, and notice that now the third satablade also has failed. Woun't be able to reset this one before tomorrow morning. Hope Nexsan has some idea by then to what has triggered this problem.

Thu mar 31 11:50 All disks and filesystems are up! Still got no idea on why this error occured, so we might have to take the filesystem down again if Nexsan engineering maybe has some firmware upgrades that fixes the problem.

Total downtime: 31 hours, 30 minutes