FIMM was down for scheduled maintanance 2005/05/09 08:00-10:00 = 2 hours of the full cluster.
The work that was done was:
o upgraded firmware on SATABlades
o move /local from the local disk of each node, to a shared disk, to save precious space for local /scratch usage.
NOTUR 2005 conference, http://www.notur.no/notur2005
The 5th anual gathering on High Performance Computing in Norway will be held in Trondheim, May 30-31, 2005. Please see http://www.notur.no/notur2005 for details.
Scheduled downtime on fimm
Fimm will be down monday May 9th. 08:00-12:00 for firmware upgrades on the SATABlade disk solution, and possibly other minor changes. This is to fix the bug that triggered the disk crashes on March 30th.
http://www.parallaw.uib.no/syslog/56
http://www.parallaw.uib.no/syslog/56
Intel compilers upgraded on fimm
The intel fortran and C/C++ compilers were upgraded from v8.1.023 to v8.1.027. This should fix a couple of internal compiler-errors we've been triggering.
PGI compilers upgraded on fimm
The Portland Group compilers were upgraded from v5.2 to v6.0.
portland compiler on fire + mpich
The portland group compilers v6.0 was installed on fire. Also mpich for the portland compilers was installed under /local/mpich/pgi/.
New NOTUR cpu-hour quota period starts April 11
The new NOTUR cpu-hour quota period starts April 11 instead of
April 4; all existing accounts and projects remain open until then.
Linux cluster ‘fire’ reinstalled
Fire is now back on-line. All nodes has been re-installed with the Rocks linux cluster distribution. Unfortunately the installation took a bit more time than expected because front-end node for some reason refused to install rocks distribution. Everything worked well when we used node 5 as front-end node.
Total downtime: 52 hours, 30 minutes
Total downtime: 52 hours, 30 minutes
disks on fimm failed
2 out of 3 raid arrays on fimm has failed, so /home/fimm and /work* is gone at the moment. This is a major fault, and can take some time before it's fixed.
Will update this entry when I know more.
Wed Mar 30 04:32 Multiple disks and raid-controllers failed on two separate storage units.
10:15 Started restore of /home/fimm from backup, just in case we're unable to recover the filesystems on disk.
10:35 Got confirmation from Nexsan support.
13:20 Chatted with Nexsan-support. They'll call me back ASAP.
15:43 Called Nexsan up again.. Where's the support??
16:23 Got procedure to reset drives from serial menu. This seems to make the system functional again. Haven't tested accessing the volumes from linux yet.
18:39 Try accessing the volumes from linux, and notice that now the third satablade also has failed. Woun't be able to reset this one before tomorrow morning. Hope Nexsan has some idea by then to what has triggered this problem.
Thu mar 31 11:50 All disks and filesystems are up! Still got no idea on why this error occured, so we might have to take the filesystem down again if Nexsan engineering maybe has some firmware upgrades that fixes the problem.
Total downtime: 31 hours, 30 minutes
Will update this entry when I know more.
Wed Mar 30 04:32 Multiple disks and raid-controllers failed on two separate storage units.
10:15 Started restore of /home/fimm from backup, just in case we're unable to recover the filesystems on disk.
10:35 Got confirmation from Nexsan support.
13:20 Chatted with Nexsan-support. They'll call me back ASAP.
15:43 Called Nexsan up again.. Where's the support??
16:23 Got procedure to reset drives from serial menu. This seems to make the system functional again. Haven't tested accessing the volumes from linux yet.
18:39 Try accessing the volumes from linux, and notice that now the third satablade also has failed. Woun't be able to reset this one before tomorrow morning. Hope Nexsan has some idea by then to what has triggered this problem.
Thu mar 31 11:50 All disks and filesystems are up! Still got no idea on why this error occured, so we might have to take the filesystem down again if Nexsan engineering maybe has some firmware upgrades that fixes the problem.
Total downtime: 31 hours, 30 minutes
Linux cluster “FIRE” is being reinstalled
The linux cluster fire.bccs.uib.no will be down tuesday/wednesday March 29./30.. All nodes will be re-installed with the Rocks linux cluster distribution. This is a major system upgrade, so most software will have to be rebuilt to run on fire after the upgrade.