Both operating system disks failed in a short timeframe in Grunch making the system unoperational. We are trying to recover from the failure ASAP.
Update 14:00_06.10.2017: grunch server is up again. both os disks are replaced and grunch server are reinstalled.
Author Archives: Lóránd Szentannai
Hexagon: planned power maintenance – September 25th
There is a planned maintenance on the electric power line in the server room for the 25th of September. Therefore Hexagon, related file systems and storage enclosures has to be taken offline.
The maintenance is scheduled to start at 20:00. According to plan, Hexagon should be back by the end of the day.
Running jobs will be stopped. All scheduled jobs in the queue will be started automatically when the system is operational again.
Update 20.09.2017: Please note the time change. The maintenance window has been moved from Saturday, 23rd of September to Monday, 25th of September 20:00.
Update 26.09.2017: UPS maintenance is over yesterday night, and we have a problem to take hexagon online due to some filesystem storage issues. We are working on it and we apologize for inconveniences.
Update 26.09.2017: Hexagon is up and available again since 09:08 AM.
Update 14:00_26.09.2017: Hexagon work file system crashed unexpectedly, we are working on it. sorry for inconveniences.
Update 14:40_26.09.2017 Hexagon has to be taken down due to hardware issues related to work filesystem. We try to resolve problems as soon as possible.
Update 16:40_26.09.2017 Hexagon is back online again. problem with work filesystem is resolved.
The maintenance is scheduled to start at 20:00. According to plan, Hexagon should be back by the end of the day.
Running jobs will be stopped. All scheduled jobs in the queue will be started automatically when the system is operational again.
Update 20.09.2017: Please note the time change. The maintenance window has been moved from Saturday, 23rd of September to Monday, 25th of September 20:00.
Update 26.09.2017: UPS maintenance is over yesterday night, and we have a problem to take hexagon online due to some filesystem storage issues. We are working on it and we apologize for inconveniences.
Update 26.09.2017: Hexagon is up and available again since 09:08 AM.
Update 14:00_26.09.2017: Hexagon work file system crashed unexpectedly, we are working on it. sorry for inconveniences.
Update 14:40_26.09.2017 Hexagon has to be taken down due to hardware issues related to work filesystem. We try to resolve problems as soon as possible.
Update 16:40_26.09.2017 Hexagon is back online again. problem with work filesystem is resolved.
Hexagon: IMR volumes offline
The network equipment connecting Hexagon and IMR has to be changed and needs a maximum two hours downtime.
Therefore IMR volumes will be unmounted on Tuesday, 29th of August from 09:00 AM for approximately two hours. By that time, please stop all your processes on Hexagon which are using the IMR volumes.
Therefore IMR volumes will be unmounted on Tuesday, 29th of August from 09:00 AM for approximately two hours. By that time, please stop all your processes on Hexagon which are using the IMR volumes.
Hexagon & Grunch: Planned downtime for 25th of August
On Friday, 25th of August maintenance on electric lines in the server room will be carried out. Therefore Hexagon must be switched off. All related file systems (/work, /work-common) will be also off.
The maintenance will start at 07:00 and according to the plan should last until 13:00 o'clock.
During this time work-common will not be available on Grunch .
Update:
The maintenance will start at 07:00 and according to the plan should last until 13:00 o'clock.
During this time work-common will not be available on Grunch .
Update:
- 25.08.2017 07:00: Maintenance has started.
- 25.08.2017 12:50: Storage controller issues are delaying startup of the machine. We are working on the fix.
- 25.08.2017 15:05: Storage controller issues were remediated. Some disks are rebuilding for /work-common filesystem, thus performance impact might be expected for a couple of days.
- 25.08.2017 15:20: Hexagon is up again.
Hexagon: /migrate read-only
Since /migrate area is going to be decommissioned, it was remounted on Hexagon in read-only mode to hinder new data being written on.
Hexagon: power blink
Lightning caused again crashing of the high speed network on Hexagon and several nodes.
Hexagon is up again starting from 11:15.
Hexagon: HSN problems, rebooted
The high speed network stopped working today due to power spikes around 14:30.
The machine had to be rebooted and is up again since 16:40.
The machine had to be rebooted and is up again since 16:40.
Hexagon: emergency reboot
We will reboot Hexagon at 11:40 to apply important security updates.
Please accept our apologies for short notice.
Update 12:50 - Access to the machine is re-opened.
Please accept our apologies for short notice.
Update 12:50 - Access to the machine is re-opened.
Hexagon: emergency reboot
Hexagon had to be rebooted to apply important security updates.
The machine is up and login enabled from 10:30.
Please accept our apologies for short notice.
The machine is up and login enabled from 10:30.
Please accept our apologies for short notice.
Hexagon: HPN-SSH not supported
HPN-SSH has been obsoleted by standard OpenSSH releases.
Due to unstable release cycle and security concerns related to that, we have discontinued the use of HPN-SSH on Hexagon.
Due to unstable release cycle and security concerns related to that, we have discontinued the use of HPN-SSH on Hexagon.