Lustre filesystem (/work) hang on hexagon, Dec. 27th

The /work file system on hexagon hangs, we are doing debug dumps and will restart the system. Existing jobs will have to be re-submitted.

Update 15:00, one of the disk controllers have problems, the downtime will be longer than anticipated. We will update this note when we have more information.

Update 20:00, we will need to wait for support on Monday before continuing the work to fix the controller.

Update Monday Dec 29th, 12:00, we are currently waiting for a new controller.

Update Monday Dec 29th, 17:15, the shipment with the controller is expected to arrive on Wednesday 31st. We are sorry for this delay.

Update Wednesday Dec 31st, 14:50, we have got a notice that the expected delivery of the replacement controller is delayed even further, to Monday Jan. 5th. We are looking to other ways to get the file system working.

Update Thursday Jan 1st, 04:00, the system is running again with a workaround. We will have to reboot the system again when the replacement controller arrives (so long-running jobs will have to be resubmitted).

Update Monday Jan 5th, 13:50, the new controller has now arrived we have scheduled this to be replaced on Monday the 12th at 13:30.