Software

•
Several compilers and libraries have been updated on hexagon. For maximum performance and stability we recommend that people recompile their programs. Note that for the Pathscale compiler we are waiting for a fixed license file, we will update this note when it is back online.
Update:Pathscale is now working again.

MPI:
xt-mpt: 3.4.1 -> 3.4.2

Libs/math:
xt-libsci 10.3.8 -> 10.3.9
petsc 3.0.0.5 -> 3.0.0.6
libfast 1.0.5 -> 1.0.6
fftw 3.2.1 -> 3.2.2

Compiler:
xt-gcc 4.3.3 -> 4.4.1
pathscale 3.2 -> 3.2.99
pgi 9.0.2 -> 9.0.3
xt-asyncpe (wrapper) 3.2 -> 3.3

Debugger tools:
NEW: xt-atp 1.0
NEW: mrnet 2.0.1.1
xt-lgdb (gnu debugger) 1.1 -> 1.2


NOTES:

New Module command functionality:
New Feature to "module help". The "module help product/version" command will now provide information about that specific product version (release-notes).

gcc
See http://gcc.gnu.org/gcc-4.4 for more information.

Known problem: GCC 4.4.x seg faults when instrumented with CrayPat

xt-atp
ATP 1.0 (Abnormal Termination Processing):
Purpose:
--------
Abnormal Termination Processing (ATP) is a system that
monitors Cray XT System user applications, and should an
application take a system trap, ATP performs analysis on
the dying application.

Limitations:
- When using ATP, an applications crash will not produce a
core dump.
- When using ATP, the application cannot be checkpointed.
- ATP does not support threaded application processes.
- ATP has been tested at 1024 cores. At scales above this there
are performance issues that need to be researched.

Documentation:
--------------
See the intro_atp man page and "module help xt-atp".


mrnet
Initial release of MRNet - The Multicase/Reduction Network.

Documentation:
--------------
http://www.paradyn.org/mrnet/release-2.1/UsersGuide.html


xt-libsci
10.3.9 supports gcc 4.4.x

petsc
Support for the GCC 4.4 compiler.
Bug fixed:
752765 Malfunction of CASK SpMV

Documentation:
--------------
http://www.mcs.anl.gov/petsc/petsc-as/documentation/index.html


xt-mpt
Bugs Fixed:
752156 PTL_PT_VAL_FAILED when doing large transfer mpi_allgather
748473 MPI_Gather slower

libfast
Provide optimized functions for round() and roundf() in C,
and ANINT() and DNINT() in Fortran. These functions round
a floating point number to the nearest whole number.


fftw
Improve performance of some copy operations of complex arrays on x86
machines. Add configure flag to disable alloca(), which is broken in mingw64.
Planning in FFTW_ESTIMATE mode for r2r transforms became slower between fftw-3.1.3 and 3.2. This regression has now been fixed.

•
Due to a needed security update that requires a reboot we will be forced to do the next maintenance of hexagon earlier than planned. We will therefore have a scheduled maintenance starting on Thursday Sep. 10th at 13:00.

Job-scheduler reservation is now in place so that only jobs that can finish (according to requested walltime) before the scheduled maintenance will be allowed to start.

During the maintenance we will install a security update as well as replacing a few faulty hardware components.

We will update this note when we have more information about expected length or ongoing progress for the maintenance.

As usual, send any questions to support-uib@notur.no.

Update 16:30: Machine is now up again and ready for use.

•
Several libraries and have been updated on hexagon.
For maximum performance and stability users are encouraged to log out and in again and then recompile their programs and libraries.

Updates:
xt-asyncpe 3.2
xt-libsci 10.3.8
petsc 3.0.0.5
libfast_mv 1.0.5
mpt 3.4.1
java-jdk 1.6.0_15
PGI compiler 9.0.2
Intel compiler 11.1.046

Removed
xt-libsci 10.3.5
petsc 3.0.0.3
libfast_mv 1.0.3
mpt 3.1.2

•
Several libraries have been updated on hexagon.
For maximum performance and stability users are encouraged to log out and in again and then recompile their programs and libraries.

Updates:

MPI:
xt-mpt 3.3.0 -> 3.4.0

Libs/math:
xt-libsci 10.3.6 -> 10.3.7
hdf5 1.8.2.3 -> 1.8.3.0
netcdf_hdf5parallell 4.0.0.3 -> 4.0.1.0
netcdf 4.0.0.3 -> 4.0.1.0
petsc 3.0.0.3 -> 3.0.0.4
acml 4.2.0 -> 4.3.0

Compiler:
xt-asyncpe (wrapper) 3.0 -> 3.1
pgi 8.0.6 -> 9.0.1

NOTES:

xt-mpt
Bug fixes related to SHMEM as well as Intel compiler.

hdf5/netcdf
New features and bugfixes.

See here for more information:
http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.3-RELEASE.txt

http://www.unidata.ucar.edu/software/netcdf/release-notes-4.0.1.html

petsc
Feature update: CASK 1.1, plus bugfixes.

CASK 1.1 includes new sparse matrix vector multiplication for
transposed matrices improves the performance by 5-30% depending
on nonzero pattern of the matrix.

Support for the Intel compiler.

ACML
New feature release. Note that this release does not support the pathscale compiler.

See ACML site for more information:
http://developer.amd.com/cpu/Libraries/acml/features/pages/default.aspx

PGI
New major release.

See PGI site for more information:
http://www.pgroup.com/doc/pgiwsrn901.pdf

•
Several libraries have been updated on hexagon. For maximum performance and stability users are encouraged to log out and in again and then recompile their programs and libraries.

Updated:
xt-asyncpe 3.0
xt-libsci 10.3.6
MPT 3.3.0
libfast_mv 1.0.4
PETSc 3.0.0.3

* xt-asyncpe 3.0
Bug fix and support in the compiler driver scripts for the Intel
compiler. No catamount support, CLE CNL only.
* xt-libsci 10.3.6
Bug fixes.
* MPT 3.3.0
Bug Fixes and improved MPI-IO by default.
* libfast_mv 1.0.4
Bug fixes and minor features.
* PETSc 3.0.0.3
Bug fixes and Istanbul support.

•
26 May, 08:30 is planned time for Hexagon software upgrade. It will be upgraded from UNICOS/lc 2.0 to CLE 2.1UP02. It is an major software upgrade and will take from several hours to several days. We will use our bests to minimize the downtime.
Lustre FS is going to be upgraded from version 1.4 to 1.6 which will need /work filesystem check for several hours. We kindly ask hexagon users to remove all unused files from /work filesystem, this will result in shortening downtime.
ALL programs which are going to be used after software upgrade MUST be recompiled! This is very important! Running application compiled for current OS release (2.0) can create unexpected results after upgrade.

Update: Upgrade time moved to 26.05.2009 08:30. Therefore hexagon reserved from 08:30, 26th of May. Long jobs which are not able to complete before the downtime will not start.

Update: May 27th 00:00: The upgrade will continue tomorrow. The machine will be unavailable until the upgrade is finished.

Update: May 27th 16:00: We have started recompiling software on Hexagon.

Update: May 27th 21:30 Software upgrade finished. Hexagon is back online.

As it was mentioned before this is MAJOR software upgrade. Now hexagon is running CLE 2.1UP02, with Lustre 1.6 filesystem

Notes:

* All programs MUST be RECOMPILED!.

* The following programs/modules was removed as they are not supported anymore:
gmalloc
gnet
iobuf
libscifft
openGLUT

* This software was replaced by Cray versions:
all hdf5:
hdf5
hdf5-parallel
all netcdf:
netCDF (for version 3.6.2)
netcdf (for version 4.0.1)
java/jdk 1.6.0

* The following software will be shortly recompiled.
amber
antlr
berkley-upc
cdo
coreutils
git
gnuplot
grads
grib_api
gsl
ncl_ncarg
nco
ncview
nedit
nwchem
imagemagick
jasper
libdap
libnc-dap
matlab
pgplot
python (static)
subversion
vim 7.2 or newer
WPS
WRF

* The libraries like:
zlib
libxml2
libpng
glib2.4.2
are available by default without modules

* Module name: changed program name and version number structure, like:
%ProgramName/%Version
eg. nwchem-cnl/5.1.1
netCDF/3.6.2
While loading modules, users are advised to use as much as possible only the program name, the optimal version will be loaded by default:
module load nwchem

* Please update your PBS scripts as well as environment to load correct modules.

•
Several libraries have been updated on hexagon. For maximum performance and stability users are encouraged to log out and in again and then recompile their programs and libraries.

MPI
xt-mpt 3.1.2 -> 3.2.0

Libs/math:
xt-libsci 10.3.3 -> 10.3.4
hdf5 1.8.2.1 -> 1.8.2.2
netcdf_hdf5parallell 4.0.0.1 -> 4.0.0.2
netcdf 4.0.0.1 -> 4.0.0.2

Compiler:
xt-asyncpe (wrapper) 2.3 -> 2.4
pgi 8.0.4 -> 8.0.5

NOTES:

xt-mpt

MPI-IO performance improvements for collective buffering on MPI collective
writes.

This optimization is enabled by setting the MPIIO hint romio_cb_write to "enable" and setting the environment variable MPICH_MPIIO_CB_ALIGN to 2. Other values of this environment variable are 0 and 1, where 0 is for the original algorithm in MPT 3.0 and earlier and 1 is for the algorithm introduced in MPT 3.1. The MPICH_MPIIO_CB_ALIGN section of the "mpi" man page gives more details. If you are not already using collective buffering, read the MPICH_MPIIO_HINTS section for more information.

xt-libsci

SuperLU has been removed from xt-libsci release. It is released with the PETSc 3.0.0.1 and later releases. The xt-libsci module no longer loads the fftw module by default.

fftw

Performance improvements for some multidimensional r2c/c2r transforms.
Fortran documentation now recommends not using dfftw_execute, because of reports of problems with various Fortran compilers; it is better to use dfftw_execute_dft etc.

•
Several libraries have been updated on hexagon.

MPI:
xt-mpt 3.1.1 -> 3.1.2

Libs/math:
xt-libsci 10.3.2 -> 10.3.3
petsc 3.0.0 -> 3.0.0.1
hdf5 1.8.2 -> 1.8.2.1
netcdf_hdf5parallell 4.0 -> 4.0.0.1
netcdf 4.0 -> 4.0.0.1

Compiler:
xt-asyncpe 2.1 -> 2.3 (wrapper)
pgi 8.0.3 -> 8.0.4

NOTES:

xt-mpt:

MPI_Reduce has been optimized to be SMP aware and this optimization is
enabled by default. The SMP aware algorithm performs significantly better
than the default algorithm for most message sizes. Performance improvements of over 3x for message sizes below 128K have been observed. A new environment variable MPICH_REDUCE_LARGE_MSG can be used to adjust the cutoff for when this optimization is enabled. See the man page for more info.

xt-libsci:

- libGoto 1.29 includes moderate performance improvements in BLAS and LAPACK.

- CRAFFT1.1 (Cray Adaptive FFT) is a productivity enhancement for the efficient use of Fast Fourier transforms with little programming effort. CRAFFT 1.1 adds single precision support. See intro_crafft for a description of the double precision API. Replace the "z" and "d" in the double precision
routine names by "c" or "s" to use the single precision routines.
E.g.crafft_d2z1d in double precision would be crafft_s2c1d in single
precision.
The fftw/3.2.0 module must be loaded to use CRAFFT1.1. If the FFTW module is not loaded, then the user's link stage will fail with unresolved references to FFTW routines.
Prior to running a CRAFFT-linked executable, users must copy the correct
FFTW wisdom files into their current run directory. The wisdom files are
fftw_wisdom-3.2 for double precision and fftw_wisdom_single-3.2 for single
precision, and are found at the following location: /opt/xt-libsci/10.3.3/

netcdf_hdf5parallell:

Known problem:
Use of the '-fsecond-underscore' compiler option with pathscale compilers is needed. Failure to do so will result in a link error.