Logo

HPC @ Uni.lu

High Performance Computing in Luxembourg

It boils down to ensuring that you have a consistent environment with the family of CC variables well defined.

fgeorgatos@gaia-1:~: $ module purge
fgeorgatos@gaia-1:~: $ module load PRACE/20130605-goolf-1.4.10
fgeorgatos@gaia-1:~: $ module list
Currently Loaded Modulefiles:
  1) GCC/4.7.2
  2) hwloc/1.6.2-GCC-4.7.2
  3) OpenMPI/1.6.4-GCC-4.7.2
  4) gompi/1.4.10
  5) OpenBLAS/0.2.6-gompi-1.4.10-LAPACK-3.4.2
  6) FFTW/3.3.3-gompi-1.4.10
  7) ScaLAPACK/2.0.2-gompi-1.4.10-OpenBLAS-0.2.6-LAPACK-3.4.2
  8) goolf/1.4.10
  9) make/3.82-goolf-1.4.10
 10) Java/1.7.0_10
 11) Bash/4.2-goolf-1.4.10
 12) ncurses/5.9-goolf-1.4.10
 13) tcsh/6.18.01-goolf-1.4.10
 14) Tcl/8.5.12-goolf-1.4.10
 15) Tk/8.5.12-goolf-1.4.10
 16) zlib/1.2.7-goolf-1.4.10
 17) Szip/2.1-goolf-1.4.10
 18) HDF5/1.8.7-goolf-1.4.10
 19) netCDF/4.1.3-goolf-1.4.10
 20) Perl/5.16.3-goolf-1.4.10
 21) PRACE/20130605-goolf-1.4.10
fgeorgatos@gaia-1:~: $ module load $EBDEVELPRACE
fgeorgatos@gaia-1:~: $ module list
Currently Loaded Modulefiles:
  1) EasyBuild/1.6.0
  2) GCC/4.7.2
  3) hwloc/1.6.2-GCC-4.7.2
  4) OpenMPI/1.6.4-GCC-4.7.2
  5) gompi/1.4.10
  6) OpenBLAS/0.2.6-gompi-1.4.10-LAPACK-3.4.2
  7) FFTW/3.3.3-gompi-1.4.10
  8) ScaLAPACK/2.0.2-gompi-1.4.10-OpenBLAS-0.2.6-LAPACK-3.4.2
  9) goolf/1.4.10
 10) /opt/apps/HPCBIOS.20130715/software/zlib/1.2.7-goolf-1.4.10/easybuild/zlib-1.2.7-goolf-1.4.10-easybuild-devel
 11) /opt/apps/HPCBIOS.20130715/software/PRACE/20130605-goolf-1.4.10/easybuild/PRACE-20130605-goolf-1.4.10-easybuild-devel
fgeorgatos@gaia-1:~: $ env|grep CC=
EBDEVELGCC=/opt/apps/HPCBIOS.20130715/software/GCC/4.7.2/easybuild/GCC-4.7.2-easybuild-devel
MPICC=mpicc
EBROOTGCC=/opt/apps/HPCBIOS.20130715/software/GCC/4.7.2
EBVERSIONGCC=4.7.2
OMPI_CC=gcc
EBVARCC=gcc
EBVAROMPI_CC=gcc
EBVARMPICC=mpicc
CC=gcc
fgeorgatos@gaia-1:~: $ which $MPICC
/opt/apps/HPCBIOS.20130715/software/OpenMPI/1.6.4-GCC-4.7.2/bin/mpicc
fgeorgatos@gaia-1:~: $ cd ~/arena/hello
fgeorgatos@gaia-1:~/arena/hello: $ $MPICC hello.c -o hello_mpi
fgeorgatos@gaia-1:~/arena/hello: $ ./hello_mpi
--------------------------------------------------------------------------
The OpenFabrics (openib) BTL failed to initialize while trying to
allocate some locked memory.  This typically can indicate that the
memlock limits are set too low.  For most HPC installations, the
memlock limits should be set to "unlimited".  The failure occured
here:

  Local host:    gaia-1
  OMPI source:   btl_openib_component.c:1200
  Function:      ompi_free_list_init_ex_new()
  Device:        mlx4_0
  Memlock limit: 65536

You may need to consult with your system administrator to get this
problem fixed.  This FAQ entry on the Open MPI web site may also be
helpful:

    http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   gaia-1
  Local device: mlx4_0
--------------------------------------------------------------------------
0: We have 1 processors
fgeorgatos@gaia-1:~/arena/hello: $

Despite the complaint about the OpenFabrics error above, the build has gone well and the executable actually reported 1 processor.

Hints:

  • Always try to be very conscious of the modules & environment variables that you use.
  • Do not redefine variables that have already been set by modules, otherwise you won’t be able to unload/purge properly
  • If the above is unavoidable, spawn a subshell (type bash), which you can exit to cleanup your experimentation
  • The last tip of spawning a subshell is a good habit in general anyway, since it protects you from both bugs & errors.