Logo

HPC @ Uni.lu

High Performance Computing in Luxembourg

Introduction

Environment modules are a standard technology across HPC sites to permit developing and using complex software builds, allowing multiple versions of software stacks and combinations thereof.

Here follows an example; notice that it may only be visible on a cluster node: (ie. ask for an interactive node slot, if you haven’t done so already, with -I in your submission command)

### bring in a minimal playground environment
fgeorgatos@gaia-48:~: $ bash # use a trick to make it easy to reset the environment
fgeorgatos@gaia-48:~: $ module load standard   
fgeorgatos@gaia-48:~: $ module av

--------------- /opt/apps/default/modules/standard ----------------
CUDA/5.0.35-1
FFTW/3.3.3-gompi-1.4.10
GCC/4.7.2
Inspector/2013_update6
Java/1.7.0_10
Java/1.7.0_15
Java/1.7.0_21
LAPACK/3.4.2-gompi-1.4.10
MATLAB/2013a
MVAPICH2/1.9b-GCC-4.7.2
OpenBLAS/0.2.6-gompi-1.4.10-LAPACK-3.4.2
OpenMPI/1.6.4-GCC-4.7.2
ScaLAPACK/2.0.2-gompi-1.4.10-OpenBLAS-0.2.6-LAPACK-3.4.2
TotalView/8.11.0-2-linux-x86-64
VTune/2013_update10
gompi/1.4.10
goolf/1.4.10
hwloc/1.6.2-GCC-4.7.2
icc/2013.3.163
ictce/5.3.0
ifort/2013.3.163
imkl/11.0.3.163
impi/4.1.0.030

fgeorgatos@gaia-48:~: $ exit # clean up the environment, go out of the "standard" view

Let’s look at another example, which attempts to list the packages starting with letter i (this is a convenient choice, because Intel Cluster Studio packages fit under that pattern)

fgeorgatos@gaia-1:~: $ module av i

-------------------------- /opt/apps/HPCBIOS/modules/compiler ---------------------------
icc/11.1.073      icc/2013.1.117    ifort/11.1.073    ifort/2013.1.117
icc/11.1.075      icc/2013.2.146    ifort/11.1.075    ifort/2013.2.146
icc/2011.10.319   icc/2013.3.163    ifort/2011.10.319 ifort/2013.3.163
icc/2011.13.367   icc/2013.4.183    ifort/2011.13.367 ifort/2013.4.183
icc/2011.6.233    icc/2013.5.192    ifort/2011.6.233  ifort/2013.5.192

----------------------------- /opt/apps/HPCBIOS/modules/mpi -----------------------------
impi/4.0.0.028       impi/4.0.2.003       impi/4.1.0.030
impi/4.0.0.028-32bit impi/4.1.0.027       impi/4.1.1.036

--------------------------- /opt/apps/HPCBIOS/modules/numlib ----------------------------
imkl/10.2.6.038  imkl/10.3.12.361 imkl/11.0.1.117  imkl/11.0.3.163  imkl/11.0.5.192
imkl/10.3.10.319 imkl/10.3.6.233  imkl/11.0.2.146  imkl/11.0.4.183

-------------------------- /opt/apps/HPCBIOS/modules/toolchain --------------------------
iccifort/2011.13.367 ictce/4.0.6          ictce/5.2.0          ictce/5.5.0
ictce/3.2.2.u3       ictce/4.1.13         ictce/5.3.0
ictce/4.0.10         ictce/5.1.1          ictce/5.4.0

As can be seen above, multiple versions of each piece of software may be provided; Zooming in on the Intel C compiler, icc, there may be several versions of it. This is necessary, because different software has different dependencies and in an environment serving multiple users, such a collection makes much sense.

Let’s try to obtain only the set of icc compilers, the above list expanded may include a few:

fgeorgatos@gaia-1:~: $ module av icc

-------------------------- /opt/apps/HPCBIOS/modules/compiler ---------------------------
icc/11.1.073    icc/2011.10.319 icc/2011.6.233  icc/2013.2.146  icc/2013.4.183
icc/11.1.075    icc/2011.13.367 icc/2013.1.117  icc/2013.3.163  icc/2013.5.192

-------------------------- /opt/apps/HPCBIOS/modules/toolchain --------------------------
iccifort/2011.13.367

By default the last version (as sorted alphabetically) will be loaded when asked:

fgeorgatos@gaia-1:~: $ module purge # cleanup any previously loaded modules
fgeorgatos@gaia-1:~: $ module load icc
fgeorgatos@gaia-1:~: $ module list
Currently Loaded Modulefiles:
  1) icc/2013.5.192
fgeorgatos@gaia-1:~: $ icc --version
icc (ICC) 13.1.3 20130607
Copyright (C) 1985-2013 Intel Corporation.  All rights reserved.

fgeorgatos@gaia-1:~: $

Sometimes, we may need to swap to an alternative one:

fgeorgatos@gaia-1:~: $ module swap icc icc/2013.3.163
fgeorgatos@gaia-1:~: $ module list
Currently Loaded Modulefiles:
  1) icc/2013.3.163
fgeorgatos@gaia-1:~: $ icc --version
icc (ICC) 13.1.1 20130313
Copyright (C) 1985-2013 Intel Corporation.  All rights reserved.

fgeorgatos@gaia-1:~: $

And then, it may be the case that we want the whole bundle of Intel tools together; in such situation, we will call for the whole toolchain, defined as ictce:

fgeorgatos@gaia-1:~: $ module purge # cleanup
fgeorgatos@gaia-1:~: $ module load ictce/5.3.0 # the most well tested toolchain, so far
fgeorgatos@gaia-1:~: $ module list
Currently Loaded Modulefiles:
  1) icc/2013.3.163     3) impi/4.1.0.030     5) ictce/5.3.0
  2) ifort/2013.3.163   4) imkl/11.0.3.163
fgeorgatos@gaia-1:~: $

N.B. The above toolchain includes all of the following:

  • Intel C Compiler
  • Intel Fortran Compiler
  • Intel MPI stack
  • Intel MKL libraries # this one provides for BLAS (Linear Algebra) fast implementations, FFT and many more
  • the wrapper modulefile ictce, ie. Intel compiler toolchain & computing environment

If you have not already done so, this is a good moment to start looking in the Intel Cluster Studio XE2013; it is supposed to deliver the most optimized software stacks for generic usage for HPC, on commodity processors: http://software.intel.com/en-us/intel-cluster-studio-xe This potentially enables you to develop on the same source code trees on HPC platforms and your laptop!

Caveats

  • Environment modules assume that during unloading, all previously loaded variables are there as they were set in the first place
  • Then, if you manually start modifying PATH, LD_LIBRARY_PATH etc (which you can do), you will basically break unload/purge functionality
  • An elegant way to get around this is: start a shell bash, make your experimentation as desired, then exit (ie. reset to previous state) and unload/purge

What is the 1 thing that gets me away best?

OK, that’s not exactly having an automatic single answer because the software world can be a complex labyrinth.

Yet, for most people one of the following two is a really good bet: (ie. the Intel & GNU toolchains, compatible with the standard PRACE environment)

module load PRACE/20130605-ictce-5.3.0

fgeorgatos@gaia-1:~: $ module purge
fgeorgatos@gaia-1:~: $ module av PRACE

---------- /opt/apps/HPCBIOS/modules/toolchain -----------
PRACE/20130605-goolf-1.4.10 PRACE/20130605-ictce-5.3.0
fgeorgatos@gaia-1:~: $ module load PRACE
fgeorgatos@gaia-1:~: $ module list
Currently Loaded Modulefiles:
  1) icc/2013.3.163
  2) ifort/2013.3.163
  3) impi/4.1.0.030
  4) imkl/11.0.3.163
  5) ictce/5.3.0
  6) make/3.82-ictce-5.3.0
  7) Java/1.7.0_10
  8) Bash/4.2-ictce-5.3.0
  9) ncurses/5.9-ictce-5.3.0
 10) tcsh/6.18.01-ictce-5.3.0
 11) Tcl/8.5.12-ictce-5.3.0
 12) Tk/8.5.12-ictce-5.3.0
 13) zlib/1.2.7-ictce-5.3.0
 14) Szip/2.1-ictce-5.3.0
 15) HDF5/1.8.9-ictce-5.3.0
 16) flex/2.5.35-ictce-5.3.0
 17) Bison/2.5-ictce-5.3.0
 18) Doxygen/1.8.1.1-ictce-5.3.0
 19) netCDF/4.1.3-ictce-5.3.0
 20) PRACE/20130605-ictce-5.3.0
fgeorgatos@gaia-1:~: $

module load PRACE/20130605-goolf-1.4.10

fgeorgatos@gaia-1:~: $ module purge
fgeorgatos@gaia-1:~: $ module load PRACE/20130605-goolf-1.4.10
fgeorgatos@gaia-1:~: $ module list
Currently Loaded Modulefiles:
  1) GCC/4.7.2
  2) hwloc/1.6.2-GCC-4.7.2
  3) OpenMPI/1.6.4-GCC-4.7.2
  4) gompi/1.4.10
  5) OpenBLAS/0.2.6-gompi-1.4.10-LAPACK-3.4.2
  6) FFTW/3.3.3-gompi-1.4.10
  7) ScaLAPACK/2.0.2-gompi-1.4.10-OpenBLAS-0.2.6-LAPACK-3.4.2
  8) goolf/1.4.10
  9) make/3.82-goolf-1.4.10
 10) Java/1.7.0_10
 11) Bash/4.2-goolf-1.4.10
 12) ncurses/5.9-goolf-1.4.10
 13) tcsh/6.18.01-goolf-1.4.10
 14) Tcl/8.5.12-goolf-1.4.10
 15) Tk/8.5.12-goolf-1.4.10
 16) zlib/1.2.7-goolf-1.4.10
 17) Szip/2.1-goolf-1.4.10
 18) HDF5/1.8.7-goolf-1.4.10
 19) netCDF/4.1.3-goolf-1.4.10
 20) Perl/5.16.3-goolf-1.4.10
 21) PRACE/20130605-goolf-1.4.10
fgeorgatos@gaia-1:~: $

So, what is all this PRACE fuss about?

fgeorgatos@gaia-1:~: $ module show PRACE
-------------------------------------------------------------------
/opt/apps/HPCBIOS/modules/toolchain/PRACE/20130605-ictce-5.3.0:

module-whatis	 The PRACE Common Production Environment (PCPE) is a set of software tools and libraries
 that are planned to be available on all PRACE execution sites. The PCPE also defines a set of environment
 variables that try to make compilation on all sites as homogeneous and simple as possible. - Homepage: http://www.prace-ri.eu/PRACE-Common-Production
conflict	 PRACE
setenv		 EBROOTPRACE /opt/apps/HPCBIOS.20130715/software/PRACE/20130605-ictce-5.3.0
setenv		 EBVERSIONPRACE 20130605
setenv		 EBDEVELPRACE /opt/apps/HPCBIOS.20130715/software/PRACE/20130605-ictce-5.3.0/easybuild/PRACE-20130605-ictce-5.3.0-easybuild-devel
-------------------------------------------------------------------

ie. visit at least the URL reported above to find more details: http://www.prace-ri.eu/PRACE-Common-Production

Hint: We are still missing a small number of packages (GridFTP, hpn-ssh, Mayavi, ParaView, pigz, Visit) to be 100% PRACE CPE compliant; since we have no users asking them yet, these will be handled in low-gear mode; for more info refer to: https://github.com/hpcugent/easybuild-easyblocks/issues/82

What’s next?

  • Show the capabilities of any individual module, like module show
  • Explore (show) the tools built with the open source toolchain, called goolf (ie. GNU)
  • Explore (show) the tools built with the vendor-provided toolchain, called ictce (ie. Intel)
  • Try to understand the various features of modules, perhaps starting with module -h
  • Read on the next guides about how to use modules within your scripts and how to develop code
  • Do not miss reading a guide on shell initialization, especially if you run multi-node jobs
  • Building software can get tricky; if in doubt, consult community of hpc-users@uni.lu for advice.

References: