How to Use Modules?
Introduction
Environment modules are a standard technology across HPC sites to permit developing and using complex software builds, allowing multiple versions of software stacks and combinations thereof.
Here follows an example; notice that it may only be visible on a cluster node:
(ie. ask for an interactive node slot, if you haven’t done so already, with -I in your submission command)
### bring in a minimal playground environment
fgeorgatos@gaia-48:~: $ bash # use a trick to make it easy to reset the environment
fgeorgatos@gaia-48:~: $ module load standard
fgeorgatos@gaia-48:~: $ module av
--------------- /opt/apps/default/modules/standard ----------------
CUDA/5.0.35-1
FFTW/3.3.3-gompi-1.4.10
GCC/4.7.2
Inspector/2013_update6
Java/1.7.0_10
Java/1.7.0_15
Java/1.7.0_21
LAPACK/3.4.2-gompi-1.4.10
MATLAB/2013a
MVAPICH2/1.9b-GCC-4.7.2
OpenBLAS/0.2.6-gompi-1.4.10-LAPACK-3.4.2
OpenMPI/1.6.4-GCC-4.7.2
ScaLAPACK/2.0.2-gompi-1.4.10-OpenBLAS-0.2.6-LAPACK-3.4.2
TotalView/8.11.0-2-linux-x86-64
VTune/2013_update10
gompi/1.4.10
goolf/1.4.10
hwloc/1.6.2-GCC-4.7.2
icc/2013.3.163
ictce/5.3.0
ifort/2013.3.163
imkl/11.0.3.163
impi/4.1.0.030
fgeorgatos@gaia-48:~: $ exit # clean up the environment, go out of the "standard" view
Let’s look at another example, which attempts to list the packages starting with letter i
(this is a convenient choice, because Intel Cluster Studio packages fit under that pattern)
fgeorgatos@gaia-1:~: $ module av i
-------------------------- /opt/apps/HPCBIOS/modules/compiler ---------------------------
icc/11.1.073 icc/2013.1.117 ifort/11.1.073 ifort/2013.1.117
icc/11.1.075 icc/2013.2.146 ifort/11.1.075 ifort/2013.2.146
icc/2011.10.319 icc/2013.3.163 ifort/2011.10.319 ifort/2013.3.163
icc/2011.13.367 icc/2013.4.183 ifort/2011.13.367 ifort/2013.4.183
icc/2011.6.233 icc/2013.5.192 ifort/2011.6.233 ifort/2013.5.192
----------------------------- /opt/apps/HPCBIOS/modules/mpi -----------------------------
impi/4.0.0.028 impi/4.0.2.003 impi/4.1.0.030
impi/4.0.0.028-32bit impi/4.1.0.027 impi/4.1.1.036
--------------------------- /opt/apps/HPCBIOS/modules/numlib ----------------------------
imkl/10.2.6.038 imkl/10.3.12.361 imkl/11.0.1.117 imkl/11.0.3.163 imkl/11.0.5.192
imkl/10.3.10.319 imkl/10.3.6.233 imkl/11.0.2.146 imkl/11.0.4.183
-------------------------- /opt/apps/HPCBIOS/modules/toolchain --------------------------
iccifort/2011.13.367 ictce/4.0.6 ictce/5.2.0 ictce/5.5.0
ictce/3.2.2.u3 ictce/4.1.13 ictce/5.3.0
ictce/4.0.10 ictce/5.1.1 ictce/5.4.0
As can be seen above, multiple versions of each piece of software may be provided;
Zooming in on the Intel C compiler, icc, there may be several versions of it.
This is necessary, because different software has different dependencies
and in an environment serving multiple users, such a collection makes much sense.
Let’s try to obtain only the set of icc compilers, the above list expanded may include a few:
fgeorgatos@gaia-1:~: $ module av icc
-------------------------- /opt/apps/HPCBIOS/modules/compiler ---------------------------
icc/11.1.073 icc/2011.10.319 icc/2011.6.233 icc/2013.2.146 icc/2013.4.183
icc/11.1.075 icc/2011.13.367 icc/2013.1.117 icc/2013.3.163 icc/2013.5.192
-------------------------- /opt/apps/HPCBIOS/modules/toolchain --------------------------
iccifort/2011.13.367
By default the last version (as sorted alphabetically) will be loaded when asked:
fgeorgatos@gaia-1:~: $ module purge # cleanup any previously loaded modules
fgeorgatos@gaia-1:~: $ module load icc
fgeorgatos@gaia-1:~: $ module list
Currently Loaded Modulefiles:
1) icc/2013.5.192
fgeorgatos@gaia-1:~: $ icc --version
icc (ICC) 13.1.3 20130607
Copyright (C) 1985-2013 Intel Corporation. All rights reserved.
fgeorgatos@gaia-1:~: $
Sometimes, we may need to swap to an alternative one:
fgeorgatos@gaia-1:~: $ module swap icc icc/2013.3.163
fgeorgatos@gaia-1:~: $ module list
Currently Loaded Modulefiles:
1) icc/2013.3.163
fgeorgatos@gaia-1:~: $ icc --version
icc (ICC) 13.1.1 20130313
Copyright (C) 1985-2013 Intel Corporation. All rights reserved.
fgeorgatos@gaia-1:~: $
And then, it may be the case that we want the whole bundle of Intel tools together;
in such situation, we will call for the whole toolchain, defined as ictce:
fgeorgatos@gaia-1:~: $ module purge # cleanup
fgeorgatos@gaia-1:~: $ module load ictce/5.3.0 # the most well tested toolchain, so far
fgeorgatos@gaia-1:~: $ module list
Currently Loaded Modulefiles:
1) icc/2013.3.163 3) impi/4.1.0.030 5) ictce/5.3.0
2) ifort/2013.3.163 4) imkl/11.0.3.163
fgeorgatos@gaia-1:~: $
N.B. The above toolchain includes all of the following:
- Intel C Compiler
- Intel Fortran Compiler
- Intel MPI stack
- Intel MKL libraries # this one provides for BLAS (Linear Algebra) fast implementations, FFT and many more
- the wrapper modulefile
ictce, ie. Intel compiler toolchain & computing environment
If you have not already done so, this is a good moment to start looking in the Intel Cluster Studio XE2013; it is supposed to deliver the most optimized software stacks for generic usage for HPC, on commodity processors: http://software.intel.com/en-us/intel-cluster-studio-xe This potentially enables you to develop on the same source code trees on HPC platforms and your laptop!
Caveats
- Environment modules assume that during unloading, all previously loaded variables are there as they were set in the first place
- Then, if you manually start modifying
PATH,LD_LIBRARY_PATHetc (which you can do), you will basically break unload/purge functionality - An elegant way to get around this is: start a shell
bash, make your experimentation as desired, then exit (ie. reset to previous state) and unload/purge
What is the 1 thing that gets me away best?
OK, that’s not exactly having an automatic single answer because the software world can be a complex labyrinth.
Yet, for most people one of the following two is a really good bet: (ie. the Intel & GNU toolchains, compatible with the standard PRACE environment)
module load PRACE/20130605-ictce-5.3.0
fgeorgatos@gaia-1:~: $ module purge
fgeorgatos@gaia-1:~: $ module av PRACE
---------- /opt/apps/HPCBIOS/modules/toolchain -----------
PRACE/20130605-goolf-1.4.10 PRACE/20130605-ictce-5.3.0
fgeorgatos@gaia-1:~: $ module load PRACE
fgeorgatos@gaia-1:~: $ module list
Currently Loaded Modulefiles:
1) icc/2013.3.163
2) ifort/2013.3.163
3) impi/4.1.0.030
4) imkl/11.0.3.163
5) ictce/5.3.0
6) make/3.82-ictce-5.3.0
7) Java/1.7.0_10
8) Bash/4.2-ictce-5.3.0
9) ncurses/5.9-ictce-5.3.0
10) tcsh/6.18.01-ictce-5.3.0
11) Tcl/8.5.12-ictce-5.3.0
12) Tk/8.5.12-ictce-5.3.0
13) zlib/1.2.7-ictce-5.3.0
14) Szip/2.1-ictce-5.3.0
15) HDF5/1.8.9-ictce-5.3.0
16) flex/2.5.35-ictce-5.3.0
17) Bison/2.5-ictce-5.3.0
18) Doxygen/1.8.1.1-ictce-5.3.0
19) netCDF/4.1.3-ictce-5.3.0
20) PRACE/20130605-ictce-5.3.0
fgeorgatos@gaia-1:~: $
module load PRACE/20130605-goolf-1.4.10
fgeorgatos@gaia-1:~: $ module purge
fgeorgatos@gaia-1:~: $ module load PRACE/20130605-goolf-1.4.10
fgeorgatos@gaia-1:~: $ module list
Currently Loaded Modulefiles:
1) GCC/4.7.2
2) hwloc/1.6.2-GCC-4.7.2
3) OpenMPI/1.6.4-GCC-4.7.2
4) gompi/1.4.10
5) OpenBLAS/0.2.6-gompi-1.4.10-LAPACK-3.4.2
6) FFTW/3.3.3-gompi-1.4.10
7) ScaLAPACK/2.0.2-gompi-1.4.10-OpenBLAS-0.2.6-LAPACK-3.4.2
8) goolf/1.4.10
9) make/3.82-goolf-1.4.10
10) Java/1.7.0_10
11) Bash/4.2-goolf-1.4.10
12) ncurses/5.9-goolf-1.4.10
13) tcsh/6.18.01-goolf-1.4.10
14) Tcl/8.5.12-goolf-1.4.10
15) Tk/8.5.12-goolf-1.4.10
16) zlib/1.2.7-goolf-1.4.10
17) Szip/2.1-goolf-1.4.10
18) HDF5/1.8.7-goolf-1.4.10
19) netCDF/4.1.3-goolf-1.4.10
20) Perl/5.16.3-goolf-1.4.10
21) PRACE/20130605-goolf-1.4.10
fgeorgatos@gaia-1:~: $
So, what is all this PRACE fuss about?
fgeorgatos@gaia-1:~: $ module show PRACE
-------------------------------------------------------------------
/opt/apps/HPCBIOS/modules/toolchain/PRACE/20130605-ictce-5.3.0:
module-whatis The PRACE Common Production Environment (PCPE) is a set of software tools and libraries
that are planned to be available on all PRACE execution sites. The PCPE also defines a set of environment
variables that try to make compilation on all sites as homogeneous and simple as possible. - Homepage: http://www.prace-ri.eu/PRACE-Common-Production
conflict PRACE
setenv EBROOTPRACE /opt/apps/HPCBIOS.20130715/software/PRACE/20130605-ictce-5.3.0
setenv EBVERSIONPRACE 20130605
setenv EBDEVELPRACE /opt/apps/HPCBIOS.20130715/software/PRACE/20130605-ictce-5.3.0/easybuild/PRACE-20130605-ictce-5.3.0-easybuild-devel
-------------------------------------------------------------------
ie. visit at least the URL reported above to find more details: http://www.prace-ri.eu/PRACE-Common-Production
Hint: We are still missing a small number of packages (GridFTP, hpn-ssh, Mayavi, ParaView, pigz, Visit) to be 100% PRACE CPE compliant; since we have no users asking them yet, these will be handled in low-gear mode; for more info refer to: https://github.com/hpcugent/easybuild-easyblocks/issues/82
What’s next?
- Show the capabilities of any individual module, like
module show - Explore (show) the tools built with the open source toolchain, called
goolf(ie. GNU) - Explore (show) the tools built with the vendor-provided toolchain, called
ictce(ie. Intel) - Try to understand the various features of modules, perhaps starting with
module -h - Read on the next guides about how to use modules within your scripts and how to develop code
- Do not miss reading a guide on shell initialization, especially if you run multi-node jobs
- Building software can get tricky; if in doubt, consult community of hpc-users@uni.lu for advice.