How to Build Against Libraries Made via EasyBuild?
Linking against an easybuild-provided library is no different than with any other library build in a regular manner. You still need to ensure that all compiler & linker parameters are passed and in the correct order.
ie. your LD\* variables must be correct, along with whatever else your compiler or Makefile needs to be fed with.
If you expect though the environment to be automagically populated with the correct variables by just loading a module, there is one surprise complication though: You really have to load yet another module doing exactly what you ask. This is reasonable because otherwise you would be taken away the freedom to make some complex builds/linking.
Currently, this is only needed while in interactive mode, because EasyBuild will do tune the environment in the most complete way while building your easyconfig (on the assumption that you would write an easyblock, if you really wanted to deviate from the default procedure). This is where the surprise comes from, for most :-)
For now, the simplest approach, while on interactive build mode, is:
load the EBDEVEL\* modulefile of a module which depends on your desired library;
(a subset of) the defined envirnonment variables will now to do the expected tuning.
You can also make a template easyconfig that depends on your library of interest and nothing else
(in fact, this is how to boostrap your build: its generated EBDEVEL\* is what you need, nearly always!).
The described approach is more of a workaround until each automated build will provide something like an EBDEBUG* modulefile with the desired environment. Note that, that even then it is not the final solution, because what happens when you depend on two or more libraries, concurrently? You will eventually be forced to use a template easyconfig file, to retrieve a good development environment, which indeed requires some preparation and insight in the build process!
Boost is a notable library, so let’s use that as an example, to better understand how this works:
We picked ABySS as the workaround, but this is completely arbitrary and it is only used to save keystrokes. Any other similar modulefile depending on Boost will do, if you have only one dependency.
fgeorgatos@gaia-31:~$ module load ABySS # this one depends on Boost
fgeorgatos@gaia-31:~$ module list # notice the dependent library in slot #11
Currently Loaded Modulefiles:
1) icc/2011.6.233 4) imkl/10.3.6.233 7) zlib/1.2.7-ictce-4.0.6 10) Python/2.7.3-ictce-4.0.6
2) ifort/2011.6.233 5) ictce/4.0.6 8) libreadline/6.2-ictce-4.0.6 11) Boost/1.49.0-ictce-4.0.6-Python-2.7.3
3) impi/4.0.2.003 6) bzip2/1.0.6-ictce-4.0.6 9) ncurses/5.9-ictce-4.0.6 12) ABySS/1.3.4-ictce-4.0.6-Python-2.7.3
fgeorgatos@gaia-31:~$ echo $EBDEVELABYSS
/opt/apps/HPCBIOS.20130301/software/ABySS/1.3.4-ictce-4.0.6-Python-2.7.3/easybuild/ABySS-1.3.4-ictce-4.0.6-Python-2.7.3-easybuild-devel
fgeorgatos@gaia-31:~$ grep -i boost !$
grep -i boost $EBDEVELABYSS
if { ![is-loaded /opt/apps/HPCBIOS.20130301/software/Boost/1.49.0-ictce-4.0.6-Python-2.7.3/easybuild/Boost-1.49.0-ictce-4.0.6-Python-2.7.3-easybuild-devel] } {
module load /opt/apps/HPCBIOS.20130301/software/Boost/1.49.0-ictce-4.0.6-Python-2.7.3/easybuild/Boost-1.49.0-ictce-4.0.6-Python-2.7.3-easybuild-devel
setenv CPPFLAGS "-I/opt/apps/HPCBIOS.20130301/software/imkl/10.3.6.233/mkl/include -I/opt/apps/HPCBIOS.20130301/software/Boost/1.49.0-ictce-4.0.6-Python-2.7.3/include"
setenv _LMFILES_ "/opt/apps/HPCBIOS.20130301/modules/all/EasyBuild/1.2.0:/opt/apps/HPCBIOS.20130301/modules/all/icc/2011.6.233:/opt/apps/HPCBIOS.20130301/modules/all/ifort/2011.6.233:/opt/apps/HPCBIOS.20130301/modules/all/impi/4.0.2.003:/opt/apps/HPCBIOS.20130301/modules/all/imkl/10.3.6.233:/opt/apps/HPCBIOS.20130301/modules/all/ictce/4.0.6:/opt/apps/HPCBIOS.20130301/modules/all/bzip2/1.0.6-ictce-4.0.6:/opt/apps/HPCBIOS.20130301/modules/all/zlib/1.2.7-ictce-4.0.6:/opt/apps/HPCBIOS.20130301/modules/all/libreadline/6.2-ictce-4.0.6:/opt/apps/HPCBIOS.20130301/modules/all/ncurses/5.9-ictce-4.0.6:/opt/apps/HPCBIOS.20130301/modules/all/Python/2.7.3-ictce-4.0.6:/opt/apps/HPCBIOS.20130301/modules/all/Boost/1.49.0-ictce-4.0.6-Python-2.7.3"
setenv EBVARCPPFLAGS "-I/opt/apps/HPCBIOS.20130301/software/imkl/10.3.6.233/mkl/include -I/opt/apps/HPCBIOS.20130301/software/Boost/1.49.0-ictce-4.0.6-Python-2.7.3/include"
setenv LDFLAGS "-L/opt/apps/HPCBIOS.20130301/software/icc/2011.6.233/compiler/lib/intel64 -L/opt/apps/HPCBIOS.20130301/software/imkl/10.3.6.233/mkl/lib/intel64 -L/opt/apps/HPCBIOS.20130301/software/imkl/10.3.6.233/compiler/lib/intel64 -L/opt/apps/HPCBIOS.20130301/software/Boost/1.49.0-ictce-4.0.6-Python-2.7.3/lib"
setenv EBVARLDFLAGS "-L/opt/apps/HPCBIOS.20130301/software/icc/2011.6.233/compiler/lib/intel64 -L/opt/apps/HPCBIOS.20130301/software/imkl/10.3.6.233/mkl/lib/intel64 -L/opt/apps/HPCBIOS.20130301/software/imkl/10.3.6.233/compiler/lib/intel64 -L/opt/apps/HPCBIOS.20130301/software/Boost/1.49.0-ictce-4.0.6-Python-2.7.3/lib"
setenv LOADEDMODULES "EasyBuild/1.2.0:icc/2011.6.233:ifort/2011.6.233:impi/4.0.2.003:imkl/10.3.6.233:ictce/4.0.6:bzip2/1.0.6-ictce-4.0.6:zlib/1.2.7-ictce-4.0.6:libreadline/6.2-ictce-4.0.6:ncurses/5.9-ictce-4.0.6:Python/2.7.3-ictce-4.0.6:Boost/1.49.0-ictce-4.0.6-Python-2.7.3"
In fact, the above also sets the “cc” family of variables and gets you just ready to call the makefile build; because the configured variables are many, let’s examine the “cc” subset of it:
fgeorgatos@gaia-31:~$ grep -i 'cc' $EBDEVELABYSS
setenv I_MPI_CC "icc"
setenv EBVARCC "mpicc"
setenv EBVARMPICC "mpicc"
setenv MPICC "mpicc"
setenv MPICH_CC "icc"
setenv EBVARCC_SEQ "icc"
setenv CC_SEQ "icc"
setenv CC "mpicc"
setenv EBVARMPICH_CC "icc"
setenv EBVARI_MPI_CC "icc"
Let’s inspect one more example which is even more generic, since many dependencies are included here:
fgeorgatos@gaia-10:~: $ module load Mothur
fgeorgatos@gaia-10:~: $ module list
Currently Loaded Modulefiles:
1) icc/2013.3.163 7) zlib/1.2.7-ictce-5.3.0
2) ifort/2013.3.163 8) ncurses/5.9-ictce-5.3.0
3) impi/4.1.0.030 9) libreadline/6.2-ictce-5.3.0
4) imkl/11.0.3.163 10) biodeps/1.6-ictce-5.3.0
5) ictce/5.3.0 11) gzip/1.5-ictce-5.3.0
6) bzip2/1.0.6-ictce-5.3.0 12) Mothur/1.30.2-ictce-5.3.0-biodeps-1.6
fgeorgatos@gaia-10:~: $ grep -i 'cc"' $EBDEVELMOTHUR
setenv I_MPI_CC "icc"
setenv EBVARCC "mpicc"
setenv EBVARMPICC "mpicc"
setenv MPICC "mpicc"
setenv MPICH_CC "icc"
setenv EBVARCC_SEQ "icc"
setenv CC_SEQ "icc"
setenv CC "mpicc"
setenv EBVARMPICH_CC "icc"
setenv EBVARI_MPI_CC "icc"
fgeorgatos@gaia-10:~: $ env|grep -i mpicc
fgeorgatos@gaia-10:~: $ module load $EBDEVELMOTHUR
fgeorgatos@gaia-10:~: $ env|grep -i mpicc
MPICC=mpicc
EBVARCC=mpicc
EBVARMPICC=mpicc
CC=mpicc
fgeorgatos@gaia-10:~: $ which $CC
/opt/apps/HPCBIOS.20130715/software/impi/4.1.0.030/bin64/mpicc
fgeorgatos@gaia-10:~: $ which $MPICC
/opt/apps/HPCBIOS.20130715/software/impi/4.1.0.030/bin64/mpicc
fgeorgatos@gaia-10:~: $
Finally, the conclusion is that when you module load $EBDEVELMOTHUR in this case, you will have a good build environment
for software that has the same dependencies (ie. your Makefile should be possible to be written in a clean and generic way).
