Software installation

What will not work

Any procedure that will try to install software globally and requires root or superuser privileges, e.g. trying to execute sudo will not work on the clusters, and in particular you should not try it on the access nodes.
This includes most of the package managers like apt-get, dpkg, yum and rpm.

Please be aware that there are some exceptions, such as pip for python, which allow you to specify a parameter (e.g. --user or --prefix) that allows the installation to be performed in your home directory instead of globally. Other approaches that employ sandboxing exist, such as virtualenv for Python and RVM for Ruby.

For more help check out the documentation on software installation or the other entries on software.

EasyBuild

EasyBuild is the easiest and fastest way to install a software for yourself. It creates a module that you can load like the other software available on the cluster. The only requirement is an easyconfig file.

Check out the ULHPC tutorial from our latest HPC school on how to use EasyBuild, and also the official HPC UGent docs.

Additonal sources for easyconfigs

If you don’t find an easyconfig for your software on the cluster with:

$> module load base/EasyBuild
$> eb -S <name>

as described in the tutorial, you can check out the EasyConfigs page. Since we freeze the easybuild versions on the cluster when we deploy a set of software, you might find newer versions or other software there. Of course you can also write your own easyconfig or adjust an existing one. More details about that are also in the tutorial linked above.

If you want to use an easyconfig from hpcugent’s (or any other) github repository, you can follow this procedure:

$> mkdir -p ~/git/hpcugent
$> cd ~/git/hpcugent
$> git clone https://github.com/hpcugent/easybuild-easyconfigs.git

$> module load base/EasyBuild
$> eb -r ~/git/hpcugent/easybuild-easyconfigs/easybuild/easyconfigs gtk+-2.24.28-intel-2015a.eb --try-toolchain ictce,7.3.5

Toolchains

Since we use a different toolchain (set of compilers and common libraries, see the definition) on our clusters than HPC UGent’s default, it’s usually a good idea to add --try-toolchain ictce,7.3.5 to your easybuild call. Otherwise it will try to build the full toolchain, which takes a long time and is likely to fail.

RESIF

RESIF is a tool built on top of EasyBuild to facilitate the installation of sets of software. It is used by the UL HPC Management Team to build the whole software stack on the UL HPC clusters. You should consider using RESIF if you want to install several softwares that are available through EasyBuild.

You can find instructions how to use it in the tutorial from the latest HPC school.

From source

If you have to install software manually from source, make sure to read its installation instructions carefully, in particular, look for a section about user-specific or local installation which describes the way to install the software in your HOME directory.

If the installation follows the common configure - make - make install regime, you can easily change the installation location with the --prefix option to the configure step:

$> ./configure --prefix=$HOME

Also for other installation regimes there is often a --prefix option or something similar (e.g. --user, --installpath).

You can find more information in the documentation about compiling.

Python packages

Installation in your $HOME directory

There are (at least) three possible ways to install a python package in your $HOME directory. All of them have to be followed by a configuration of your environment, more specifically the PYTHONPATH.

1. Install with PIP

PIP is the easiest and recommended way to install Python packages.

$> pip install --user <packagename>

2. Install from source

If the package is not available with PIP, you can install if from source in the following way.

Download the source and unpack it. Change to the source directory.

$> python setup.py install --prefix=$HOME/.local

3. Install with easy_install

When using easy_install, you usually have to adjust the PYTHONPATH before the installation. If the directory is not there yet, you have to create it first.

$> mkdir -r $HOME/.local/lib/python2.7/site-packages
$> export PYTHONPATH=$HOME/.local/lib/python2.7/site-packages:$PYTHONPATH

Make sure to insert the right python version if you’re not using 2.7 (see also next section).

$> easy_install --prefix=$HOME/.local <packagename>

Configure your environment

After you installed a python pacakge to your home directory or any other custom location, you need to add that location to your PYTHONPATH. This only has to be done once. The exact path depends on the python version it was installed with. If you followed the instructions above, this path is ~/.local/lib/pythonX.Y/site-packages, where X and Y are the major and minor version of the python you are using (e.g. 2.7 or 3.4). You can check it with python --version. The command to add this location to your environment is

export PYTHONPATH=$HOME/.local/lib/python2.7/site-packages:$PYTHONPATH

(replace 2.7 with your python version). In order to make the change permanent, add this line to your ~/.bash_private (or ~/.bashrc). After this you have to reload the bash configuration.

$> . ~/.bash_private

Virtual environments

In general, Python virtual environments can be really useful to separate different sets of python packages and avoid issues with conflicting versions. They allow you to install packages in isolated locations with easy mechanics to switch between different environments. This is particularly useful if you want to separate packages for different projects or provide a common environment for multiple people.

First, install the virtualenv package with pip like described above and make sure you have added the location to your PYTHONPATH. Then you have to create a directory for your virtual environment and configure it.

$> mkdir <venv_dir>
$> virtualenv <venv_dir>
$> . ./<venv_dir>/bin/activate
$> pip install <python_package>

Replace <venv_dir> with the location where you want your packages to reside and <python_package> with the package name. You can leave the virtualenv with typing

$> deactivate

This brings you back to your usual environment. When you want to use the virtualenv again, you have to activate it with the . ./<venv_dir>/bin/activate command.

Perl modules

To install perl modules yourself, it’s easiest if you set up a local perl library with the local::lib module. This can also allow you to have multiple separate sets of perl modules at different locations (e.g. one for each of your projects).

First, load your desired perl version:

$> module load lang/Perl/5.22.0-ictce-7.3.5-bare

Now, set up a local perl library (see documentation):

$> wget "http://search.cpan.org/CPAN/authors/id/H/HA/HAARG/local-lib-2.000015.tar.gz"
$> tar -xzf local-lib-2.000015.tar.gz
$> cd local-lib-2.000015
$> perl Makefile.PL --bootstrap
$> make test && make install
$> echo '[ $SHLVL -eq 1 ] && eval "$(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)"' >> ~/.bash_private
$> . ~/.bash_private

Finally, install the perl modules with cpan:

$> cpan
cpan> install Config::IniFiles