HPC @ Uni.lu

High Performance Computing in Luxembourg

The UL HPC platform uses two schedulers on the different generation clusters currently in production:

Depending on your computing hardware/software environment requirements, you may need to opt for one or the other clusters:

Please note that all new computing equipment is added to the iris cluster.

A transition guide from OAR to SLURM is given below.

OAR to SLURM guide

Basic commands:

Command OAR (Gaia/Chaos clusters) SLURM (Iris cluster)
Submit passive/batch job oarsub -S [script] sbatch [script]
Start interactive job oarsub -I srun -p interactive --qos qos-interactive --pty bash
Queue status oarstat squeue
User job status oarstat -u [user] squeue -u [user]
Specific job status (detailed) oarstat -f -j [jobid] scontrol show job [jobid]
Delete (running/waiting) job oardel [jobid] scancel [jobid]
Hold job oarhold [jobid] scontrol hold [jobid]
Resume held job oarresume [jobid] scontrol release [jobid]
Node list and properties oarnodes scontrol show nodes

Job specification:

Specification OAR SLURM
Script directive #OAR #SBATCH
Nodes request -l nodes=[count] -N [min[-max]]
Cores request -l core=[count] -n [count]
Cores-per-node request -l nodes=[ncount]/core=[ccount] -N [ncount] --ntasks-per-node=[ccount]
Walltime request -l [...],walltime=hh:mm:ss -t [min] OR -t [days-hh:mm:ss]
Job array --array [count] --array [specification]
Job name -n [name] -J [name]
Job dependency -a [jobid] -d [specification]
Property request -p "[property]='[value]'" -C [specification]

Environment variables:

Environment variable OAR SLURM
Resource list $OAR_NODEFILE $SLURM_NODELIST #List not file! See note #2
Submitting user name $OAR_USER $SLURM_JOB_USER
Working directory at submission $OAR_WORKING_DIRECTORY $SLURM_SUBMIT_DIR
  • Note #1: each scheduler provides many more environment variables within a job; the list above shows the ones having equivallent purpose. For a comprehensive list of available variables, check out the manual pages of OAR and SLURM or simply run env | egrep "OAR|SLURM" within a job.
  • Note #2: you can easily create a nodefile in the style of OAR, from a SLURM job with srun hostname | sort -n > hostfile.