HPC @ Uni.lu

High Performance Computing in Luxembourg

FAQ: How Does OAR Manage the Memory?

OAR isolates users with cpusets, allocating corresponding cores and memory banks derived from your OAR job submission.

You can read a computing node’s number of memory banks in the following special file: /dev/cpuset/oar/cpuset.mems

For instance, if you reserve one core on gaia-73, you will only get one memory bank (1/16 of the total memory = 64GB):

$ cat /dev/cpuset/oar/cpuset.mems

The special file /dev/cpuset/oar/${USER}_${OAR_JOB_ID}/cpuset.mems is created on each node reserved for your job, which shows the reserved memory banks. For example on gaia-73:

$ cat /dev/cpuset/oar/hcartiaux_2694923/cpuset.mems

In this example, the job 2694923 can use only 1 memory bank (cpuset.mems uses zero-based numbering), so if this job would try to use more than 64GB of memory, it will trigger the Out of memory killer mechanism.

On gaia-1 to gaia-72, there are 2 memory banks per node, so if you reserve one core, you will be limited to one memory bank and you can trigger the OOM even if half of the node’s memory is free.

In summary, please adapt your reservation according to the planned memory usage, for memory-bound workflows you may need to reserve complete nodes in order to ensure that you have the full memory available and not get influenced by other user’s jobs.