Gaia uses Lustre for its $SCRATCH filesystem. Lustre has three major functional units:
- The metadata servers (MDS), holding metadata targets (MDT) which stores metadata such as filenames, directories, permissions, file properties, etc.
- The object storage servers (OSS), holding the object storage targets (OST) which stores the file data. The total capacity of the filesystem is the sum of the OST capacities.
- The clients, that access and use the data.
Gaia Lustre infrastructure is composed of 2 MDS servers and 6 OSS servers (2 MDT and 24 OST). The following schema describes our infrastructure:
You can list the MDTs and OSTs with the command
- The MDT 0 is stored on the MDS Nexsan enclosure
- The OST 0 to 5 are stored on the OSS1 Nexsan enclosure
- The OST 6 to 11 are stored on the OSS2 Nexsan enclosure
- The OST 12 to 17 are stored on the OSS3&4 Netapp enclosure
- The OST 18 to 23 are stored on the OSS5&6 Netapp enclosure
Lustre is able to distribute the segments of a single file across several OSTs. This technique is called file striping.
File striping permits to increase the throughput of operations by taking advantage of several OSSs and OSTs, by allowing one or more clients to read/write different parts of the same file in parallel. On the other hand, striping small files can decrease the performance.
File striping allows file sizes larger than a single OST, large files MUST be striped over several OSTs in order to avoid filling a single OST and harming the performance for all users.
We can tune file striping using 3 properties:
|Property||Effect||Default||Accepted values||Advised values|
|stripe_size||Size of the file stripes in bytes||1048576 (1m)||> 0||> 0|
|stripe_count||Number of OST to stripe across||2||-1 (use all the OSTs), 1-24||1-6|
|stripe_offset||Index of the OST where the first stripe of files will be written||-1 (automatic)||-1, 0-23||-1|
You can see the settings applied on a directory or file with this command:
How to set the file striping parameters
You can tune the file striping parameters per directory or create a new file with specific striping parameters with the command
Newly created files and directories will inherit these parameters from their parent directory. However, the parameters cannot be changed on an existing file.
usage: setstripe -d <directory> (to delete default striping) usage: setstripe [--stripe-count|-c <stripe_count>] [--stripe-index|-i <start_ost_idx>] [--stripe-size|-S <stripe_size>]
|stripe_size||Number of bytes on each OST. Can be specified with k, m or g (in KB, MB and GB respectively)|
|stripe_count||Number of OSTs to stripe over|
|start_ost_idx||OST index of first stripe (default is -1, let lustre choose the optimal OST)|
- Set the striping parameters for a directory containing only small files (< 20MB)
- Set the striping parameters for a directory containing only large files between 100MB and 1GB
- Set the striping parameters for a directory containing files larger than 1GB
Note that these are simple examples, the optimal settings defer depending on the application (concurrent threads accessing the same file, size of each write operation, etc).
For more details, you can read the following external resources: