Lustre Filesystem Guide¶
Lustre is an open-source, distributed parallel filesystem designed for:
Scalability - Supports very large scale infrastructure
High-performance - Parallel I/O across multiple storage nodes
High-availability - Fault-tolerant architecture
POSIX compliance - Conforms to POSIX standards for filesystem interfaces
Lustre is generally used as primary storage for compute jobs.
Note
Lustre filesystems are mounted on Orion and Hercules only. Refer to the tables on Storage.
Important
Long term storage of data should not be on cluster storage as most directories are not backed up and some directories are subject to regular purges. Data should be moved off of the cluster when not in use.
Directory Structure¶
NOAA projects are provided space under either /work or /work2 as determined by RDHPCS. This document will refer to /work, but information here may apply to /work2 as well.
MSU users may request space on /work. To request space, email help@hpc.msstate.edu with the cluster name, the user or project name, a justification, and the storage allocation size you need. MSU users are provided space under /scratch automatically, however note that this space is subject to regular purges. Shared project scratch directories can be created upon request and with justification.
Directories follow the structure outline in this table:
Directory Type |
Path |
|---|---|
NOAA Project |
|
MSU Project |
|
MSU User |
|
MSU Project Scratch |
|
MSU User Scratch |
|
Best Practices¶
Data Lifecycle¶
Input Data: Copy to
/workbefore running jobsJob Execution: Read/write from
/workduring computationResults: Transfer important data to project directories
For details on how to transfer files to the cluster, see the File Transfer Guide for more details.
Advanced Usage: File Striping¶
Lustre stripes files across multiple storage nodes for parallel I/O performance.
Warning
Improper file striping can cause I/O performance issues for your jobs. Proceed with caution.
Default Striping¶
/work
File Size |
Stripe Count |
Stripe Size |
|---|---|---|
< 1 TiB |
1 |
2 GiB |
1 TiB - 1 PiB |
4 |
2 GiB |
> 1 PiB |
16 |
2 GiB |
/work2
File Size |
Stripe Count |
Stripe Size |
|---|---|---|
< 256 MiB |
1 |
1 GiB |
256 MiB - 1 TiB |
1 |
1 GiB |
1 TiB - 16 TiB |
4 |
1 GiB |
> 16 TiB |
8 |
1 GiB |
Checking Striping¶
# View stripe layout of a file
lfs getstripe $WORK/users/$USER/myfile.txt
# View default stripe layout for a directory
lfs getstripe $WORK/users/$USER/my_project
Example output:
lcm_layout_gen: 0
lcm_mirror_count: 1
lcm_entry_count: 3
lcme_id: N/A
lcme_mirror_id: N/A
lcme_flags: 0
lcme_extent.e_start: 0
lcme_extent.e_end: 1073741824
stripe_count: 1 stripe_size: 2097152 pattern: raid0 stripe_offset: -1
lcme_id: N/A
lcme_mirror_id: N/A
lcme_flags: 0
lcme_extent.e_start: 1073741824
lcme_extent.e_end: 1099511627776
stripe_count: 4 stripe_size: 2097152 pattern: raid0 stripe_offset: -1
lcme_id: N/A
lcme_mirror_id: N/A
lcme_flags: 0
lcme_extent.e_start: 1099511627776
lcme_extent.e_end: EOF
stripe_count: 16 stripe_size: 2097152 pattern: raid0 stripe_offset: -1
This means that there is a progressive file layout for files of various sizes: lcm_entry_count is the number of striping definitions associated with the file or path. stripe_count and stripe_size are self explanatory. All file sizes are given in KiB.
Setting Striping on Directories¶
# Set stripe size to 4 MiB and stripe count to 4
lfs setstripe -S 4M -c 4 $WORK/users/$USER/my_project
# Set to use all available nodes
lfs setstripe -c -1 $WORK/users/$USER/my_project
Note
This affects only new files created in the directory.
Migrating Existing Files¶
# Change stripe layout of existing file
lfs migrate -S 4M -c 4 $WORK/users/$USER/large_file.txt
When to Use Striping¶
Scenario |
Recommendation |
|---|---|
Single large file (>10GiB) |
High stripe count (8-16) |
Many small files |
Low stripe count (1-2) |
Parallel I/O from many nodes |
High stripe count |
Sequential I/O |
Low stripe count |
Rule of thumb: Stripe over as few objects as needed, no more.
Progressive File Layouts (PFL)¶
PFL allows different striping parameters for different file sizes in the same directory.
Creating a PFL Layout¶
# Define tiers:
# - First 256 MiB: 1 stripe
# - 256 MiB-2 GiB: 4 stripes
# - Above 2 GiB: all nodes
lfs setstripe -E 256M -c 1 -E 2G -c 4 -c -1 $WORK/users/$USER/my_project
200 MiB file: Single stripe (under 256 MiB threshold)
2.5 GiB file: First 256 MiB on 1 stripe, remainder on 4 stripes
5 GiB file: First 4 GiB on defined tiers, rest on all nodes
Common Commands¶
# Check stripe layout
lfs getstripe <file|directory>
# Set stripe on directory (new files)
lfs setstripe -S <size> -c <count> <directory>
# Migrate existing file
lfs migrate -S <size> -c <count> <file>
# Check filesystem statistics
lfs df /work
lfs df -h /work
# Check OST usage
lfs df -T /work
Troubleshooting¶
Slow I/O Performance¶
Possible causes:
Running from
/homeinstead of/work(check for symbolic links that refer to file paths in/home)Small stripe count for large files
Too many small files with high striping
“No space left on device”¶
Even with no quota, you may hit filesystem limits:
# Check actual usage
lfs df -h /work
Do not run find or du across /work to locate large files. It is a shared filesystem and the
traversal slows it for every user. For assistance, contact the helpdesk at help@hpc.msstate.edu.