Atlas

Atlas is a Cray CS500 Linux cluster. Atlas has 2 login nodes, 2 development nodes, 2 data-transfer nodes, 228 standard compute nodes, 8 big memory nodes, and 21 GPU nodes with various Nvidia GPU configurations.

The system as a whole provides:

• 11,520 Intel Xeon Platinum 8260 (Cascade Lake) processor cores

• 768 Intel Xeon Gold 6548Y+ (Emerald Rapids) processor cores

• 640 AMD EPYC Milan 7713 processor cores

• 101 Terabytes (TB) of RAM

• 258 Compute Nodes (228 standard + 8 big memory + 21 GPU)

• 8 NVIDIA V100 GPUs (32 GB RAM per GPU)

• 40 NVIDIA A100 GPUs (80 GB RAM per GPU)

• 48 NVIDIA L40S GPUs (48 GB RAM per GPU)

• Mellanox HDR100 InfiniBand Interconnect (100 Gbit/s node-to-node)

The peak performance of Atlas is 565 TeraFLOPS. The system is a collaboration between Mississippi State University and the U.S. Department of Agriculture’s Agricultural Research Service (ARS).

Node Types

Node Type	Cores	Memory	GPUs	Count
Login	48	384 GB	None	2
DTN	48	192 GB	None	2
Development	48	512 GB	None	2
Compute	48	384 GB	None	228
Compute (Big Mem)	48	1.5 TB	None	8
GPU (V100)	48	384 GB	2x V100 32GB	4
GPU (A100)	128	2 TB	8x A100 80GB mig=7	2
GPU (A100)	128	2 TB	8x A100 80GB mig=1	3
GPU (L40S)	64	1.5 TB	4x L40S 48GB	12

When to Use Atlas

Important

This cluster is approved for USDA-ARS use only.

• Atlas: General HPC Workloads

• V100: Legacy GPU code, medium models

• A100: Large models, high memory needs

• L40S: Inference, graphics, modern workloads

Connection

ssh username@atlas-login.hpc.msstate.edu

For more details on connections, see Getting Started.

Node Naming

Service Nodes

atlas-TYPE-NUMBER.hpc.msstate.edu

Examples:

• atlas-login-1.hpc.msstate.edu

• atlas-devel-1.hpc.msstate.edu

Compute Nodes

atlas-NUMBER.hpc.msstate.edu

Examples:

• atlas-0001.hpc.msstate.edu (CPU)

• atlas-0243.hpc.msstate.edu (A100)

• atlas-0246.hpc.msstate.edu (L40S)

Partitions

See Partitions and Limits

Advanced Cluster Specifications

CPUs

CPU Architecture

CPU/V100

A100

L40S

CPU/V100 Compute

Atlas CPU Compute and V100 Compute uses the Intel Xeon Platinum 8260 Processor which has 24 cores, 35.75MB of cache, and runs at 2.40GHz with a TDP of 165 Watts. It is in the Cascade Lake-SP family that features advanced instructions sets such as SSE4.2 and AVX-512 as well as being fabricated on the 14nm process. Since each processor consists of only a single die, there are only one NUMA node per CPU. Atlas CPU/V100 Compute nodes consist of two of these CPUs per node, so each node can be treated as “two” nodes due to the memory divide. If you’d like to learn more about NUMA, see NUMA Design.

A100 Compute

Atlas A100 Compute uses the AMD EPYC Milan 7713 Processor which has 64 cores, 256MB of cache, and runs at 2.0GHz with a TDP of 225 Watts. It is in the Zen 3 Milan family that features advanced instructions sets such as SSE4.2 and AVX2 as well as being fabricated on the 7nm process. Since each processor consist of 9 dies, it has 8 CCXs grouped in paris to make a CCD. Each one of these CCDs acts as a NUMA node. The ninth die is solely for memory and I/O.Therefore, there are four NUMA nodes per CPU. Atlas A100 Compute consist of two of these CPUs per node, so each node can be treated as “eight” sockets due to the memory divide. If you’d like to learn more about NUMA, see NUMA Design.

L40S Compute

Atlas L40S Compute uses the Intel Xeon Gold 6548Y+ which has 32 cores, 60MB of cache, and runs at 2.5GHz with a TDP of 250 Watts. It is in the Emerald Rapids family that features advanced instructions sets such as SSE4.2, AVX512, and AMX as well as being fabricated on the 10nm process. Since each processor consists of two dies, there are two NUMA nodes per CPU. Atlas L40S Compute consist of two of these CPUs per node, so each node can be treated as “four” nodes due to the memory divide. If you’d like to learn more about NUMA, see NUMA Design.

Pipeline Design

CPU/V100 Compute

The Intel Xeon Platinum 8260 is an out-of-order cpu meaning it is able to execute several instructions during each cycle on each core. It features a 4-wide decode stage meaning it can run 4 instructions per cycle. This may not be super relevant when programming, but its significant to your compiler, so make sure to use the relevant recommended flags.

A100 Compute

The AMD EPYC Milan 7713 is an out-of-order cpu meaning it is able to execute several instructions during each cycle on each core. It features a 4-wide decode stage with a 6-wide dispatch stage meaning it can run 6 instructions per cycle, where at least two must be from op-cache (branch prediction). This may not be super relevant when programming, but its significant to your compiler, so make sure to use the relevant recommended flags.

L40S Compute

The Intel Xeon Gold 6548Y+ is an out-of-order cpu meaning it is able to execute several instructions during each cycle on each core. It features a 6-wide decode stage meaning it can run 6 instructions per cycle. This may not be super relevant when programming, but its significant to your compiler, so make sure to use the relevant recommended flags.

Recommended C Flags

GCC

Below, you’ll find the recommended flags for the GCC compiler.

-march=cascadelake -mprefer-vector-width=512 -fopt-info-vec-all -O3

NVCC

V100

Below, you’ll find the recommended flags for the NVCC compiler with the V100.

-arch=sm_70 -Xcompiler -O3

A100

Below, you’ll find the recommended flags for the NVCC compiler with the A100.

-arch=sm_80 -Xcompiler -O3

L40S

Below, you’ll find the recommended flags for the NVCC compiler with the L40S.

-arch=sm_89 -Xcompiler -O3

Or if you would like to compile for all three GPUs, these are the recommended flags:

nvcc flags for all three

Intel-oneapi

Below, you’ll find the recommended flags for the Intel-oneapi compiler.

-intel-flags

GPUs

GPU Architecture

V100

A100

L40S

V100

Atlas has four nodes with 2x NVIDIA Tesla V100 PCIe 32GB. This card uses the Volta architecture and is built on the 12nm process. It consists of 5120 CUDA cores, 640 Tensor cores, and 32GB of HBM2 memory. It is capable of doing 7 TFLOPs of FP64 with 897 GB/s memory bandwidth and a TDP of 250 Watts.

Example SBATCH Parameters

#SBATCH --partition=gpu-v100
#SBATCH --gres=gpu:v100:1

See Running Jobs for more information.

A100

Atlas has five nodes of 8x NVIDIA A100 PCIe 80GB, two of which are configured to subdivide the GPUs using NVIDIA MIG. This card uses the Ampere architecture and is built on the 7nm process. It consists of 6912 CUDA cores, 432 Tensor cores, and 80GB of HBM2e memory. It is capable of doing 31 TFLOPs of FP64 with 1.94 TB/s memory bandwidth and a TDP of 300 Watts.

Example SBATCH Parameters

#SBATCH --partition=gpu-a100
#SBATCH --gres=gpu:a100:1

Or, for MIGs

#SBATCH --partition=gpu-a100-mig7
#SBATCH --gres=gpu:nvidia_a100_1g.10gb:1

See Running Jobs for more information.

L40S

Atlas has twelve nodes of 4x NVIDIA L40S. This card uses the Ada Lovelace architecture and is built on the 5nm process. It consists of 18176 CUDA cores, 568 Tensor cores, and 48GB of GDDR6 memory. It is capable of doing 1.4 TFLOPs of FP64 with 864 GB/s memory bandwidth and a TDP of 350 Watts.

Example SBATCH Parameters

#SBATCH --partition=gpu-l40s
#SBATCH --gres=gpu:l40s:1

See Running Jobs for more information.

GPU Performance Analysis

Not every GPU is built the same, and this is very true with these cards. Each has its own strong suit and is designed with different workflows in mind. Additionally some cards may support different features. You’ll find support and performance metrics (in TFLOPs) in the table below.

GPU	FP64	FP32	TF32	FP16	BF16	FP8	VRAM	Memory Bandwidth
V100	7.1	14.1	N/A	28.3	N/A	N/A	32GB	897 GB/s
A100	9.7	19.5	156	312	312	N/A	80GB	1.94 TB/s
L40S	1.43	91.6	183	362	362	733	48GB	864 GB/s

Networking

Cluster Interconnect

Atlas uses Mellanox’s HDR Infiniband fabric as the interconnect, providing low latency RDMA between nodes, operating at 100 Gbit/s to the node.

Internet Access

Atlas has two DTN nodes each with a 40 Gbit/s uplink to the internet.