Ptolemy¶

Ptolemy

Ptolemy is an HPE Apollo 6500 Linux Cluster. Ptolemy has 2 login nodes, 2 development nodes, 1 data-transfer node, and 8 compute nodes, each equipped with 8 NVIDIA A100 GPUs. It uses NVIDIA/Mellanox HDR100 InfiniBand interconnect. The system as a whole provides:

8 Nodes
1,024 AMD EPYC Milan 7713 processor cores @ 2.0 GHz
8 Terabytes (TB) of RAM
64 NVIDIA A100 GPUs (80GB ea.)
NVIDIA/Mellanox HDR200 Interconnect (200Gbit/s node-to-node)

Ptolemy is funded by:

Offices of the President, Provost and Executive Vice President
Office of Research and Economic Development
The Division of Agriculture, Forestry and Veterinary Medicine
The Division of Finance and Administration

Node Types¶

Node Type	Cores	Memory	GPUs	Count
Login	128	512 GB	None	2
DTN	128	512 GB	None	1
Development	128	512 GB	1x A100 (80GB, mig=7)	2
GPU (A100)	128	1 TB	8x A100 (80GB, mig=1)	4
GPU (A100)	128	1 TB	8x A100 (80GB, mig=2)	2
GPU (A100)	128	1 TB	8x A100 (80GB, mig=7)	2

When to Use Ptolemy¶

Deep learning training
GPU-accelerated computing
ML inference
CUDA development
Workloads needing MIG (Multi-Instance GPU)

Connection¶

ssh username@ptolemy-login.arc.msstate.edu

For more details on connections, see Getting Started.

Node Naming¶

ptolemy-TYPE-NUMBER.arc.msstate.edu

Examples:

ptolemy-login-1.arc.msstate.edu
ptolemy-devel-2.arc.msstate.edu
ptolemy-gpu-01.arc.msstate.edu

Partitions¶

See Partitions and Limits

GPU Configurations¶

Nodes	GPUs	MIG	Best For
ptolemy-gpu-[01-04]	8x Nvidia A100 80GB	mig=1	Full GPU access (>40GB)
ptolemy-gpu-[05-06]	8x Nvidia A100 80GB	mig=2	Medium workloads (>10GB)
ptolemy-gpu-[07-08]	8x Nvidia A100 80GB	mig=7	Small jobs (<10GB)

Example Job¶

#!/bin/bash
#SBATCH --job-name=ptolemy-gpu
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --gres=gpu:a100:1
#SBATCH --mem=64G
#SBATCH --time=04:00:00
#SBATCH --partition=<valid_gpu_partition> 
module purge
module load cuda
python train_model.py

See Running Jobs for more information.

Advanced Cluster Specifications¶

CPUs¶

CPU Architecture¶

Ptolemy uses the AMD EPYC Milan 7713 Processor which has 64 cores, 256MB of cache, and runs at 2.0GHz with a TDP of 225 Watts. It is in the Zen 3 Milan family that features advanced instructions sets such as SSE4.2 and AVX2 as well as being fabricated on the 7nm process. Since each processor consist of 9 dies, it has 8 NUMA nodes per CPU, the ninth die is solely for memory and I/O. Ptolemy nodes consist of two of these CPUs per node, each node can be treated as “sixteen” nodes due to the memory divide. If you’d like to learn more about NUMA, see NUMA Design.

Pipeline Design¶

The AMD EPYC Milan 7713 is an out-of-order cpu meaning it is able to execute several instructions during each cycle on each core. It features a 4-wide decode stage with a 6-wide dispatch stage meaning it can run 6 instructions per cycle, where at least two must be from op-cache (branch prediction). This may not be super relevant when programming, but its significant to your compiler, so make sure to use the relevant recommended flags.

Recommended C Flags¶

GCC

Below, you’ll find the recommended flags for the GCC compiler.

-march=znver3 -mprefer-vector-width=512 -fopt-info-vec-all -O3

NVCC

Below, you’ll find the recommended flags for the NVCC compiler.

-arch=sm_80 -Xcompiler -O3

AOCC

Below, you’ll find the recommended flags for the AMD Optimized C compiler.

TODO: aocc flags

GPUs¶

GPU Architecture¶

Ptolemy has eight nodes of 8x NVIDIA A100 SXM4 80GB. This card uses the Ampere architecture and is built on the 7nm process. It consists of 6912 CUDA cores, 432 Tensor cores, and 80GB of HBM2e memory. It is capable of doing 31 TFLOPs of FP64 with 2.04 TB/s memory bandwidth and a TDP of 400 Watts.

Networking¶

Cluster Innerconnect¶

Ptolemy uses Mellanox’s HDR Infiniband fabric as the interconnect, providing low latency RDMA between nodes, operating at 200 Gbit/s to the node.

Internet Access¶

Ptolemy has one DTN node with dual 40 Gbit/s uplinks to the internet.