Morrill

Morrill

Morrill is a heterogeneous Linux Cluster with Dell PowerEdge C6525 CPU nodes and HPE Apollo 6500 GPU nodes. Morrill has 2 login nodes, 2 development nodes, 1 data-transfer node, 100 total CPU compute nodes, and 4 GPU compute nodes equipped with 8 NVIDIA A100 GPUs each.

The system as a whole provides:

  • 6,400 AMD EPYC Milan 7543 processor cores @ 2.8 GHz

  • 512 AMD EPYC Milan 7713 processor cores @ 2.0 GHz

  • 26 Terabytes (TB) of RAM

  • 104 Compute Nodes (100 CPU + 4 GPU)

  • 32 NVIDIA A100 GPUs (80 GB ea.)

  • NVIDIA Mellanox HDR100 InfiniBand Interconnect (100 Gb/s node-to-node)

Morrill was originally sold as a ‘Condo’ environment where groups could purchase the rights to a specific amount of hardware. Morrill has since been opened up to allow non-condo-style users like our other clusters. Due to some initial condo-style purchases, some groups are granted pre-emptive rights over others to maintain their hardware provisions.

Node Types

Node Type

Cores

Memory

GPUs

Count

Login

32

64 GB

None

2

DTN

128

128 GB

None

1

Development

128

128 GB

None

2

Compute

64

256 GB

None

100

GPU (A100)

128

1 TB

8x A100 80GB

4

When to Use Morrill

  • Large-scale GPU training

  • Multi-GPU workloads

  • High memory GPU jobs

  • Production ML training

Connection

ssh username@morrill-login.hpc.msstate.edu

For more details on connections, see Getting Started.

Node Naming

Service Nodes

morrill-TYPE-NUMBER.hpc.msstate.edu

Examples:

  • morrill-login.hpc.msstate.edu

Compute Nodes

morrill-RACK-NUMBER.hpc.msstate.edu

Examples:

  • morrill-01-01.hpc.msstate.edu

  • morrill-04-01.hpc.msstate.edu (GPU)

Partitions

See Partitions and Limits

Example Job

#!/bin/bash
#SBATCH --job-name=morrill-multi-gpu
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16
#SBATCH --gres=gpu:a100:4
#SBATCH --mem=256G
#SBATCH --time=08:00:00
#SBATCH --partition=gpu-a100

module purge
module load cuda
python -m torch.distributed.run --nproc_per_node=4 train.py

See Running Jobs for more information.

Advanced Cluster Specifications

CPUs

CPU Architecture

Morrill CPU Compute uses the AMD EPYC Milan 7543 Processor which has 32 cores, 256MB of cache, and runs at 2.8GHz with a TDP of 225 Watts. It is in the Zen 3 Milan family that features advanced instructions sets such as SSE4.2 and AVX2 as well as being fabricated on the 7nm process. Since each processor consist of 9 dies, it has 8 CCXs grouped in paris to make a CCD. Each one of these CCDs acts as a NUMA node. The ninth die is solely for memory and I/O.Therefore, there are four NUMA nodes per CPU. Morrill nodes consist of two of these CPUs per node, each node can be treated as “eight” sockets due to the memory divide. If you’d like to learn more about NUMA, see NUMA Design.

Morrill GPU Compute uses the AMD EPYC Milan 7713 Processor which has 64 cores, 256MB of cache, and runs at 2.0GHz with a TDP of 225 Watts. It is in the Zen 3 Milan family that features advanced instructions sets such as SSE4.2 and AVX2 as well as being fabricated on the 7nm process. Since each processor consist of 9 dies, it has 8 CCXs grouped in paris to make a CCD. Each one of these CCDs acts as a NUMA node. The ninth die is solely for memory and I/O.Therefore, there are four NUMA nodes per CPU. Morrill nodes consist of two of these CPUs per node, each node can be treated as “eight” sockets due to the memory divide. If you’d like to learn more about NUMA, see NUMA Design.

Pipeline Design

The AMD EPYC Milan processor family is an out-of-order cpu meaning it is able to execute several instructions during each cycle on each core. It features a 4-wide decode stage with a 6-wide dispatch stage meaning it can run 6 instructions per cycle, where at least two must be from op-cache (branch prediction). This may not be super relevant when programming, but its significant to your compiler, so make sure to use the relevant recommended flags.

../../_images/zen3-milan-fig.png

GPUs

GPU Architecture

Morrill has four nodes of 8x NVIDIA A100 SXM4 80GB. This card uses the Ampere architecture and is built on the 7nm process. It consists of 6912 CUDA cores, 432 Tensor cores, and 80GB of HBM2e memory. It is capable of doing 31 TFLOPs of FP64 with 2.04 TB/s memory bandwidth and a TDP of 400 Watts.

Networking

Cluster Innerconnect

Morrill uses Mellanox’s HDR Infiniband fabric as the interconnect, providing low latency RDMA between nodes, operating at 100 Gbit/s to the node.

Internet Access

Morrill has one DTN node with dual 40 Gbit/s uplinks to the internet.