Hercules

Hercules is a Dell/EMC PowerEdge C6520 Linux cluster. Hercules has 4 login nodes, 2 development nodes, 4 data-transfer nodes, and 1,024 compute nodes. In 2025 Hercules was expanded with an additional 512 Dell PowerEdge 6600 compute nodes with Intel Sapphire-Rapids Processors.

The system as a whole provides:

• 81,920 Intel Xeon Platinum 8380 processor cores @ 2.3 GHz

• 81,920 Intel Xeon Platinum 8460Y+ processor cores @ 2.0GHz

• 512 Terabytes (TB) of RAM

• 1,024 Compute Nodes (512 + 512)

• NVIDIA/Mellanox InfiniBand HDR/NDR Interconnect (200Gbit/s node-to-node)

Hercules has a peak performance of 3.01 PetaFLOPS, and debuted at 369th overall on the November 2022 TOP500 Supercomputer Sites list.

When to Use Hercules

• High-core count requirements

• Memory up to 512 GB per node

• MPI parallel workloads

Connection

ssh username@hercules-login.hpc.msstate.edu

For more details on connections, see Getting Started.

Node Naming

Service Nodes

hercules-TYPE-NUMBER.hpc.msstate.edu

Examples:

• hercules-login-1.hpc.msstate.edu

• hercules-devel-2.hpc.msstate.edu

Compute Nodes

hercules-RACK-NUMBER.hpc.msstate.edu

Examples:

• hercules-01-01.hpc.msstate.edu

• hercules-09-01.hpc.msstate.edu (Hercules-2)

Partitions

Partition	Total Nodes	Nodes	Mem/Node	Mem/Task	Tasks/Node	Allowed Qos
hercules*	512	hercules-01-[01-64]…hercules-08-[01-64]	500 GB	6.25 GB	80	All
hercules-2	512	hercules-09-[01-64]…hercules-16-[01-64]	500 GB	6.25 GB	80	All
development	2	hercules-devel-[1-2]	251 GB	3.14 GB	80	All
service	4	hercules-dtn-[1-4]	251 GB	3.14 GB	72	batch,urgent,windfall,normal,debug,long

QoS

QoS	MaxCPUs	MaxNodes	MaxTime	Priority	Notes
normal	-	5	48 hours	20	Default QoS
priority	-	UNLIMITED	UNLIMITED	50	By request and approval only
special	-	UNLIMITED	UNLIMITED	20	By request and approval only
debug	-	250	30 minutes	30	Debugging Jobs
batch	-	250	8 hours	20	NOAA Default
windfall	-	250	8 hours	1	NOAA Low Priority
urgent	-	250	8 hours	40	NOAA High Priority
ood	2	1	8 days	1	Low-Resource, Long-Running for Interactive apps
novel	-	UNLIMITED	8 hours	50
long	-	125	16 hours	20	by RDHPCS approval only

Example Job

#!/bin/bash
#SBATCH --job-name=hercules-mpi
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=40
#SBATCH --cpus-per-task=1
#SBATCH --mem=64G
#SBATCH --time=04:00:00

module purge
module load openmpi
srun ./mpi_program

See Running Jobs for more information.

Advanced Cluster Specifications

CPUs

CPU Architecture

Hercules is divided into two partitions, Hercules V1 and Hercules V2. Although very similar, they do use different CPUs.

V1

V2

Hercules V1

Hercules V1 uses the Intel Xeon Platinum 8380 which has 40 cores, 60MB of cache, and runs at 2.30GHz with a TDP of 270 Watts. It is in the Ice Lake family that features advanced instructions sets such as SSE4.2 and AVX-512 as well as being fabricated on the 10nm process. Since each processor consist of only a single die, it has one NUMA node per CPU. Hercules V1 nodes consist of two of these CPUs per node, each node can be treated as “two” sockets due to the memory divide. If you’d like to learn more about NUMA, see NUMA Design.

Hercules V2

Hercules V2 uses the Intel Xeon Platinum 8460Y+ which has 40 cores, 105MB of cache, and runs at 2.00GHz with a TDP of 300 Watts. It is in the Sapphire Rapids family that features advanced instructions sets such as SSE4.2, AVX-512, and AMX as well as being fabricated on the 10nm process. Since each processor consist of four dies, it has four NUMA node per CPU. Hercules V2 nodes consist of two of these CPUs per node, each node can be treated as “eight” sockets due to the memory divide. If you’d like to learn more about NUMA, see NUMA Design.

Pipeline Design

V1

The Intel Xeon Platinum 8380 is an out-of-order cpu meaning it is able to execute several instructions during each cycle on each core. It features a 5-wide decode stage and a 4-wide issue stage meaning it can run 5 instructions per cycle, up to 4 of which can be arithmetic instuctions. This may not be super relevant when programming, but its significant to your compiler, so make sure to use the relevant recommended flags.

V2

The Intel Xeon Platinum 8460Y+ is an out-of-order cpu meaning it is able to execute several instructions during each cycle on each core. It features a 6-wide decode stage and a 5-wide issue stage meaning it can run 6 instructions per cycle, up to 5 of which can be arithmetic instuctions. This may not be super relevant when programming, but its significant to your compiler, so make sure to use the relevant recommended flags.

Recommended C Flags

V1

GCC

Below, you’ll find the recommended flags for the GCC compiler.

-march=icelake-server -mprefer-vector-width=512 -fopt-info-vec-all -O3

Intel-oneapi

Below, you’ll find the recommended flags for the Intel-oneapi compiler.

-intel-flags

V2

GCC

Below, you’ll find the recommended flags for the GCC compiler.

-march=sapphirerapids -mprefer-vector-width=512 -fopt-info-vec-all -O3

Intel-oneapi

Below, you’ll find the recommended flags for the Intel-oneapi compiler.

-intel-flags

Note

Performance uplift due to compiler flags is heavily dependent on the code being optimized, read about what these flags do and experiement to find the best optimization

Networking

Cluster Innerconnect

Hercules uses both Mellanox’s HDR and NDR Infiniband fabric as the interconnect, providing low latency RDMA between nodes, operating at 200 Gbit/s to the node. Hercules follows a Fat Tree topology with a leaf switch for every 16 nodes. In total, Hercules incorporates 32 leaf switches and 8 spine switches.

Internet Access

Hercules has four DTN nodes each with a 40 Gbit/s uplink to the internet.