Hercules

Hercules

Hercules is a Dell/EMC PowerEdge C6520 Linux cluster. Hercules has 4 login nodes, 2 development nodes, 4 data-transfer nodes, and 1,024 compute nodes. In 2025 Hercules was expanded with an additional 512 Dell PowerEdge 6600 compute nodes with Intel Sapphire-Rapids Processors.

The system as a whole provides:

  • 81,920 Intel Xeon Platinum 8380 processor cores @ 2.3 GHz

  • 81,920 Intel Xeon Platinum 8460Y+ processor cores @ 2.0GHz

  • 512 Terabytes (TB) of RAM

  • 1,024 Compute Nodes (512 + 512)

  • NVIDIA/Mellanox InfiniBand HDR/NDR Interconnect (200Gbit/s node-to-node)

Hercules has a peak performance of 3.01 PetaFLOPS, and debuted at 369th overall on the November 2022 TOP500 Supercomputer Sites list.

Node Types

Node Type

Cores

Memory

Count

Login

160

256 GB

4

DTN

160

96 GB

4

Development

160

192 GB

2

Compute (Hercules)

80

512 GB

512

Compute (Hercules-2)

80

512 GB

512

When to Use Hercules

  • High-core count requirements

  • Memory up to 512 GB per node

  • MPI parallel workloads

Connection

ssh username@hercules-login.hpc.msstate.edu

For more details on connections, see Getting Started.

Node Naming

Service Nodes

hercules-TYPE-NUMBER.hpc.msstate.edu

Examples:

  • hercules-login-1.hpc.msstate.edu

  • hercules-devel-2.hpc.msstate.edu

Compute Nodes

hercules-RACK-NUMBER.hpc.msstate.edu

Examples:

  • hercules-01-01.hpc.msstate.edu

  • hercules-09-01.hpc.msstate.edu (Hercules-2)

Partitions

See Partitions and Limits

Example Job

#!/bin/bash
#SBATCH --job-name=hercules-mpi
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=40
#SBATCH --cpus-per-task=1
#SBATCH --mem=64G
#SBATCH --time=04:00:00

module purge
module load openmpi
mpirun -np 80 ./mpi_program

See Running Jobs for more information.

Advanced Cluster Specifications

CPUs

CPU Architecture

Hercules is divided into two partitions, Hercules V1 and Hercules V2. Although very similar, they do use different CPUs.


Hercules V1 uses the Intel Xeon Platinum 8380 which has 40 cores, 60MB of cache, and runs at 2.30GHz with a TDP of 270 Watts. It is in the Ice Lake family that features advanced instructions sets such as SSE4.2 and AVX-512 as well as being fabricated on the 10nm process. Since each processor consist of only a single die, it has one NUMA node per CPU. Atlas nodes consist of two of these CPUs per node, each node can be treated as “two” sockets due to the memory divide. If you’d like to learn more about NUMA, see NUMA Design.


Hercules V2 uses the Intel Xeon Platinum 8460Y+ which has 40 cores, 105MB of cache, and runs at 2.00GHz with a TDP of 300 Watts. It is in the Sapphire Rapids family that features advanced instructions sets such as SSE4.2, AVX-512, and AMX as well as being fabricated on the 10nm process. Since each processor consist of four dies, it has four NUMA node per CPU. Atlas nodes consist of two of these CPUs per node, each node can be treated as “eight” sockets due to the memory divide. If you’d like to learn more about NUMA, see NUMA Design.

Pipeline Design


The Intel Xeon Platinum 8380 is an out-of-order cpu meaning it is able to execute several instructions during each cycle on each core. It features a 5-wide decode stage and a 4-wide issue stage meaning it can run 5 instructions per cycle, up to 4 of which can be arithmetic instuctions. This may not be super relevant when programming, but its significant to your compiler, so make sure to use the relevant recommended flags.

../../_images/ice-lake-fig.png


The Intel Xeon Platinum 8460Y+ is an out-of-order cpu meaning it is able to execute several instructions during each cycle on each core. It features a 6-wide decode stage and a 5-wide issue stage meaning it can run 6 instructions per cycle, up to 5 of which can be arithmetic instuctions. This may not be super relevant when programming, but its significant to your compiler, so make sure to use the relevant recommended flags.

../../_images/sapphire-rapids-fig.png

Networking

Cluster Innerconnect

Hercules uses both Mellanox’s HDR and NDR Infiniband fabric as the interconnect, providing low latency RDMA between nodes, operating at 200 Gbit/s to the node. Hercules follows a Fat Tree topology with a leaf switch for every 16 nodes. In total, Hercules incorporates 32 leaf switches and 8 spine switches.

Internet Access

Hercules has four DTN nodes each with a 40 Gbit/s uplink to the internet.