Orion¶

Orion

Orion is a Dell/EMC PowerEdge C6420 cluster. Orion has 4 login nodes, 2 development nodes, 4 data-transfer nodes, and 1,800 compute nodes, 8 of which are large-memory nodes with increased RAM capacity.

The system as a whole provides:

72,000 Intel Xeon Gold 6148 processor cores @ 2.4 GHz
345 Terabytes (TB) of RAM
1,800 Compute Nodes (1,792 standard @ 192 GB, 8 large-memory @ 384 GB)
Mellanox HDR100 InfiniBand Interconnect (100 Gbit/s node-to-node)

Orion has a peak performance of nearly 5.5 PetaFLOPS and debuted on the June 2019 TOP500 Supercomputer Sites list as the 62nd most powerful computer in the world and 4th fastest computer in U.S. academia. However, due to technical reasons the initial rankings were conducted on only 85% of the available nodes. Later benchmarking effort utilizing nearly all of the available nodes resulted in a ranking of 60th most powerful and 5th fastest computer in U.S. Academia on the November 2019 TOP500 Supercomputer Sites list.

Node Types¶

Node Type	Cores	Memory	Count
Login	40	192 GB	4
DTN	40	96 GB	4
Development	40	192 GB	2
Compute	40	192 GB	1792
Compute (Big Mem)	40	384 GB	8

When to Use Orion¶

General CPU workloads
Memory requirements up to 384 GB
Jobs under 24 hours
Testing and development

Connection¶

ssh username@orion-login.hpc.msstate.edu

For more details on connections, see Getting Started.

Node Naming¶

Service Nodes

orion-TYPE-NUMBER.hpc.msstate.edu

Examples:

orion-login-1.hpc.msstate.edu
orion-devel-2.hpc.msstate.edu

Compute Nodes

orion-RACK-NUMBER.hpc.msstate.edu

Partitions¶

See Partitions and Limits

Example Job¶

#!/bin/bash
#SBATCH --job-name=orion-job
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --time=02:00:00

module purge
module load python
python analysis.py

See Running Jobs for more information.

Advanced Cluster Specifications¶

CPUs¶

CPU Architecture¶

Orion uses the Intel Xeon Gold 6148 Processor which has 20 cores, 27.5MB of cache, and runs at 2.40GHz with a TDP of 150 Watts. It is in the Skylake-SP family that features advanced instructions sets such as SSE4.2 and AVX-512 as well as being fabricated on the 14nm process. Since each processor consist of only a single die, it has one NUMA node per CPU. Orion nodes consist of two of these CPUs per node, each node can be treated as “two” nodes due to the memory divide. If you’d like to learn more about NUMA, see NUMA Design.

Pipeline Design¶

The Intel Xeon Platinum 8260 is an out-of-order cpu meaning it is able to execute several instructions during each cycle on each core. It features a 4-wide decode stage meaning it can run 4 instructions per cycle. This may not be super relevant when programming, but its significant to your compiler, so make sure to use the relevant recommended flags.

Recommended C Flags¶

GCC

Below, you’ll find the recommended flags for the GCC compiler.

-march=skylake-avx512 -mprefer-vector-width=512 -fopt-info-vec-all -O3

Intel-oneapi

Below, you’ll find the recommended flags for the Intel-oneapi compiler.

-flags-for-intel

Networking¶

Cluster Innerconnect¶

Orion uses Mellanox’s HDR Infiniband fabric as the interconnect, providing low latency RDMA between nodes, operating at 100 Gbit/s to the node. Orion follows a Fat Tree topology with a leaf switch for every 49 nodes. In total, Orion incorporates 38 leaf switches and 14 spine switches.

Internet Access¶

Orion has four DTN nodes each with a 40 Gbit/s uplink to the internet.