Programming for HPC

Learn to write efficient parallel code that exploits modern CPUs and GPUs. This section covers programming models from auto-vectorization to MPI, with practical examples you can run on ARC clusters.

Who This Is For

Software developers and scientists who write compute-intensive code and want to:

• Exploit SIMD vector units on modern CPUs

• Scale across multi-core processors

• Accelerate workloads on GPUs (A100, V100, L40S)

• Profile and optimize performance bottlenecks

• Manage code with version control

Topic Overview

Topic	Key Skill
Parallel Computing Theory	Amdahl’s Law, Flynn’s taxonomy, scaling limits
Hardware & Data Layout	Memory hierarchy, cache-friendly code, DOP vs OOP
Building Parallel Code	CMake/Autotools with MPI, OpenMP, CUDA flags
Git Basics	Version control for HPC projects
Auto-vectorization	SIMD, compiler flags, measuring bandwidth
OpenMP	Thread-level CPU parallelism with #pragma omp
CUDA	GPU kernels, device memory, transfers
C++17 std::par	GPU-accelerated standard algorithms
OpenMP Offload	GPU programming with #pragma omp target
Distributed-memory parallelism (MPI)	Message passing, distributed computing
Kokkos	Performance portability across CPU/GPU
Profiling	LIKWID hardware counters, roofline model
Valgrind	Memory errors, data races, cache analysis
GPU Profiling with Nsight	Nsight Systems & Nsight Compute

Getting Started

1. Start with theory — Theory and Hardware explain why parallelism matters and how hardware works

2. Set up version control — Learn Git basics for managing your code

3. Build your parallel code — Learn CMake/Autotools with MPI, OpenMP, and CUDA

4. Try auto-vectorization — autovec shows compiler-driven performance gains

5. Scale to multi-core — openmp for CPU parallelism

6. Accelerate with GPUs — cuda, stdpar, and omp-offload for GPU programming

7. Distribute across nodes — mpi for cluster-scale jobs

8. Profile and optimize — profiling, valgrind, and nsight to find and fix bottlenecks

Examples

Throughout this section, there will be example code. You can either copy and paste them from documentation, or you can download them here

Running Examples on ARC Clusters

Each module includes code examples. To run them on ARC clusters:

1. Clone or copy examples to your home directory

2. Create a SLURM job script (see Running Jobs)

3. Request appropriate resources:

   • CPU jobs: #SBATCH --cpus-per-task=8 (adjust for thread count)

   • GPU jobs: #SBATCH --gres=gpu:1 (Ptolemy, Atlas, or Morrill)

4. Load required modules:

   module load gcc     # Or nvhpc for CUDA/stdpar
   module load cuda    # GPU jobs only
   

See SLURM Job Scripts for complete examples.

Common Benchmark: Stream Triad

Most modules use the Stream Triad kernel as a benchmark:

\[A[i] = B[i] + \alpha \times C[i]\]

This memory-bound operation lets you compare performance across programming models (serial → OpenMP → CUDA) on the same algorithm.