Kokkos¶

Learning Objectives

Explain the performance-portability problem and how an abstraction layer solves it
Describe Kokkos execution spaces, memory spaces, and Views
Write parallel kernels with parallel_for, KOKKOS_LAMBDA, and MDRangePolicy
Move data between host and device with mirror views and deep_copy
Build the same source for serial, OpenMP, CUDA, or HIP backends by changing one flag

The Portability Problem¶

Every GPU vendor has its own native programming model:

Model	Targets
CUDA	NVIDIA GPUs only
HIP	AMD GPUs (and NVIDIA via a shim)
SYCL / oneAPI	Intel GPUs (and others)
OpenMP / OpenACC	CPUs and, with offload, GPUs

A code written in CUDA runs only on NVIDIA hardware. Porting it to AMD or Intel means rewriting every kernel.

Kokkos solves this by interposing a C++ abstraction layer. You write the algorithm once using Kokkos constructs; at compile time you select a backend (Serial, OpenMP, CUDA, HIP), and Kokkos generates the appropriate low-level code. It is plain C++ — no special compiler required.

Kokkos Core Concepts¶

Execution and Memory Spaces¶

Kokkos separates where code runs from where data lives:

An execution space is a place that runs parallel work — Kokkos::Serial, Kokkos::OpenMP, Kokkos::Cuda, Kokkos::HIP.
A memory space is a place that stores data — host DRAM (HostSpace), CUDA device memory (CudaSpace), etc.

On a GPU build, the default execution and memory spaces are the device; on a CPU build, they are the host.

Views¶

A Kokkos::View is a reference-counted, multidimensional array that lives in a memory space:

Kokkos::View<double**> A("A", N, N);   // N x N array of doubles on the default device

The double** denotes rank-2; the string "A" is a debug label. By default a View lives in device memory.

Parallel Execution¶

Kokkos::parallel_for launches a kernel. The iteration space is described by an execution policy, and the body is a KOKKOS_LAMBDA:

// 1-D range policy
Kokkos::parallel_for("init", N*N, KOKKOS_LAMBDA(const int idx) { /* ... */ });

// 2-D range policy
Kokkos::parallel_for("matmul",
  Kokkos::MDRangePolicy<Kokkos::Rank<2>>({0, 0}, {N, N}),
  KOKKOS_LAMBDA(const int i, const int j) { /* ... */ });

Synchronization and Host Access¶

GPU kernels are asynchronous, so call Kokkos::fence() before timing or reading results. To inspect device data on the host:

auto h_C = Kokkos::create_mirror_view_and_copy(Kokkos::HostSpace(), C);
// h_C(i, j) is now safe to read on the host

Kokkos::deep_copy(C, 0.0) fills a View; deep_copy(dst, src) moves data between spaces.

Matrix Multiply Example¶

Kokkos::View<double**> A("A", N, N), B("B", N, N), C("C", N, N);

// Initialize matrices
Kokkos::parallel_for("init", N*N, KOKKOS_LAMBDA(const int idx) {
  const int i = idx / N;
  const int j = idx % N;
  A(i, j) = 1.0;
  B(i, j) = 2.0;
});

// Matrix multiply: parallel over (i, j), sequential reduction over k
Kokkos::parallel_for("matrix_multiply",
  Kokkos::MDRangePolicy<Kokkos::Rank<2>>({0, 0}, {N, N}),
  KOKKOS_LAMBDA(const int i, const int j) {
    double sum = 0.0;
    for (int k = 0; k < N; ++k) {
      sum += A(i, k) * B(k, j);
    }
    C(i, j) = sum;
  });

Kokkos::fence();

// Copy result to host
auto h_C = Kokkos::create_mirror_view_and_copy(Kokkos::HostSpace(), C);

Building for Any Backend¶

The same matmul.cpp produces a CPU or GPU binary depending only on the build flag:

“Multicore CPU (OpenMP)” bash ./build.sh -t openmp export OMP_NUM_THREADS=8 ./matmul

“NVIDIA GPU (CUDA)” bash ./build.sh -t cuda ./matmul

“Serial (single thread)” bash ./build.sh -t serial ./matmul

Running on ARC Clusters¶

Load dependencies:
```
module load gcc
```

Build Kokkos for your target:

# For GPU (Ptolemy, Atlas, Morrill)
./build.sh -t cuda

Submit job:

#SBATCH --gres=gpu:1  # For GPU builds
sbatch run.slurm