Building Parallel Code

Learning Objectives

  • Explain why parallel builds require different compilers and flags than serial builds

  • Configure CMake and Autotools for MPI, OpenMP, and CUDA builds

  • Select appropriate compiler flags for optimization and vectorization

  • Reserve compute resources for builds instead of using login nodes

  • Verify parallel executables with ldd and test runs

  • Avoid common pitfalls: wrong compilers, missing modules, architecture mismatches

Building parallel applications requires the same build systems as serial code (CMake, Autotools), but with special compiler wrappers, flags, and library dependencies to enable parallelism.

Why parallel builds differ:

  • MPI: Code runs across multiple processes on potentially different nodes; requires special compilers (mpicc, mpicxx) that know how to link MPI libraries and set up the runtime

  • OpenMP: Multi-threading within a single process; requires compiler flags (-fopenmp) that generate thread-creation code and link thread libraries

  • CUDA/GPU: Code runs on a different processor entirely; requires NVCC or HIP compilers that compile both host (CPU) and device (GPU) code, then link CUDA runtime libraries

Where to Build

Never build on login nodes. Build operations are CPU- and memory-intensive, and can degrade the experience for all users. Reserve a compute or devel node instead.

Why not login nodes?

  • Login nodes are shared by all users for editing files and submitting jobs

  • Heavy compilation can consume CPU/memory, slowing down the entire cluster

  • Build failures or infinite loops can crash the login node

  • Compute nodes have more resources and are designed for heavy workloads

Interactive Build Session

Reserve a node for an interactive build where you can see compiler output in real-time:

# Interactive build session (4 cores, 1 hour)
# You get a shell on a compute node to run make/cmake interactively
srun -c4 --time=1:00:00 --pty bash

How this works:

  • srun requests resources from the SLURM scheduler

  • -c4 requests 4 CPU cores (adjust based on your build’s parallelism)

  • --time=1:00:00 sets a 1-hour limit (adjust for larger builds)

  • --pty creates a pseudo-terminal so you get an interactive shell

  • Once the scheduler allocates a node, you’re on a compute node, not the login node

Batch Build Job

For large builds or long compilation, submit a batch job instead:

# Create build-job.sh
#!/bin/bash
#SBATCH --job-name=build
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=8
#SBATCH --time=2:00:00
#SBATCH --mem=16G

module purge
module load gcc openmpi

cd $PROJECT_DIR
make -j8

# Submit the build job
sbatch build-job.sh

# Check job status
squeue -u $USER

# View output when complete
tail build-job.sh.out

Why batch for large builds?

  • Jobs queue properly and don’t block if you disconnect

  • Can request more resources (memory, GPUs, many cores)

  • Output is saved to a file for later review

  • Scheduler optimizes cluster utilization

Common Build Patterns

CMake with MPI

CMake is the modern standard for C/C++ builds. MPI requires special compiler wrappers that automatically include the right flags and library paths.

mkdir build; cd build
cmake .. \
  -DCMAKE_C_COMPILER=mpicc \
  -DCMAKE_CXX_COMPILER=mpicxx \
  -DCMAKE_INSTALL_PREFIX=$HOME/.local
make -j$(nproc)
make install

Why use mpicc/mpicxx instead of gcc/g++?

  • These are compiler wrappers, not different compilers

  • They call gcc/g++ internally but add MPI-specific flags automatically

  • They set include paths (-I/usr/include/mpi) so headers are found

  • They set library paths (-L/usr/lib/mpi) so linking works

  • They link against the MPI library (-lmpi) automatically

  • Using gcc directly would result in “undefined reference to MPI_Init” errors

CMake flags explained:

  • -DCMAKE_C_COMPILER=mpicc: Tell CMake to use the MPI wrapper for C

  • -DCMAKE_CXX_COMPILER=mpicxx: Tell CMake to use the MPI wrapper for C++

  • -DCMAKE_INSTALL_PREFIX=$HOME/.local: Install to user-writable location (no root needed)

Why make -j$(nproc)?

  • -j enables parallel compilation (multiple files at once)

  • $(nproc) uses all available cores (faster builds)

  • For MPI builds, this compiles each source file in parallel, then links once

CMake with CUDA

CUDA builds require the NVCC compiler and GPU architecture specifications.

cmake .. \
  -DCMAKE_CXX_COMPILER=nvcc \
  -DCMAKE_CUDA_ARCHITECTURES=80 \
  -DCMAKE_INSTALL_PREFIX=$HOME/.local

Why nvcc?

  • NVCC is NVIDIA’s compiler that handles both host (CPU) and device (GPU) code

  • It separates .cu files into host code (compiled with g++) and device code (compiled with nvcc’s PTX assembler)

  • It links against CUDA runtime libraries automatically

Why CMAKE_CUDA_ARCHITECTURES?

  • GPUs have specific compute capabilities (e.g., 80 = Ampere, 90 = Hopper)

  • This flag tells NVCC which GPU architectures to generate code for

  • 80 targets NVIDIA A100/Ampere GPUs (common on HPC clusters)

  • Multiple architectures: -DCMAKE_CUDA_ARCHITECTURES=70;80;90

  • Using native detects the current GPU: -DCMAKE_CUDA_ARCHITECTURES=native

CMake with OpenMP

OpenMP requires only a compiler flag, no special compiler wrapper:

cmake .. \
  -DCMAKE_CXX_COMPILER=g++ \
  -DCMAKE_CXX_FLAGS="-fopenmp -O3" \
  -DCMAKE_INSTALL_PREFIX=$HOME/.local

How OpenMP differs from MPI:

  • OpenMP uses compiler flags, not wrapper compilers

  • -fopenmp tells the compiler to recognize #pragma omp directives

  • The compiler generates thread-creation code automatically

  • Links against libgomp (GNU OpenMP runtime) automatically

Autotools with MPI

Legacy projects often use Autotools (./configure && make). MPI requires setting environment variables for the configure script.

./configure \
  --prefix=$HOME/.local \
  CC=mpicc CXX=mpicxx
make -j$(nproc)
make install

How Autotools works:

  • ./configure checks for compilers, libraries, and system features

  • It generates a Makefile tailored to your system

  • CC=mpicc sets the C compiler to the MPI wrapper

  • CXX=mpicxx sets the C++ compiler to the MPI wrapper

  • --prefix sets the installation directory

Why environment variables instead of flags?

  • Autotools expects compiler names in CC, CXX, FC variables

  • These are checked during configuration to find headers and libraries

  • Setting them before ./configure ensures the right compilers are detected

Compiler Flags for Parallelism

Different compilers use different flags for the same features. Here’s a reference for common parallelism targets:

Target

GCC/Clang

NVHPC

Intel

Why This Flag?

OpenMP

-fopenmp

-mp

-qopenmp

Enable #pragma omp directives and link thread runtime

Optimization

-O3 -march=native

-O3 -tp=native

-O3 -xHost

Aggressive optimization + CPU-specific instructions

Vectorization

-ffast-math -ffinite-math-only

-fast

-fp-model fast=2

Allow unsafe math for SIMD vectorization

MPI

mpicc/mpicxx

mpicc/mpicxx

mpicc/mpicxx

Use MPI wrapper compilers (not direct flags)

CUDA

N/A

-ccbin g++

N/A

Specify host compiler for NVCC (NVHPC only)

Understanding the Flags

Optimization flags (-O3, -O2, -O0):

  • -O0: No optimization (fastest compilation, slowest code) - use for debugging

  • -O2: Moderate optimization (good balance) - default for many projects

  • -O3: Aggressive optimization (fastest code, longer compile) - use for production

  • Higher optimization enables loop unrolling, inlining, vectorization

Architecture-specific flags:

  • -march=native (GCC/Clang): Generate code optimized for the current CPU

  • -tp=native (NVHPC): Same, for NVIDIA compilers

  • -xHost (Intel): Same, for Intel compilers

  • Why use these? Enables CPU-specific instructions (AVX-512, AVX2, etc.)

  • Warning: Code may not run on older CPUs; only use if targeting a specific machine

Vectorization flags:

  • -ffast-math: Relax IEEE math compliance for speed (allows reordering, assumes no NaNs)

  • -ffinite-math-only: Assume no NaN/Inf, enables more optimizations

  • Why needed? Compilers can’t vectorize code that must handle special floating-point cases

  • Trade-off: Faster code, but may produce slightly different results on edge cases

Common Pitfalls

1. Missing Libraries

Problem: Compiler can’t find MPI headers or CUDA libraries.

Why it happens: Parallel libraries aren’t in the default search paths. You must load them with module load.

Solution:

# Load required modules BEFORE building
module load gcc openmpi cuda

# Now build
mkdir build; cd build
cmake .. -DCMAKE_C_COMPILER=mpicc
make

How to find available modules:

module avail mpi
module avail cuda
module avail openmp

2. Wrong Compiler Wrappers

Problem: Using gcc instead of mpicc for MPI code.

Why it fails:

gcc -o my-mpi-program main.c  # ERROR: undefined reference to `MPI_Init'

Solution:

mpicc -o my-mpi-program main.c  # WORKS: links MPI library automatically

Why mpicc works:

  • mpicc is a wrapper that calls gcc with hidden flags

  • It adds -I/usr/include/mpi (header path)

  • It adds -L/usr/lib/mpi -lmpi (library path and link flag)

  • You don’t need to know these paths manually

3. Architecture Mismatches

Problem: Build on one machine, run on another with different CPU.

Why it fails:

  • If you compile with -march=native on a login node (older CPU)

  • Then run on a compute node (newer CPU), it usually works

  • But if you compile on a newer CPU and run on an older one, it crashes

  • AVX-512 instructions don’t exist on older CPUs

Solution:

# Option 1: Don't use -march=native (most portable)
gcc -O3 -o program main.c

# Option 2: Target the oldest CPU you'll run on
gcc -O3 -march=haswell -o program main.c

# Option 3: Build on the actual compute node
srun --pty bash  # Get on a compute node
# Then build there with -march=native

Build Verification

After building, verify your executable is correctly linked and functional.

Check Linked Libraries

# Check for linked libraries
ldd my-program | grep -E 'mpi|cuda|openmp'

What ldd shows:

  • Lists all shared libraries your program depends on

  • Shows where each library is loaded from

  • If a library shows “not found”, you have a runtime problem

Expected output for MPI program:

libmpi.so.40 => /usr/lib64/libmpi.so.40
libopen-rte.so.40 => /usr/lib64/libopen-rte.so.40
libopen-pal.so.40 => /usr/lib64/libopen-pal.so.40

Run a Quick Test

# Run a quick test on a compute node
srun -n4 ./my-program --test

Why test with srun?

  • MPI programs need to be launched with mpiexec or srun

  • Running ./my-program directly won’t work for MPI code

  • srun -n4 launches 4 MPI processes

  • Test mode (--test) verifies correctness without full workload

Verify OpenMP Threading

# Check if OpenMP is working
export OMP_NUM_THREADS=4
./my-program

# Add this to your program to verify:
#pragma omp parallel
printf("Running on thread %d of %d\n", omp_get_thread_num(), omp_get_num_threads());

See Also

  • MPI - Message Passing Interface for distributed memory

  • OpenMP - Shared memory parallelism with threads

  • CUDA - GPU programming with NVIDIA CUDA

  • Auto-Vectorization - Compiler SIMD optimization