Multi-processing vs. Multi-threading

When running jobs on HPC clusters using Slurm, choosing between multi-threading and multiprocessing directly affects how you request resources—and getting it wrong can waste compute capacity or limit your program’s performance.

This guide explains the difference, when to use each approach, and how to configure your SLURM job scripts accordingly.

Nomenclature

Process

An independent instance of a program with its own memory space. In HPC, a process typically corresponds to one Slurm task.

Thread

A lightweight execution unit within a process. Threads share the process’s memory space and are used for multi-threaded parallelism.

Rank

In MPI programs, a rank is the identifier of a specific process in a communicator. For example, rank 0 often represents the first, commonly the “master” process.

Task

Slurm terminology for a unit of work; usually maps to a process.

Node

A single physical or virtual machine in a cluster. A node can run multiple tasks or threads.

Core

A hardware CPU core; threads and processes are scheduled on cores.

Hybrid Parallelism

Using both multiple processes (MPI ranks) and multiple threads per process (OpenMP or threading libraries) simultaneously.

Multi-Threading

Multi-threading uses multiple threads within a single process. Threads share the same memory space, which makes communication fast but limits scalability across multiple nodes.

Many interpreted languages, like Python, are effectively multi-threaded only because of the Global Interpreter Lock (GIL), which prevents true parallel execution of threads in CPU-bound tasks. Python can use multiple threads for I/O-bound work efficiently, but CPU-bound tasks usually require multiprocessing or native extensions.

Slurm Example: Allocating Threads

#!/bin/bash
#SBATCH --job-name=threads_example
#SBATCH --output=threads_output.txt
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --time=00:10:00

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

# Run a multi-threaded program (e.g., OpenMP or Python using threads)
./my_multithreaded_program

  • --ntasks=1 indicates a single process.

  • --cpus-per-task specifies the number of threads.

  • OpenMP-aware programs or Python thread pools will use OMP_NUM_THREADS threads.

Multiprocessing

Multiprocessing uses multiple processes, each with its own memory space. This is ideal for CPU-bound tasks in Python, MPI programs, and distributed workloads. Multiprocessing scales across nodes if combined with MPI or other communication frameworks.

Slurm Example: Allocating Processes

#!/bin/bash
#SBATCH --job-name=mpi_example
#SBATCH --output=mpi_output.txt
#SBATCH --ntasks=16
#SBATCH --time=00:10:00

# Run an MPI program
srun ./my_mpi_program

  • --ntasks=16 launches 16 separate processes.

  • Each process can run on a different core or node.

Combining Threads and Processes

Some HPC applications use hybrid parallelism, combining multiple processes with multi-threading inside each process (common with MPI + OpenMP).

Slurm Example: Hybrid MPI + Threads

#!/bin/bash
#SBATCH --job-name=hybrid_example
#SBATCH --output=hybrid_output.txt
#SBATCH --ntasks=4
#SBATCH --cpus-per-task=8
#SBATCH --time=00:10:00

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

# Run a hybrid MPI + OpenMP program
srun ./my_hybrid_program

  • Launches 4 MPI processes.

  • Each MPI process spawns 8 threads for computation.

  • Total cores used = ntasks * cpus-per-task = 32.

Matching Hybrid Parallelism to NUMA

One of the biggest reasons to use hybrid parallelism over multi-threading or multiprocessing is its ability to perform well among several NUMA nodes, or “sockets”. Due to NUMA’s segregated memory hierarchy, both multi-threading and multiprocessing have flaws. Multi-threading only works well within a single NUMA node, and multiprocessing has many I/O operations saturating any communication bus (even intra-cpu busses). Hybrid parallelism decreases the I/O of multiprocessing by leveraging multi-threading per NUMA node. NUMA is explained in further detail in the Memory Divide section.

For the best performance, match cpus-per-task to the number of cores per NUMA node and ntasks to the number of NUMA nodes. You can also reference your cluster’s specifications and match your parameters to its recommended sbatch configuration.

Key Points

  • Use multi-threading for shared-memory, single-process parallelism (OpenMP, Python threads).

  • Use multiprocessing for CPU-bound tasks, true parallelism, or multi-node scaling (MPI, Python multiprocessing).

  • Hybrid models combine both for maximum performance on multi-core nodes.

  • Always match Slurm allocations (ntasks, cpus-per-task) to your program’s parallelization model to avoid over- or under-utilization.