Job Scripts

Complete guide to writing SLURM job scripts, common job types, and advanced tips.

Basic Job Script Structure

SLURM job scripts contain directives (lines starting with #SBATCH) and commands to run your workload.

#!/bin/bash
# Lines starting with #SBATCH are SLURM directives
#SBATCH --job-name=myjob
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --time=01:00:00
#SBATCH --output=myjob.out

# Regular shell commands follow
module purge
module load python
echo "Running job..."
python my_script.py

Essential Directives

Job Identification

#SBATCH --job-name=myjob          # Job name (max 128 chars)
#SBATCH --output=output.out       # Standard output file
#SBATCH --error=error.err         # Standard error file
#SBATCH --mail-user=email@msu.edu # Email address
#SBATCH --mail-type=END,FAIL      # When to email (NONE, START, END, FAIL, ALL)

Resources

#SBATCH --nodes=1                 # Number of nodes
#SBATCH --ntasks=1                # Number of tasks (MPI processes)
#SBATCH --cpus-per-task=4         # CPUs per task
#SBATCH --mem=8G                  # Total memory
#SBATCH --mem-per-cpu=2G          # Memory per CPU
#SBATCH --time=01:00:00           # Wall time (HH:MM:SS)

Partition and QoS

#SBATCH --qos=normal              # Quality of Service
#SBATCH --account=myproject       # Account to charge

Complete Examples by Workload Type

CPU-Only

Most common type. Use for serial or multi-threaded CPU workloads.

#!/bin/bash
#SBATCH --job-name=cpu-job
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=16G
#SBATCH --time=02:00:00

module purge
module load python
python cpu_intensive.py

MPI (Multi-Node)

For distributed memory parallel computing.

#!/bin/bash
#SBATCH --job-name=mpi-job
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=16
#SBATCH --cpus-per-task=1
#SBATCH --mem=32G
#SBATCH --time=04:00:00

module purge
module load openmpi
mpirun -np 64 ./mpi_program

GPU

See GPU Usage for details.

#!/bin/bash
#SBATCH --job-name=gpu-job
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --gres=gpu:1
#SBATCH --mem=32G
#SBATCH --time=02:00:00
#SBATCH --partition=<valid_gpu_partition> 
module purge
module load cuda
python gpu_training.py

Memory-Intensive

#!/bin/bash
#SBATCH --job-name=big-mem
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=256G
#SBATCH --time=12:00:00
#SBATCH --partition=bigmem
#SBATCH --output=big-mem.out

module purge
module load python
python process_large_dataset.py

Job Arrays

Run many similar jobs efficiently without creating separate scripts.

#!/bin/bash
#SBATCH --job-name=array-job
#SBATCH --array=1-100
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=2
#SBATCH --mem=4G
#SBATCH --time=01:00:00

# $SLURM_ARRAY_TASK_ID ranges from 1 to 100
python process_sample.py --id $SLURM_ARRAY_TASK_ID

Key features:

• Each array task gets a unique ID ($SLURM_ARRAY_TASK_ID)

• Use %A for master job ID, %a for array task ID in output files

• Limit concurrent tasks: #SBATCH --array=1-100%10 (10 at a time)

Example with unique output files:

#SBATCH --output=output_%A_%a.txt  # Each task writes to separate file

Use cases:

• Parameter sweeps

• Batch simulations

• Processing multiple files in parallel

Interactive

Get a shell on a compute node for testing or debugging.

# Basic interactive session
srun --pty --time=02:00:00 /bin/bash

# With resources
srun --pty --nodes=1 --ntasks=1 --cpus-per-task=4 --mem=8G --time=02:00:00 /bin/bash

# Interactive GPU session
srun --pty --partition=<valid_gpu_partition> --gres=gpu:1 --time=04:00:00 /bin/bash

Short Burst

Quick tests or small computations.

#!/bin/bash
#SBATCH --job-name=burst
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --time=00:10:00    # 10 minutes

python quick_test.py

Cluster-Specific Job Examples

Different clusters have different partition names and resource options:

Orion

Standard CPU Job:

#!/bin/bash
#SBATCH --job-name=orion-cpu
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --time=04:00:00
#SBATCH --output=orion-job.out

module purge
module load python
python analysis.py

Big Memory Job:

#!/bin/bash
#SBATCH --job-name=orion-bigmem
#SBATCH --nodes=1
#SBATCH --mem=300G
#SBATCH --partition=bigmem
#SBATCH --time=04:00:00

python memory_intensive.py

Hercules

Standard CPU Job:

#!/bin/bash
#SBATCH --job-name=hercules-cpu
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --time=04:00:00
#SBATCH --output=hercules-job.out

module purge
module load python
python analysis.py

Ptolemy

GPU Job (A100):

#!/bin/bash
#SBATCH --job-name=ptolemy-gpu
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --gres=gpu:a100:1
#SBATCH --time=04:00:00
#SBATCH --partition=gpu-a100
#SBATCH --output=ptolemy-job.out

module purge
module load cuda pytorch
python train_model.py

MIG GPU Job:

#!/bin/bash
#SBATCH --job-name=ptolemy-mig
#SBATCH --nodes=1
#SBATCH --gres=gpu:mig:1g.10gb
#SBATCH --partition=gpu-a100-mig7
#SBATCH --time=04:00:00

python mig_job.py

Atlas

V100 GPU Job:

#!/bin/bash
#SBATCH --job-name=gpu-v100
#SBATCH --nodes=1
#SBATCH --gres=gpu:v100:1
#SBATCH --partition=gpu-v100
#SBATCH --time=04:00:00
#SBATCH --output=gpu-v100.out

module purge
module load cuda
python tensorflow_job.py

A100 GPU Job:

#!/bin/bash
#SBATCH --job-name=gpu-a100
#SBATCH --nodes=1
#SBATCH --gres=gpu:a100:1
#SBATCH --partition=gpu-a100
#SBATCH --time=04:00:00

module purge
module load cuda
python pytorch_job.py

L40S GPU Job:

#!/bin/bash
#SBATCH --job-name=gpu-l40s
#SBATCH --nodes=1
#SBATCH --gres=gpu:l40s:1
#SBATCH --partition=gpu-l40s
#SBATCH --time=04:00:00

python inference_job.py

Morrill

CPU Job:

#!/bin/bash
#SBATCH --job-name=morrill-cpu
#SBATCH --nodes=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G
#SBATCH --time=04:00:00
#SBATCH --output=morrill-cpu.out

python analysis.py

GPU Job (A100):

#!/bin/bash
#SBATCH --job-name=morrill-gpu
#SBATCH --nodes=1
#SBATCH --gres=gpu:a100:1
#SBATCH --partition=gpu-a100
#SBATCH --time=04:00:00
#SBATCH --output=morrill-gpu.out

module purge
module load cuda
python train_model.py

Choosing the Right Job Type

Workload Type	Job Type	Key Directives
Serial script	CPU-only	--cpus-per-task=1
Multi-threaded	CPU-only	--cpus-per-task=N
MPI parallel	Multi-node	--nodes, --ntasks-per-node
Deep learning	GPU	--gres=gpu:N
Many similar tasks	Array	--array=1-N
Testing/Debugging	Interactive	srun --pty

Best Practices

1. Always specify resources - Don’t rely on defaults

2. Use meaningful job names - Easier to identify in queue

3. Redirect output - Capture stdout and stderr with --output and --error

4. Load modules explicitly - Don’t rely on environment

5. Request only what you need - Helps scheduler be more efficient

6. Test with small jobs first - Verify before scaling up

7. Use job arrays for batch processing - More efficient than individual jobs

Common Mistakes

Wrong shebang

#!/bin/bash   # Correct
#!/bin/sh     # May not work for all scripts

Missing output redirection

#SBATCH --output=myjob.%j.out   # %j = job ID

Requesting too much memory

#SBATCH --mem=1000G    # Too much, will fail
#SBATCH --mem=128G     # Better

Wrong time format

#SBATCH --time=24:00:00   # Correct (24 hours)
#SBATCH --time=24:00      # Wrong format

Related Documentation

• Job Management - Controlling and monitoring jobs

• Interactive Sessions - Testing on compute nodes

• GPU Usage Guide - GPU-specific jobs

• Partition Reference - Cluster-specific partitions

• Job Arrays - Submitting many similar jobs efficiently

• Troubleshooting - Fix job problems