Job Management

It can be helpful, from time to time, to query/modify the state of running and/or completed jobs. SLURM provides a rich CLI to do this.

Checking Job Status

# See all your jobs
squeue --user $USER

# See specific job
squeue --job 12345

# See all jobs on system
squeue

# Detailed view
squeue --user $USER --output "%.18J %.8P %.8T %.10M %.12L %.20F"

Job States

State	Meaning
PD	Pending (waiting for resources)
R	Running
CG	Completing
CD	Completed
F	Failed
TO	Timeout
CANCELLED	Cancelled

Canceling Jobs

# Cancel a specific job
scancel 12345

# Cancel all your jobs
scancel --user $USER

# Cancel jobs by name
scancel --name=myjob --user $USER

# Cancel array job (specific index)
scancel 12345_[3-5]

Tip

Escape or quote [ if using zsh, it will interpret it otherwise

Holding and Releasing Jobs

# Hold a job (prevent scheduling)
scontrol hold 12345

# Release a held job
scontrol release 13345

# Hold all your jobs
scontrol hold --user $USER

Updating Running Jobs

# Update job time limit
scontrol update jobid=12345 time=04:00:00

# Update job memory
scontrol update jobid=12345 memory=64G

# Update partition
scontrol update jobid=12345 partition=windfall

Job History

# Recent jobs
sacct --user $USER --output jobid,state,start,end

# Jobs from today
sacct --user $USER --starttime today --output jobid,state

# Detailed accounting
sacct --job 12345 --long