Job Management

Commands for submitting, monitoring, and controlling jobs.

Submitting Jobs

# Submit a batch job
sbatch job_script.sh

# Submit with options
sbatch --partition=debug job_script.sh

# Submit array job
sbatch --array=1-50 job_array.sh

Checking Job Status

# See all your jobs
squeue --user $USER

# See specific job
squeue --job 12345

# See all jobs on system
squeue

# Detailed view
squeue --user $USER --output "%.18J %.8P %.8T %.10M %.12L %.20F"

Job States

State	Meaning
PD	Pending (waiting for resources)
R	Running
CG	Completing
CD	Completed
F	Failed
TO	Timeout
CANCELLED	Cancelled

Canceling Jobs

# Cancel a specific job
scancel 12345

# Cancel all your jobs
scancel --user $USER

# Cancel jobs by name
scancel --name=myjob --user $USER

# Cancel array job (specific index)
scancel 12345_[3-5]

Tip

Escape or quote [ if using zsh, it will interpret it otherwise

Holding and Releasing Jobs

# Hold a job (prevent scheduling)
scontrol hold 12345

# Release a held job
scontrol release 13345

# Hold all your jobs
scontrol hold --user $USER

Updating Running Jobs

# Update job time limit
scontrol update jobid=12345 time=04:00:00

# Update job memory
scontrol update jobid=12345 memory=64G

# Update partition
scontrol update jobid=12345 partition=windfall

Viewing Job Output

# View output file
cat job.out

# Follow output in real-time
tail -f job.out

# View error file
cat job.err

# Check job accounting
sacct --job 12345 --output jobid,state,exitcode,start,end

Job History

# Recent jobs
sacct --user $USER --output jobid,state,start,end

# Jobs from today
sacct --user $USER --starttime today --output jobid,state

# Detailed accounting
sacct --job 12345 --long