Job Failed

Your job exited unexpectedly? Here’s how to diagnose and fix common issues.

Check Job Status

# View recent job information
sacct --user $USER --output jobid,state,exitcode,start,end

Common Failure Causes

1. Out of Memory

Symptom: Job killed with “Out Of Memory” or exit code 137

Solution:

  • Request more memory: #SBATCH --mem=32G

  • Optimize your code to use less memory

  • Process data in smaller chunks

2. Time Limit Exceeded

Symptom: Job state = TIMEOUT

Solution:

  • Request more time: #SBATCH --time=24:00:00

  • Optimize your code for speed

  • Check if you can use checkpoint/restart

3. Wrong Partition/Queue

Symptom: Job rejected or stuck

Solution:

  • Check available partitions: sinfo

  • Use correct partition: #SBATCH --partition=<valid_partition>

  • Verify you have access to the requested partition

4. Missing Modules/Dependencies

Symptom: Command not found or library errors

Solution:

  • Load required modules in your script

  • Use module avail to check available modules

  • Verify paths are correct

5. Disk Space Issues

Symptom: Write failures, “No space left on device”

Solution:

  • Check disk usage: df -h

  • Clean up old files

  • Use appropriate filesystem (/work,/home, etc.)

Debugging Tips

  1. Run interactively first: Test your script with srun --pty bash

  2. Add echo statements: Track where your script fails

  3. Check exit codes: echo $? after commands

  4. Review output files: Look for error messages