GPU Troubleshooting

Common GPU Issues

1. No GPUs Available

Symptom: Job stuck in PENDING state

Check:

sinfo --partition=<valid_gpu_partition>
nvidia-smi  # on a GPU node

Solutions:

  • Wait for GPUs to become available

  • Try a different GPU partition

  • Request during off-peak hours

2. CUDA Out of Memory

Symptom: RuntimeError: CUDA out of memory

Solutions:

  • Reduce batch size

  • Clear cache

  • Spread work across multiple GPUs

3. GPU Not Visible

Symptom: torch.cuda.is_available() returns False

Check:

nvidia-smi
python -c "import torch; print(torch.cuda.device_count())"

Solutions:

  • Verify --gres=gpu:1 in job script

  • Check you’re on a GPU node

  • Reload CUDA module

4. Wrong GPU Type

Symptom: Job runs on wrong GPU or fails

Solution: Specify GPU type explicitly:

#SBATCH --gres=gpu:a100:1
#SBATCH --partition=gpu-a100

5. MIG Configuration Issues

Symptom: MIG jobs fail or show wrong memory

Check:

nvidia-smi -q | grep -A 5 "MIG"

Solution: Ensure MIG request matches available configurations