Python

Python is available on all ARC clusters via environment modules. The base Python installation includes only the interpreter—you must install packages in a virtual environment.

Loading Python

# Load Python module (base interpreter only)
module load python

# Check version
python --version

Creating a Virtual Environment

This is required to install any packages beyond the base interpreter:

# View available versions
module avail python

# Load Python first
module load python

# Create virtual environment in your PROJECT directory (not home directory)
# Home directories have limited quotas (~10GB)
# Project directories have higher storage limits
python -m venv /path/to/project/venvs/myenv

# Activate it
source /path/to/project/venvs/myenv/bin/activate

# Install packages
pip install numpy pandas matplotlib scikit-learn

# Verify installation
pip list

# Deactivate when done
deactivate

Note

Virtual environments are persistent and do not need to be re-created. simply source the activate script again whenever you wish to use them.

Python uv

Due do a number of short-comings with pip and venv’s, the industry has shifted toward using uv to manage virtual environments and packages. Usage is very straight-forward.

# install uv if it's not already
pip install --user uv

# creat a virtual environment
# --seed provides a pip if you do not wish to manage packages with uv
# --python specifies which version you'd like to us
uv venv --seed --python 3.12 /path/to/venv
# if no path is specified, uv will pace the venv in $PWD/.venv

# activate it
source /path/to/venv/bin/activate

# install packages
uv pip install a b c
# or
pip install a b c

uv pip install is preferred as uv’s dependency solver is superior, pip install is maintained for compatibility with projects

Running Python Jobs

Interactive Session

srun --pty python

Batch Job

#!/bin/bash
#SBATCH --job-name=python-job
#SBATCH --output=python_output.txt
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --time=01:00:00
#SBATCH --partition=short

module purge
module load python
source ~/venvs/myenv/bin/activate

python my_script.py

Best Practices

  • Always use virtual environments - The base Python has no packages installed

  • Create environments in project directories - Avoid home directory quota limits

  • Keep venvs with your code - Easier to manage and backup

  • Activate before every session - Source the venv in job scripts and interactive sessions

  • Use pip freeze - Document your environment: pip freeze > requirements.txt

  • Memory limits - Large data processing may require more memory: #SBATCH --mem=32G

Example: Data Analysis Job

#!/bin/bash
#SBATCH --job-name=data-analysis
#SBATCH --output=analysis_output.txt
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=16G
#SBATCH --time=02:00:00

module purge
module load python
source /path/to/project/venvs/data-science/bin/activate

python analyze_data.py

Common Packages

Install these in your virtual environment as needed:

Category

Packages

Data Science

numpy, pandas, scipy

Visualization

matplotlib, seaborn, plotly

Machine Learning

scikit-learn, xgboost, lightgbm

Deep Learning

torch, tensorflow, keras

Utilities

h5py, netCDF4, requests

 
pip install numpy pandas scikit-learn matplotlib

Troubleshooting

“No module named X”

You need to install the package in your virtual environment:

source ~/venvs/myenv/bin/activate
pip install <package-name>

Virtual Environment Not Working

Make sure you:

  1. Loaded the Python module first: module load python

  2. Created the venv with that Python in a project directory: python -m venv /path/to/project/venvs/myenv

  3. Activated it: source /path/to/project/venvs/myenv/bin/activate

pip Install Fails

  • Use --user flag if not in a venv (not recommended): pip install --user <package>

  • Check disk space: df -h

  • Try a different package index: pip install --index-url https://pypi.org/simple <package>