vLLM

vLLM is installed on HPC clusters for offline inference (batch mode) only. Use vLLM to process large batches of prompts on compute nodes for tasks like text classification, summarization, or data consolidation.

Important

vLLM is not for interactive use. Always run via SLURM batch jobs.

Cluster Requirements

Only certain clusters can provide adequate performance:

Cluster (Partition)

Accelerator

Supported

Orion

None

No

Hercules (v1)

None

No

Hercules (v2)

AMX (CPU)

Yes

Ptolemy

CUDA(GPU)

Yes

Atlas

CUDA (GPU)

Yes

Morrill

CUDA (GPU)

Yes

Warning

Do not run on Orion or Hercules v1 - they lack AMX instructions and will have extremely poor performance.

Loading vLLM

Use Lmod to load vLLM into your environment:

module load vllm

This makes the vLLM tools and the example setup script available.

User Workflow

Users should never call python directly. Run vLLM via SLURM:

Copy Examples

Use the setup script to copy example files to your project directory:

# After loading the module
vllm-setup-example.sh

# By default a new folder in your current working directory will
# be created and examples will be placed there. to place them
# elsewhere, provide a path
vllm-setup-example.sh /path/to/examples

Prepare Inputs

Edit inputs.json in the examples directory with your prompts:

Format: Each conversation is an array of messages with role and content:

[
  [
    {"role": "user", "content": "What is the capital of France?"}
  ],
  [
    {"role": "user", "content": "Explain quantum computing."}
  ]
]

Each outer array element is one conversation. Include message history for multi-turn chats using the role ‘assistant’ for the AI itself.

4. Submit Batch Job

sbatch vllm-offline.sbatch

SLURM Configuration

Edit vllm-offline.sbatch for your cluster and account:

#!/bin/bash
#SBATCH --account replaceme      # Your account
#SBATCH --partition hercules-2   # Hercules v2 nodes (AMX instructions)
#SBATCH --nodes 1
#SBATCH --exclusive
#SBATCH --mem 0
#SBATCH --time=02:00:00

# vLLM will run automatically

Output

Results are saved to results.json:

[
  {"text": "Your first prompt", "response": "Model output here"},
  {"text": "Your second prompt", "response": "Model response here"}
]

Best Practices

  • Use hercules-2 partition - AMX instructions provide 2-3x speedup

  • Batch your prompts - Process multiple prompts in one job for efficiency

  • Monitor job output - Check vllm-%j.out for progress and errors

  • Request exclusive access - Use --exclusive to avoid resource contention

  • Set appropriate time - Estimate ~1 minute per 100 prompts (varies by model)