GPU Jobs¶

The gpu partition provides access to Bodhi's GPU nodes. This page covers the hardware, how to submit jobs, and the per-partition limits you need to work within.

Hardware¶

Total: 3 nodes, 192 CPUs, 12 A30 GPUs.

Check the live state anytime:

sinfo -p gpu -o "%N %c %G %t"
scontrol show node compgpu01 | grep -E "CPUTot|Gres|RealMemory"

Partition settings¶

How to submit¶

Interactive GPU session (srun)¶

srun -p gpu -A <your_account> \
     --gres=gpu:1 \
     --cpus-per-task=4 \
     --mem=32G \
     --time=04:00:00 \
     --pty bash

Inside the shell, confirm the GPU is visible:

nvidia-smi
echo "CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES"

Batch job (sbatch)¶

#!/bin/bash
#SBATCH --job-name=gpu_train
#SBATCH --partition=gpu
#SBATCH --account=<your_account>
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=4
#SBATCH --mem=32G
#SBATCH --time=12:00:00
#SBATCH --output=logs/gpu_train.%j.out
#SBATCH --error=logs/gpu_train.%j.err

module load cuda/12.2
source ~/venvs/torch/bin/activate

nvidia-smi
python train.py --data /data/input --out /data/output

Submit:

mkdir -p logs
sbatch gpu_job.sh

Choosing a QOS¶

Most jobs need nothing here — the default (normal) covers anything up to 1 day. Reach for gpu_long only when a single run genuinely needs more than that.

Multi-day runs (--qos=gpu_long)¶

For a training run or basecalling job that needs more than 24 hours:

sbatch --qos=gpu_long --time=3-00:00:00 -p gpu -A <your_account> --gres=gpu:1 gpu_job.sh

gpu_long is deliberately throttled, because a multi-day job holds a shared GPU for a long time:

• One at a time. One running job holding one GPU. A second gpu_long submission is rejected while the first runs.
• Capped fleet-wide. All gpu_long jobs together can hold at most 4 of the 12 GPUs, so long runs can never crowd out short and interactive work.
• Costs double fairshare. Usage is charged at 2×, which lowers the priority of all your later jobs (CPU and GPU) for roughly a week. Prefer checkpointing and a series of shorter jobs where your software supports it.
• Lower queue priority than normal, so it yields to normal-length work.

Check whether you have it¶

List the QoS you hold on each of your accounts:

sacctmgr show assoc user=$USER format=Account,QOS%45

Look at the row for the account you submit GPU jobs with (the one you pass to -A). gpu_long has to appear on that row. Grants are per account, so it is entirely possible to hold it on one and not another:

   Account                                           QOS
---------- ---------------------------------------------
   gpu_rbi                              high,long,normal     <- no gpu_long: -A gpu_rbi will fail
   gpu_scb           gpu_long,interactive,long,normal        <- has it: -A gpu_scb works

Also confirm you're using the right account in the first place — your default account is often not your GPU account, so -A is usually required. See Requesting access.

Confirm without burning a submission¶

--test-only validates your request against every limit and prints the verdict without queueing anything:

sbatch --test-only -p gpu -A <your_gpu_account> --qos=gpu_long \
       --gres=gpu:1 --time=3-00:00:00 --wrap='true'

Getting it enabled¶

Ask an admin, and include your username and the GPU account you submit with — the grant is specific to that pair. Point them at the admin guide, which has the exact sacctmgr command and a note on why an account-level grant alone often misses people.

Requesting more than one GPU¶

Two GPUs on a single node, with CPUs scaled automatically via DefCpuPerGPU:

sbatch -p gpu -A <your_account> --gres=gpu:2 gpu_job.sh
# gets 2 GPUs + 32 CPUs by default

All GPUs on Bodhi are currently NVIDIA A30s, so --gres=gpu:N is sufficient — there is no need to name a specific model.

Limits to keep in mind¶

• You get 16 CPUs per GPU by default, via DefCpuPerGPU. This is a default, not a cap — --cpus-per-task=32 (or more) is accepted. Bear in mind that the more CPUs you ask for, the longer you wait for a node with that many free.
• Memory default is low (12 GB). Always set --mem explicitly for real workloads.
• Wall-time is capped at 1 day unless you opt into --qos=gpu_long for up to 3 days.
• Per-account GPU caps exist. gpu_devbio is limited to 1 concurrent GPU for the whole account; gpu_rbi and gpu_scb are uncapped at the account level. If your job is stuck in PENDING with reason QOSGrpGRES or AssocGrpGRES, someone on your account is already holding the group's GPUs.

Requesting access¶

GPU access is granted through a Slurm account. If you don't yet have one:

1. Contact an administrator to request access. Provide your username and a short description of the workload.
2. The admin will add you to an existing GPU account (e.g., gpu_rbi) or create a new one for your group.
3. Once added, pass -A <account_name> on every GPU submission.

You can list the accounts you belong to with:

sacctmgr show assoc user=$USER format=User,Account,DefaultAccount,Partition

Admins: see GPU partition configuration for provisioning details.

Monitoring your GPU jobs¶

# Your pending/running jobs
squeue -u $USER

# All jobs currently charging a given account
squeue -A <account_name>

# Historical usage with allocated GPUs
sacct -u $USER -X --format=JobID,JobName,Partition,Account,AllocTRES%40,State,Elapsed

# Live GPU utilization on the node your job landed on
srun --jobid=<jobid> --pty nvidia-smi

After a job ends, seff <jobid> summarizes CPU and memory efficiency (GPU efficiency is not reported there — use sacct with AllocTRES and your own training-time metrics).

Troubleshooting¶