Admin Guide¶

Notes for Bodhi HPC system administrators.

Scheduling maintenance¶

Bodhi undergoes scheduled maintenance on the last Thursday of every month. Use a SLURM maintenance reservation to prevent jobs from being scheduled across the maintenance window.

Create a maintenance reservation¶

scontrol create reservation \
  ReservationName=monthly-maint \
  StartTime=2026-04-30T06:00:00 \
  Duration=24:00:00 \
  Nodes=ALL \
  Flags=MAINT \
  User=root

• Flags=MAINT tells the backfill scheduler not to schedule jobs that would overlap with the reservation. Jobs already running that finish before the start time are unaffected.
• Users can see the upcoming reservation with scontrol show reservation.
• Jobs whose wall time would bleed into the reservation window won't start until after it ends.
• The reservation also drives the login-splash maintenance banner: once it's created, the countdown shown to users tracks the reservation's StartTime. With no MAINT reservation, the banner falls back to the last Thursday of the month, so the two stay in sync automatically.

Useful reservation flags¶

Delete the reservation after maintenance¶

scontrol delete reservation monthly-maint

Alternative: drain nodes¶

A more manual approach that doesn't give users advance visibility:

# Before maintenance
scontrol update NodeName=ALL State=DRAIN Reason="Scheduled maintenance"

# After maintenance
scontrol update NodeName=ALL State=RESUME

The reservation approach is preferred because it gives users visibility and lets the scheduler handle everything automatically.

Extending a running job's wall time¶

Users can only decrease --time on a running job. As root (or a SLURM operator), you can increase it with scontrol update:

# Absolute new limit
scontrol update JobId=<jobid> TimeLimit=2-00:00:00

# Or increment by a delta
scontrol update JobId=<jobid> TimeLimit=+12:00:00

Verify:

scontrol show job <jobid> | grep -E "RunTime|TimeLimit|EndTime"

For long-running orchestrators that will blow past the normal QoS's 3-day cap, root's scontrol update TimeLimit= call alone is enough — the QoS check is only enforced at submit/eval time, not while the job runs. Ideally you'd also switch the job to QOS=long for clean reporting, but note:

If you need to go past the 3-day cap that the normal QoS imposes on the CPU partitions, you do need a long-QoS'd job. In practice that means resubmitting with --qos=long, not patching a running job.

Caveats¶

Interactive partition¶

The interactive partition provides a dedicated queue for interactive work with shorter time limits and a per-user job cap to prevent monopolization.

slurm.conf¶

Add the following line to /etc/slurm/slurm.conf:

Live configuration (scontrol show partition interactive):

PartitionName=interactive Nodes=compute[04,06-07] Default=NO MaxTime=2-00:00:00 DefaultTime=08:00:00 State=UP AllowQos=ALL

Per-user job limit (QOS)¶

MaxJobsPerUser is not a valid slurm.conf partition parameter — enforce it via a QOS instead:

# Create the QOS with a per-user job limit
sacctmgr add qos interactive set MaxJobsPerUser=3

# Allow all accounts to use it
sacctmgr modify account where account=root withsubaccounts set qos+=interactive

The live interactive QoS now has MaxJobsPerUser=4 but MaxSubmitJobsPerUser=3. Since the submit limit counts pending and running jobs, 3 is the effective ceiling and the 4 is unreachable — worth reconciling.

Apply and verify¶

scontrol reconfigure
scontrol show partition interactive
sacctmgr show qos interactive format=Name,MaxJobsPerUser

GPU partition¶

The gpu partition fronts Bodhi's GPU nodes (compgpu01–compgpu03 — 64 CPUs + 4 × NVIDIA A30 each). User-facing documentation lives at GPU Jobs; this section covers the admin-side configuration.

slurm.conf¶

Live definition (slurm.conf line ~190):

PartitionName=gpu Nodes=compgpu01,compgpu02,compgpu03 Default=NO State=UP \
    DefaultTime=12:00:00 MaxTime=3-00:00:00 \
    AllowAccounts=gpu_rbi,gpu_devbio,gpu_scb \
    AllowQOS=normal,high,gpu_long QOS=gpu_shared \
    DefMemPerNode=12000 \
    DefCpuPerGPU=16 \
    GraceTime=120 PriorityTier=1 DisableRootJobs=YES

GPU QoS reference¶

sacctmgr -i modify user where account=gpu_rbi user=<user1>,<user2> set qos+=gpu_long

Apply and verify¶

scontrol reconfigure
scontrol show partition gpu | grep -E "AllowAccounts|DefCpuPerGPU|MaxCPUsPerNode|DefaultTime|DefMemPerNode"

Granting a new group access¶

Three steps, in order:

1. Create the Slurm account (see Per-account GPU limits).
2. Add the account to AllowAccounts in slurm.conf, then scontrol reconfigure.
3. Tell users to submit with -p gpu -A <account> and --gres=gpu:N.

Per-account GPU limits¶

Use a dedicated Slurm account to grant a group of users access to the gpu partition with a shared GPU cap. This is cleaner than per-user limits when several users should share a quota, and it keeps the policy in one place.

Pattern: shared 1-GPU pool for a small group¶

# 1. Create the account
sacctmgr add account gpu_devbio \
  Description="GPU access for devbio group" \
  Organization=devbio

# 2. Cap the account at 1 concurrent GPU (applies to all members, shared pool)
sacctmgr modify account gpu_devbio set GrpTRES=gres/gpu=1

# 3. Add a user to the account
sacctmgr add user gibsonty account=gpu_devbio

Users submit with the account flag:

srun -p gpu -A gpu_devbio --gres=gpu:1 --pty bash
sbatch -p gpu -A gpu_devbio --gres=gpu:1 job.sh

Notes¶

• GrpTRES on the account is a shared pool across all its users. Use MaxTRESPerUser=gres/gpu=N if you also want a per-user ceiling within the pool.
• Set the limit on the account association (no where partition=... clause). sacctmgr modify ... where partition=gpu only matches existing partition-scoped association rows, which don't exist until you create them explicitly — so without that scope the cap lands on the account's root association and is inherited by members everywhere they use GPUs.
• Inherited limits do not re-display on child (user) rows in sacctmgr show assoc; they are still enforced at schedule time.
• Default account is unaffected — users keep their existing DefaultAccount and must pass -A gpu_devbio to hit this quota.

Verify¶

sacctmgr show assoc account=gpu_devbio format=Account,User,Partition,GrpTRES
sacctmgr show user gibsonty withassoc format=User,Account,DefaultAccount,Partition