Common Pain Points¶

This page covers recurring issues that Bodhi users encounter when migrating from LSF to SLURM. These aren't simple directive swaps — they're behavioral differences that catch people off guard.

Debugging OOM (Out-of-Memory) errors¶

How OOM kills look in SLURM¶

When a job exceeds its memory allocation, SLURM kills it immediately. The job state is set to OUT_OF_MEMORY:

$ sacct -j 12345 --format=JobID,JobName,State,ExitCode,MaxRSS
JobID           JobName      State ExitCode     MaxRSS
------------ ---------- ---------- -------- ----------
12345          analysis OUT_OF_ME+      0:125
12345.batch       batch OUT_OF_ME+      0:125    15.8G

You can also see this with seff:

$ seff 12345
Job ID: 12345
State: OUT_OF_MEMORY (exit code 0)
Memory Utilized: 15.80 GB
Memory Efficiency: 98.75% of 16.00 GB

Diagnosing memory usage¶

For completed jobs, use sacct:

# Check peak memory usage
sacct -j <jobid> --format=JobID,JobName,MaxRSS,MaxVMSize,State

# For array jobs, check all tasks
sacct -j <jobid> --format=JobID%20,JobName,MaxRSS,State

For running jobs, use sstat:

# Monitor memory of a running job
sstat -j <jobid> --format=JobID,MaxRSS,MaxVMSize

Fixing OOM errors¶

1. Check what your job actually used — run seff <jobid> on a similar completed job to see actual peak memory.

2. Request more memory with headroom — add 20–30% buffer above the observed peak:

   #SBATCH --mem=20G   # if your job peaked at ~15 GB
   

3. Use --mem-per-cpu for multi-threaded jobs — if your job scales memory with cores:

   #SBATCH --cpus-per-task=8
   #SBATCH --mem-per-cpu=4G   # 32 GB total
   

Tasks vs CPUs: --ntasks vs --cpus-per-task¶

LSF's -n N meant one thing: "give me N cores." SLURM splits that same idea into two separate flags, and picking the wrong one is one of the most common migration mistakes on Bodhi.

The framing difference¶

SLURM distinguishes between processes and threads within a process:

• --ntasks=N — how many independent processes your job will launch (think MPI ranks, or N separate commands)
• --cpus-per-task=N — how many threads/cores each of those processes will use

The total cores allocated to your job is ntasks × cpus-per-task. Both flags default to 1, so a bare sbatch with no CPU flags gets you exactly one core.

Which flag do I use?¶

Behavioral consequences¶

What srun does with each flag. srun cmd with --ntasks=N launches N copies of cmd in parallel. srun cmd with --cpus-per-task=N --ntasks=1 launches one copy of cmd and hands it N cores — cmd itself is responsible for spawning threads to use them.

What environment variable your tool sees. SLURM exports $SLURM_NTASKS and $SLURM_CPUS_PER_TASK. A common idiom for threaded tools:

#SBATCH --cpus-per-task=8
#SBATCH --mem=16G

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
samtools sort -@ $SLURM_CPUS_PER_TASK input.bam -o sorted.bam

Node placement. --ntasks tasks may be spread across multiple nodes unless you also set --nodes=1. --cpus-per-task cores are always on the same node — a single process can't span nodes.

Memory interaction¶

--mem is per-job (total). --mem-per-cpu multiplies by the total cores (ntasks × cpus-per-task):

#SBATCH --ntasks=4
#SBATCH --cpus-per-task=8
#SBATCH --mem-per-cpu=2G    # 4 × 8 × 2G = 64 GB total

Understanding SLURM accounts¶

What is --account?¶

In SLURM, the --account flag associates your job with a resource allocation account. This is used for:

• Fair-share scheduling — accounts that have used fewer resources recently get higher priority
• Resource tracking — PIs and admins can see how allocations are consumed
• Access control — some partitions may be restricted to certain accounts

Finding your account(s)¶

# List your SLURM associations (accounts and partitions you can use)
sacctmgr show associations user=$USER format=Account,Partition,QOS

# Shorter version — just account names
sacctmgr show associations user=$USER format=Account --noheader | sort -u

Bodhi accounts are lab/group-based. Each account corresponds to a research group or resource class:

Most users are associated with their PI's lab account. You may belong to multiple accounts (e.g., rbi for CPU jobs and gpu_rbi for GPU jobs).

Setting a default account¶

Rather than adding --account to every script, set a default:

# Set your default account (persists across sessions)
sacctmgr modify user $USER set DefaultAccount=<your_account>

You can also add it to your ~/.bashrc or a SLURM defaults file:

# In ~/.bashrc
export SBATCH_ACCOUNT=<your_account>
export SRUN_ACCOUNT=<your_account>

sacctmgr show user $USER format=DefaultAccount

In your job scripts¶

#SBATCH --account=<your_account>

Paying attention to wall time¶

SLURM enforces --time strictly¶

In SLURM, the --time (wall time) limit is a hard cutoff. When your job hits the limit:

1. SLURM sends SIGTERM to your job (giving it a chance to clean up)
2. After a 30-second grace period (KillWait=30), SLURM sends SIGKILL
3. The job state is set to TIMEOUT

$ sacct -j 12345 --format=JobID,JobName,Elapsed,Timelimit,State
JobID           JobName    Elapsed  Timelimit      State
------------ ---------- ---------- ---------- ----------
12345          longrun   02:00:00   02:00:00    TIMEOUT

Checking remaining time¶

From outside the job:

# See time limit and elapsed time
squeue -u $USER -o "%.10i %.20j %.10M %.10l %.6D %R"
#                              Elapsed ^  ^ Limit

# Detailed view
scontrol show job <jobid> | grep -E "RunTime|TimeLimit"

From inside the job (in your script):

# Remaining time in seconds — useful for checkpointing
squeue -j $SLURM_JOB_ID -h -o "%L"

Consequences of TIMEOUT¶

• Your job output may be incomplete or corrupted
• Any files being written at kill time may be truncated
• Temporary files won't be cleaned up

cleanup() {
    echo "Job hit time limit — cleaning up"
    # save checkpoint, remove temp files, etc.
}
trap cleanup SIGTERM

Bodhi partition time limits¶

#SBATCH --time=8:00:00       # 8 hours
#SBATCH --time=1-00:00:00    # 1 day

#SBATCH --qos=long
#SBATCH --time=5-00:00:00    # 5 days

sinfo -o "%12P %10l %10L %6D %8c %10m"
#            Name  TimeLimit  DefTime  Nodes  CPUs  Memory

Tips for setting wall time¶

1. Start with a generous estimate, then refine based on actual runtimes using seff or sacct.

2. Shorter jobs schedule faster — SLURM's backfill scheduler can fit shorter jobs into gaps. Requesting 2 hours instead of 7 days can dramatically reduce queue wait time.

3. Use sacct to check past runtimes:

   sacct -u $USER --format=JobID,JobName,Elapsed,State -S 2024-01-01 | grep COMPLETED
   

4. SLURM format for --time:

   Format	Meaning
   MM	Minutes
   HH:MM:SS	Hours, minutes, seconds
   D-HH:MM:SS	Days, hours, minutes, seconds
   D-HH	Days and hours

   #SBATCH --time=04:00:00      # 4 hours
   #SBATCH --time=1-00:00:00    # 1 day
   #SBATCH --time=7-00:00:00    # 7 days
   

QoS tiers¶

SLURM Quality of Service (QoS) tiers control job priority, time limits, and resource caps. Every job runs under a QoS — if you don't specify one, you get normal.

Available QoS tiers¶

Using a QoS¶

#SBATCH --qos=long
#SBATCH --time=3-00:00:00

Or on the command line:

sbatch --qos=long myjob.sh

When to use long¶

The normal QoS has a 1-day time limit. If your job needs more time, use long:

#SBATCH --qos=long
#SBATCH --time=5-00:00:00    # up to 7 days

long has OverPartQOS — it overrides partition time limits, so you can run >1 day jobs on any partition. The tradeoff: you're limited to 5 running jobs and 32 CPUs total under long.

When to use low¶

For jobs that can wait — overnight runs, large batch submissions where turnaround isn't critical:

#SBATCH --qos=low
#SBATCH --time=3-00:00:00

low allows up to 7 days and 50 concurrent jobs, but at reduced priority.

DenyOnLimit¶

All QoS tiers enforce DenyOnLimit — if your job exceeds the QoS limits (too many running jobs, too many CPUs, etc.), it is rejected at submit time rather than sitting in the queue forever. You'll see an immediate error telling you what limit was hit.

# See which QoS tiers you can use
sacctmgr show associations user=$USER format=Account,QOS

# See all QoS definitions
sacctmgr show qos format=Name,Priority,MaxWall,MaxJobsPU,MaxSubmitPU,MaxTRESPU,Flags