LSF Setup¶

Configure the pipeline for LSF (IBM Spectrum LSF) cluster execution.

Overview¶

The pipeline includes a pre-configured LSF profile at cluster/lsf/config.yaml that handles:

• Job submission via bsub
• Memory and queue allocation
• GPU job routing
• Resource limits

Quick Start¶

1
2
3
4
5

# Run with LSF profile
pixi run snakemake --profile cluster/lsf --configfile=config/config.yml

# Or use the test shortcut
pixi run test-lsf

Profile Configuration¶

Location¶

1

cluster/lsf/config.yaml

Full Configuration¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

executor: lsf
jobs: 300

default-resources:
  - 'mem_mb=8'
  - 'lsf_queue=rna'
  - 'lsf_project=aatrnaseq'
  - 'lsf_extra=""'

resources:
  - ngpu=12

set-resources:
  - rebasecall:lsf_queue="gpu"
  - rebasecall:lsf_extra="-gpu num=1:j_exclusive=yes"
  - rebasecall:ngpu=1
  - rebasecall:mem_mb=24
  - classify_charging:lsf_queue="gpu"
  - classify_charging:lsf_extra="-gpu num=1:j_exclusive=yes"
  - classify_charging:ngpu=1
  - classify_charging:mem_mb=24
  - remora_signal_stats:mem_mb=24
  - bwa_align:mem_mb=24
  - modkit_extract_calls:mem_mb=96
  - warpdemux:mem_mb=32
  - parse_warpdemux:mem_mb=8

printshellcmds: True
show-failed-logs: True
latency-wait: 15

Configuration Options¶

Global Settings¶

Default Resources¶

Applied to all rules unless overridden:

GPU Resource Limit¶

1
2

resources:
  - ngpu=12

Limits total concurrent GPU jobs to 12. Adjust based on your cluster's GPU availability.

Per-Rule Resources¶

GPU Rules¶

These rules are automatically submitted to the GPU queue:

Memory-Intensive Rules¶

Customization¶

Change Default Queue¶

Edit lsf_queue in default-resources:

1
2

default-resources:
  - 'lsf_queue=your_queue'

Change Project Tag¶

For job accounting:

1
2

default-resources:
  - 'lsf_project=your_project'

Adjust Max Jobs¶

Reduce if you're filling up the queue:

1

jobs: 100

Increase Memory for a Rule¶

Add or modify in set-resources:

1
2

set-resources:
  - your_rule:mem_mb=64

Change GPU Queue Name¶

If your GPU queue has a different name:

1
2
3

set-resources:
  - rebasecall:lsf_queue="your_gpu_queue"
  - classify_charging:lsf_queue="your_gpu_queue"

Monitoring Jobs¶

View All Jobs¶

1

bjobs -u $USER

View Job Details¶

1

bjobs -l <job_id>

View Job History¶

1

bhist -l <job_id>

Kill All Jobs¶

1

bkill 0  # Kill all your jobs

View Queue Status¶

1

bqueues

Submit Scripts¶

For long-running pipelines, use a submit script:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

#!/bin/bash
#BSUB -J aa-tRNA-seq
#BSUB -o logs/pipeline.%J.out
#BSUB -e logs/pipeline.%J.err
#BSUB -q rna
#BSUB -n 1
#BSUB -R "rusage[mem=4000]"

mkdir -p logs

pixi run snakemake --profile cluster/lsf \
    --configfile=config/config.yml

Submit:

1

bsub < run-analysis.sh

Troubleshooting¶

Jobs Pending Too Long¶

Check queue limits:

1

bqueues -l rna

Reduce concurrent jobs:

1

jobs: 50

Memory Errors¶

Increase memory for the failing rule:

1
2

set-resources:
  - failing_rule:mem_mb=128

GPU Jobs Not Starting¶

Check GPU queue availability:

1

bqueues -l gpu

Verify GPU resource syntax for your cluster.

File Sync Errors¶

Increase latency wait:

1

latency-wait: 60

Next Steps¶

• SLURM Setup - For SLURM clusters
• GPU Configuration - GPU-specific settings