GPU Configuration¶

Configure GPU resources for the aa-tRNA-seq pipeline.

GPU Requirements¶

Basecalling always needs GPU access, on both the standard and LDX/FDX demux paths; classify_charging can optionally use it too:

rebasecall benefits significantly from GPU acceleration. CPU-only execution is possible but substantially slower.

1

pixi run install-classify-gpu

GPU Resource Flow¶

flowchart LR
    subgraph GPURules[GPU Rules — always]
        A[rebasecall<br/>Dorado, per sample]
        A2[rebasecall_ldx_run<br/>Dorado, whole run]
        A3[escapepod_demux<br/>+ escapepod_demux_fdx]
    end

    subgraph GPUOpt[GPU Rule — opt-in]
        B[classify_charging<br/>escpod classify<br/>only if charging.gpu: true]
    end

    subgraph Resources
        C[POD5 Signal Data]
        D[CUDA GPU]
    end

    C --> A
    C --> A2
    C --> A3
    C --> B
    D --> A
    D --> A2
    D --> A3
    D -.-> B

Cluster Configuration¶

LSF GPU Settings¶

In cluster/lsf/config.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Limit total concurrent GPU jobs
resources:
  - ngpu=12

# GPU rule configuration
set-resources:
  - rebasecall:lsf_queue="gpu"
  - rebasecall:lsf_extra="-gpu num=1:j_exclusive=yes"
  - rebasecall:ngpu=1
  - rebasecall:mem_mb=24

classify_charging is not configured here — its GPU resources are resolved from charging.gpu inside the rule itself (see the note above).

SLURM GPU Settings¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

set-resources:
  rebasecall:
    slurm_partition: gpu
    slurm_account: gpu_rbi
    gres: "gpu:1"
  rebasecall_ldx_run:
    slurm_partition: gpu
    slurm_account: gpu_rbi
    gres: "gpu:2"
  escapepod_demux:
    slurm_partition: gpu
    slurm_account: gpu_rbi
    gres: "gpu:2"

classify_charging is not configured here — its GPU resources (slurm_partition, slurm_account, gres) are resolved from charging.gpu inside the rule itself (see the note above and workflow/rules/aatrnaseq-process.smk). Key names and current values live in cluster/slurm/config.yaml.

Configuration Options¶

GPU Concurrency Limit¶

Control how many GPU jobs run simultaneously:

1
2

resources:
  - ngpu=8  # Max 8 concurrent GPU jobs

Set this to match your available GPUs or queue limits.

CUDA Toolkit Version¶

The pipeline installs PyTorch with CUDA 12.4 support by default. To use a different CUDA version, set the CUDA_VERSION environment variable before activating the environment:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# For CUDA 11.8
export CUDA_VERSION=cu118
pixi shell

# For CUDA 12.1
export CUDA_VERSION=cu121
pixi shell

# For CPU-only (no CUDA)
export CUDA_VERSION=cpu
pixi shell

Available CUDA wheel tags: cu118, cu121, cu124, cpu

Exclusive GPU Access¶

Request exclusive GPU access to avoid memory conflicts:

1
2

set-resources:
  - rebasecall:lsf_extra="-gpu num=1:j_exclusive=yes"

1
2
3

set-resources:
  - rebasecall:gres="gpu:1"
  - rebasecall:slurm_extra="--exclusive"

GPU Type Selection¶

If your cluster has multiple GPU types:

1
2

set-resources:
  - rebasecall:lsf_extra="-gpu num=1:j_exclusive=yes:gtile='!gv100'"

1
2

set-resources:
  - rebasecall:gres="gpu:v100:1"

Local GPU Execution¶

CUDA_VISIBLE_DEVICES¶

The pipeline respects CUDA_VISIBLE_DEVICES:

1
2
3
4
5
6
7

# Use specific GPU
export CUDA_VISIBLE_DEVICES=0
pixi run snakemake --cores 4 --configfile=config/config.yml

# Use multiple GPUs (one per job)
export CUDA_VISIBLE_DEVICES=0,1
pixi run snakemake --cores 4 --resources gpu=2 --configfile=config/config.yml

Limit GPU Jobs Locally¶

1
2

pixi run snakemake --cores 8 --resources gpu=1 \
    --configfile=config/config.yml

Memory Requirements¶

GPU rules also require significant system memory:

Performance Considerations¶

Dorado (rebasecall)¶

• Processes POD5 signal data through neural network
• Throughput: ~100-500 reads/second depending on GPU
• Benefits from newer GPU architectures (Ampere, Ada Lovelace)

classify_charging (CPU)¶

• Analyzes signal at the CCA 3' end, anchored in reference coordinates
• Runs on the CPU under escpod classify; scales with --threads
• No GPU slot required

Troubleshooting¶

CUDA Out of Memory¶

Symptom:

1

RuntimeError: CUDA out of memory

Solutions:

1. Ensure exclusive GPU access:

   YAML
   1 2	set-resources: - rebasecall:lsf_extra="-gpu num=1:j_exclusive=yes"

2. Reduce concurrent GPU jobs:

   YAML
   1 2	resources: - ngpu=4 # Reduce from default

3. Check for other GPU processes:

   Bash
   1	nvidia-smi

GPU Not Detected¶

Symptom:

1

No CUDA GPUs are available

Solutions:

1. Verify CUDA installation:

   Bash
   1	nvidia-smi

2. Check CUDA_VISIBLE_DEVICES:

   Bash
   1	echo $CUDA_VISIBLE_DEVICES

3. Verify job is on GPU node:

   Bash
   1 2 3 4 5	# LSF bjobs -l <job_id> | grep -i gpu # SLURM scontrol show job <job_id> | grep -i gres

Wrong GPU Type¶

Symptom: Job runs on incompatible GPU.

Solutions:

Specify GPU type explicitly in cluster profile:

1
2

set-resources:
  - rebasecall:lsf_extra="-gpu num=1:j_exclusive=yes:gmodel=NVIDIAA100"

1
2

set-resources:
  - rebasecall:gres="gpu:a100:1"

Jobs Waiting for GPU¶

Symptom: GPU jobs pending indefinitely.

Solutions:

1. Check GPU queue status:

   Bash
   1 2 3 4 5	# LSF bqueues -l gpu # SLURM sinfo -p gpu

2. Reduce concurrent GPU jobs:

   YAML
   1 2	resources: - ngpu=2

3. Check fair share limits with your admin.

GPU Monitoring¶

NVIDIA SMI¶

Monitor GPU usage during execution:

1
2
3
4
5

# Watch GPU utilization
watch -n 1 nvidia-smi

# Log GPU stats
nvidia-smi --query-gpu=timestamp,name,utilization.gpu,utilization.memory,memory.used --format=csv -l 1 > gpu_log.csv

Check Running GPU Jobs¶

1

bjobs -u $USER -q gpu

1

squeue -u $USER -p gpu

CPU Fallback¶

If GPUs are unavailable, Dorado can run on CPU (much slower):

1
2
3

# Force CPU-only execution
export CUDA_VISIBLE_DEVICES=""
pixi run snakemake --cores 12 --configfile=config/config.yml

Next Steps¶

• LSF Setup - LSF cluster configuration
• SLURM Setup - SLURM cluster configuration