CLI Reference¶

Complete reference for the leech command-line interface.

Command overview¶

The CLI is organized into workflow-based command groups:

leech --help       # Show all commands
leech --version    # Show version

Data commands¶

leech data prepare¶

Extract training chunks from POD5 and BAM files centered on a sequence motif.

leech data prepare --pod5 FILE --bam FILE --output-dir DIR [OPTIONS]

Required:

Feature extraction:

Motif search:

Every chunk also records junction_indel (the CIGAR-measured indel length at the motif junction, 0 when exact) and junction_mapped (whether the junction mapped at all) -- automatically, with no flag to enable them. Use leech model train --sample-weight-field junction_indel to over-sample disrupted-junction chunks, and leech predict --abstain-on-junction-indel to apply --min-margin only to reads whose junction is disrupted. leech eval test reports metrics split by junction_indel == 0 vs. != 0 automatically whenever the test corpus carries the field.

Signal handling:

Signal map refinement:

Splitting and parallelism:

Examples:

# Basic: prepare training data
leech data prepare \
  --pod5 reads.pod5 --bam alignments.bam \
  --output-dir chunks/ --label 1

# Parallel with reference-based motif search
leech data prepare \
  --pod5 reads.pod5 --bam alignments.bam \
  --output-dir chunks/ --workers 8 \
  --motif CCAGGC --motif-reference fasta --skip-motif-indels

# Prepare without splitting (for multi-sample merge later)
leech data prepare \
  --pod5 reads.pod5 --bam alignments.bam \
  --output-dir chunks/ --no-split --label 1

leech data merge¶

Merge chunk files from multiple samples and split at the read level to prevent data leakage.

leech data merge -i LABEL=FILE -i LABEL=FILE -o DIR [OPTIONS]

Examples:

# Pairwise comparison (e.g., amino acid identity)
leech data merge \
  -i Ala=ala.npz -i Gly=gly.npz \
  -o merged/

# Group comparison (chemical properties)
leech data merge \
  -i basic=lys.npz -i basic=arg.npz \
  -i acidic=asp.npz -i acidic=glu.npz \
  -o merged/

# 5-fold cross-validation
leech data merge \
  -i Ala=ala.npz -i Gly=gly.npz \
  -o kfold/ --k-fold 5

Model commands¶

leech model train¶

Train a model on prepared training data.

leech model train --train-data FILES --val-data FILES --model MODEL --output-dir DIR [OPTIONS]

Required:

Available architectures (26 total; production and experimental tiers — see leech.models.model_tier). ConvLSTMRemora/ConvLSTMRemoraBase exist for comparison against the Remora paper's architecture, and ConvOnly cannot reach ONNX export (--format onnx) and is deprecated (get_model("ConvOnly", ...) raises DeprecationWarning -- zero references in escapepod-models and no fix planned, see leech.models.model_tier); neither is listed here — see leech model train --help for the full registry including those.

Core training options:

Class balancing:

Regularization and optimization:

With --scheduler cosine, warmup is part of the schedule itself: the LR ramps linearly for --warmup-epochs and then follows a cosine decay down to a floor of 1e-6. Past the epoch budget the LR holds at the floor rather than cycling back up. reduce_on_plateau keeps its own warmup handling and is driven by --checkpoint-metric, not validation loss.

Optimizer loop:

Use --grad-accum-split to raise the effective batch size when a real batch will not fit in GPU memory: --batch-size 64 --grad-accum-split 4 costs the memory of 16 samples but takes the optimizer step of 64. Keep --batch-size divisible by N -- on a ragged final sub-batch, per-sample weighting is slightly off.

--quantile-grad-clip adapts the clipping threshold to the gradient scale the run is actually seeing, which is useful when a fixed --max-grad-norm is hard to pick. It takes precedence over --max-grad-norm if both are given.

--save-optim-every N trades resumability for checkpoint size. Note that it applies by epoch number, so model_last.pt carries optimizer state only if the final epoch is a multiple of N; resuming from a checkpoint without it starts from a fresh optimizer and logs a warning. It has no effect on model_resume.pt, which always carries optimizer state -- see below.

Loss function:

Data augmentation:

Provenance (recorded in config for inference):

Output files:

• model_best.pt -- best checkpoint by --checkpoint-metric (default auto: val_auc for binary, val_f1 for multiclass). Also accepts tpr_at_fpr:<f> (TPR at a fixed FPR) and callable_at_precision:<p> (fraction of validation reads callable at a precision floor) for a low-FPR operating regime -- binary tasks only; leech model optimize's --selection-metric takes the same names.
• model_last.pt -- final epoch checkpoint, written once after training ends (early stopping or completion)
• model_resume.pt -- rolling checkpoint, written atomically after every epoch and deleted on clean exit. This is the file that makes --resume recover a run an external kill (SLURM walltime, OOM, scancel) cut off mid-loop: model_last.pt never appears in that case, since it is only written when the epoch loop actually finishes. Point --resume at model_resume.pt in anything that retries a job -- a Snakemake restart-times, a resubmitted sbatch script. Refuses to resume (raises, does not warn) if the optimizer/scheduler/loss/auxiliary-head recipe or either corpus file changed since it was written; delete it to start over instead of warm-starting a different run under it. The check is scoped to what Trainer itself can vary -- a checkpoint's --motif, --batch-size, --seq-encoding etc. are provenance recorded in config.json, not enforced here, since a Trainer built directly (bypassing leech model train) never sets them. Do not add model_resume.pt to output: in a Snakemake rule -- Snakemake deletes a failed job's declared outputs, which would erase the one file the retry needs.
• config.json -- full training configuration (needed for inference)
• metrics.json -- per-epoch metrics

Examples:

# Basic training
leech model train \
  --train-data chunks/train.json --val-data chunks/val.json \
  --model ConvLSTMDwell --output-dir models/

# With focal loss, gradient clipping, and LR scheduling
leech model train \
  --train-data chunks/train.json --val-data chunks/val.json \
  --model ConvLSTMDwell --output-dir models/ \
  --loss focal --focal-gamma 2.0 \
  --max-grad-norm 1.0 \
  --scheduler reduce_on_plateau --scheduler-patience 5

# Recover an interrupted run (SLURM walltime, OOM, scancel, ...) -- safe to
# pass unconditionally in a retry, since --resume is ignored when the file
# doesn't exist (e.g. the first attempt)
leech model train \
  --train-data chunks/train.json --val-data chunks/val.json \
  --model ConvLSTMDwell --output-dir models/ \
  --resume models/model_resume.pt

# Extend a run that finished cleanly to more epochs
leech model train \
  --train-data chunks/train.json --val-data chunks/val.json \
  --model ConvLSTMDwell --output-dir models/ --epochs 80 \
  --resume models/model_last.pt

# Large effective batch on a small GPU, with adaptive clipping
leech model train \
  --train-data chunks/train.json --val-data chunks/val.json \
  --model ConvLSTMDwell --output-dir models/ \
  --batch-size 128 --grad-accum-split 4 \
  --quantile-grad-clip \
  --scheduler cosine --warmup-epochs 2

leech model optimize¶

Grid search over signal context windows and dwell offsets to find the best configuration.

leech model optimize --train-data FILE --output-dir DIR --context-grid VALUES [OPTIONS]

Required:

Grid options:

Training options (--model, --epochs, --batch-size, --learning-rate, --device, --seed, --early-stopping) work the same as in model train. So do the loss and augmentation options above --early-stopping in that reference (--loss, --label-noise-rate, --focal-gamma/--focal-neg-gamma, --scheduler*, --warmup-epochs, --weight-decay, --max-grad-norm, every --augment-* flag, --label-smoothing, --mixed-precision) and --motif/--motif-offset from provenance -- both decorators (training_hyperparams, model_provenance) are shared between train and optimize. --selection-metric accepts the same names as model train's --checkpoint-metric (see the "Output files" note under model train above: auto/val_acc/val_f1/val_auc, plus the parametric tpr_at_fpr:<f> and callable_at_precision:<p>). --feature-channels is also passed through unchanged to every grid point -- see model train's flag of the same name.

Output:

• grid_summary.csv -- results for all grid points
• best_params.json -- best configuration (for use with model train --model-config)
• Per-grid-point subdirectories with model checkpoints

Examples:

# Coarse symmetric grid
leech model optimize \
  --train-data chunks/train.npz --val-data chunks/val.npz \
  --context-grid 200,500,1000,2000,5000 \
  --output-dir grid_results/

# Asymmetric fine grid with range syntax
leech model optimize \
  --train-data chunks/train.npz --val-data chunks/val.npz \
  --left-contexts 8000:10000:500 \
  --right-contexts 0:2000:500 \
  --output-dir grid_results/

# With dwell offset search, parallel on CPU
leech model optimize \
  --train-data chunks/train.npz --val-data chunks/val.npz \
  --context-grid 500,1000 \
  --dwell-offsets -5:5:1 \
  --device cpu --parallel 8 \
  --output-dir grid_results/

leech model benchmark¶

Time one training step end to end -- per-phase timing plus GPU utilization -- to find out which phase is the bottleneck before changing anything.

leech model benchmark --train-data FILE --output-dir DIR [OPTIONS]

Example:

leech model benchmark \
  --train-data chunks/train.npz \
  --model TCNDwellResidual \
  --num-workers 8 \
  -o bench/

leech model bundle¶

Package multiple trained pairwise models into a single versioned bundle file for deployment.

leech model bundle --model-dir DIR --output FILE --version VERSION [OPTIONS]

The command auto-discovers all subdirectories containing model_best.pt and config.json.

Example:

# Bundle all pairwise models from a training run
leech model bundle \
  --model-dir results/models/pairwise/ \
  --output bundles/aa_classifier_v0.1.0.pt \
  --version 0.1.0

# One-vs-all bundle
leech model bundle \
  --model-dir results/models/one_vs_all/ \
  --output bundles/ova_v0.1.0.pt \
  --version 0.1.0 \
  --comparison-type one_vs_all

leech model bundle-info¶

Display metadata from a model bundle.

leech model bundle-info --bundle FILE

Shows architecture, version, comparison type, included pairs, and file size.

leech model calibrate¶

Learn post-hoc calibration on the validation set. Binary models use Platt scaling (fits a, b so sigmoid(a*logit + b) is better calibrated). Multiclass models use temperature (default), matrix, or Dirichlet scaling.

leech model calibrate --model-dir DIR --val-data FILE [OPTIONS]

Binary models write platt.json; multiclass models write calibration.json to the model directory. For a parent directory with pair subdirs, calibrates each pair independently.

Examples:

# Calibrate a single model
leech model calibrate --model-dir models/Ala_vs_Gly/ --val-data val.npz

# Calibrate all pairs in a directory
leech model calibrate --model-dir models/one_vs_all/ --val-data val.npz

leech model export¶

Export a trained model as a standalone .pt file, loadable with torch.export.load() without the leech codebase.

leech model export --model-dir DIR --output FILE

Example:

leech model export --model-dir models/best/ -o exported_model.pt

leech model release¶

Publish a model bundle to GitHub Releases from a YAML spec. Requires the GitHub CLI (gh), authenticated (gh auth login) unless --dry-run.

leech model release --spec FILE [OPTIONS]

The spec requires name, version, description and bundle (path to the .pt), and accepts organism, experiment, data_version, training, metrics, prerelease and extra_assets. The release tag is model-{name}-v{version}.

Example:

leech model release --spec release/ivc-pairwise.yaml --dry-run

leech model list¶

Browse published model releases on GitHub.

leech model list [OPTIONS]

leech model fetch¶

Download a published bundle from GitHub Releases.

leech model fetch --name NAME --version VERSION [OPTIONS]

Example:

leech model fetch --name ivc-pairwise --version 1.0.0 -o models/

Evaluation commands¶

leech eval test¶

Evaluate a trained model on a held-out test set.

leech eval test --model FILE --test-data FILES --output FILE [OPTIONS]

Inference command¶

leech predict¶

Run inference on new data. Supports single models, single pairs from a bundle, or all-model aggregation from a bundle.

leech predict --pod5 FILE --bam FILE --output FILE (--model DIR | --bundle FILE (--pair NAME | --all)) [OPTIONS]

Required:

Model selection (mutually exclusive):

Bundle options (require --bundle):

Signal handling:

Confidence and abstention:

Backend:

Examples:

# Single model inference
leech predict \
  --model models/model_best.pt \
  --pod5 reads.pod5 --bam alignments.bam \
  --output predictions.bam

# Run one pair from a bundle
leech predict \
  --bundle bundles/aa_classifier.pt --pair Ala_vs_Gly \
  --pod5 reads.pod5 --bam alignments.bam \
  --output predictions.bam

# Run all models in bundle (aggregated multi-class prediction)
leech predict \
  --bundle bundles/aa_classifier.pt --all \
  --pod5 reads.pod5 --bam alignments.bam \
  --output predictions.bam

# Same but also write per-pair raw probabilities
leech predict \
  --bundle bundles/aa_classifier.pt --all --raw \
  --pod5 reads.pod5 --bam alignments.bam \
  --output predictions.bam

Typical workflows¶

Single-sample classification¶

# 1. Prepare data
leech data prepare \
  --pod5 reads.pod5 --bam alignments.bam \
  --output-dir chunks/ --label 1

# 2. Train
leech model train \
  --train-data chunks/train.json --val-data chunks/val.json \
  --model ConvLSTMDwell --output-dir models/

# 3. Evaluate
leech eval test \
  --model models/model_best.pt \
  --test-data chunks/test.json --output metrics.json

# 4. Predict
leech predict \
  --model models/ \
  --pod5 new_reads.pod5 --bam new_alignments.bam \
  --output predictions.bam

Multi-sample pairwise comparison¶

# 1. Prepare each sample without splitting
leech data prepare --pod5 ala.pod5 --bam ala.bam --output-dir chunks/ala/ --no-split --label 0
leech data prepare --pod5 gly.pod5 --bam gly.bam --output-dir chunks/gly/ --no-split --label 1

# 2. Merge and split at read level
leech data merge -i Ala=chunks/ala/all.npz -i Gly=chunks/gly/all.npz -o merged/

# 3. Train, test, predict as above

Bundle workflow (multi-pair deployment)¶

# 1. Train pairwise models (one per class pair, e.g. amino acid pair)
leech model train --train-data ala_gly/train.json ... --output-dir models/Ala_vs_Gly/
leech model train --train-data ala_val/train.json ... --output-dir models/Ala_vs_Val/
# ... repeat for each pair

# 2. Bundle all models
leech model bundle \
  --model-dir models/ --output bundle.pt --version 1.0.0

# 3. Inspect the bundle
leech model bundle-info --bundle bundle.pt

# 4. Run aggregated inference
leech predict \
  --bundle bundle.pt --all \
  --pod5 reads.pod5 --bam alignments.bam \
  --output predictions.bam

Hyperparameter optimization¶

# 1. Grid search for optimal context
leech model optimize \
  --train-data chunks/train.npz --val-data chunks/val.npz \
  --context-grid 200,500,1000,2000,5000 \
  --output-dir grid_results/

# 2. Train final model with best params
leech model train \
  --train-data chunks/train.json --val-data chunks/val.json \
  --model ConvLSTMDwell --output-dir models/ \
  --model-config grid_results/best_params.json