Data Preparation Modules¶

Iterate over BAM alignments with filtering.

Parameters:

Yields:

Read raw signal from a POD5 source for a specific read.

Parameters:

Returns:

Raises:

Examples:

>>> signal, meta = read_pod5_signal(Path("reads.pod5"), "read_001")
>>> print(f"Signal length: {len(signal)}")
>>> print(f"Sample rate: {meta['sample_rate']}")

The sequence chunks are cut from, and that focus bases index into.

Must track build_leech_read's choice exactly: under anchor="reference", with both a reference sequence and a CIGAR to map through, chunks come from the aligned reference slice; otherwise from the basecall. Motif positions are indices into this string, so handing the searcher the other one returns coordinates in the wrong frame.

Only observable with a BasecalledMotifSearcher -- ReferenceMotifSearcher ignores the sequence argument and reads the alignment instead -- which is why two of the three inference paths could pass the basecall under anchor="reference" without anyone noticing. That combination is reachable: predict selects the searcher with mode="fasta" if reference_sequences else "bam", so a run without a reference FASTA gets the basecalled searcher while chunks are still cut in reference coordinates.

Which bases of a read contribute chunks.

The single definition of that rule. Both prepare backends call it: the Python one from :func:extract_training_chunks with a built :class:LeechRead, the Rust one from leech.preparation.parallel._find_motif_positions with the ReadInfo it has not yet turned into a read. They used to carry a copy each, which drifted — the Rust copy fell back to all-bases over the query sequence while this one used the reference (issue #185).

Parameters:

Returns:

Extract all training chunks from a read, optionally filtered by motif.

Parameters:

Returns:

Save training chunks to numpy format.

Parameters:

Raises:

Examples:

>>> chunks = extract_training_chunks(read, motif="CCAGGC")
>>> save_chunks(chunks, Path("output/chunks.npz"))

Load training chunks from compressed numpy format.

Parameters:

Returns:

Examples:

>>> chunks = load_chunks(Path("output/chunks.npz"))
>>> print(f"Loaded {len(chunks)} chunks")
>>> for chunk in chunks[:5]:
...     print(f"{chunk['read_id']}: {chunk['label']}")

Shapes and dtypes of the row-streamable members of an .npz, without reading data.

Only the zip central directory and each member's .npy header are read, so this is O(number of members) regardless of file size. Members that :func:iter_npz_row_blocks cannot stream — object dtypes (pickled), Fortran order, 0-d — are omitted, so name in npz_array_members(path) is the test for "can I stream this".

Parameters:

Returns:

Examples:

>>> members = npz_array_members(Path("chunks.npz"))
>>> members["signals_flat"][0]
(6668328, 540)

Yield (row_start, {name: block}) over the rows of fixed-shape npz members.

np.load reads a whole member at once — it never memory-maps a zip member, compressed or not — so converting a large corpus means holding the numpy source and the converted output at the same time (#211). This walks the members as sequential row blocks instead, so only one block per member is resident.

Blocks are reused between iterations. The yielded arrays are views into buffers that the next iteration overwrites; copy anything you need to keep.

Parameters:

Yields:

Raises:

Load the base-to-signal maps in CSR form: (values, offsets).

Row i is values[offsets[i]:offsets[i + 1]]. Files written before v0.6.8 store these as a pickled object array (seq_to_sig_maps); those are converted here so callers see one representation.

Parameters:

Returns:

Prepare training data from BAM and POD5 files with statistics tracking.

Parameters:

Returns:

Prepare training data from BAM and POD5 files using multiprocessing.

Streams BAM reads in mega-batches so processing overlaps with BAM iteration rather than waiting for the entire BAM to be read first.

num_workers sets the number of batches in flight on either backend: threads for Rust, worker processes for the Python fallback. It is not advisory on either path.

Logs achieved reads/s as it goes, so a backend that is slower than the one it replaced shows up in the first progress line rather than at the end of the allocation.

Parameters:

Returns:

Split chunks into train/val/test sets at the READ level to prevent data leakage.

Groups chunks by read_id, then splits the read IDs into train/val/test sets. This ensures that no read appears in multiple splits, preventing the model from seeing similar signals from the same molecule during training and validation.

Parameters:

Returns:

Raises: