escpod sidecar index¶

Build the .p5s sidecar caches for a POD5 file: the read index (O(log n) lookup by read ID instead of a full-table scan) and the signal batch geometry (the per-batch row counts of the signal table). The POD5 itself is never modified.

Both caches are rebuildable from the POD5 at any time, which is what separates sidecar index from sidecar annotate — annotations are data products that exist nowhere else, so sidecar annotate refuses to replace a sidecar it cannot read rather than rebuilding it, while sidecar index will always rebuild. It is the labels-free form of the same operation; see the sidecar overview.

Usage¶

escpod sidecar index [OPTIONS] <FILES>...

Arguments¶

Options¶

Examples¶

escpod sidecar index input.pod5          # index one file
escpod sidecar index *.pod5              # index all POD5 files
escpod sidecar index data_dir/           # index a directory recursively
escpod sidecar index input.pod5 --force  # rebuild an existing sidecar

What gets cached, and why¶

The read index¶

batch_idx / row_idx per read, sorted by read UUID. Without it, resolving a set of read IDs means scanning the reads table; with it, each lookup is a binary search. Loading the index from a sidecar rather than building it measured 249 ms → 59 ms on a 33 GB file.

A locator is trusted only as far as the read it names: every indexed lookup confirms the row it landed on actually holds the requested read_id before using it.

The signal batch geometry¶

Row count is the one field an Arrow IPC footer does not carry — a Block records a batch's offset, metadata length and body length and nothing else — so recovering the counts means reading every batch's own message header. That is one scattered touch per batch, measured at 15–24 ms each when cold on a network filesystem, and it is paid on every process start, not once.

A 33 GB file with 8866 signal batches spent 4.95 s there, 78% of a cold 5000-read scattered fetch. With the geometry cached that becomes 4.82 ms.

The counts are measured, not assumed. Reading batch 0 and extrapolating is what the official pod5 library and dorado do, and what escapepod's nonuniform_signal_batch diagnostic exists to catch them out on — recording the real counts costs about fifteen bytes (they are run-length encoded, so a conformant file is one run and a short tail) and is exact for a non-uniform file too.

When to run it¶

Once, on a networked node, before submitting jobs against a large file — the same reason escpod resquiggle models fetch exists. Every sidecar write records the geometry when it is not already there (escpod sidecar index, escpod sidecar annotate, demux --annotate alike), so a sidecar cannot end up carrying a read index and nothing else.

A sidecar that already holds both caches is skipped. One missing either — for example written by an older demux --annotate, which recorded the index but not the geometry — is completed in place, preserving annotations and scores. --force re-measures even when a value is present, because rebuilding the cache is the job this command exists to do.

Notes¶

• The sidecar is bound to its POD5 by file identifier and size, so a stale or misplaced one fails loudly rather than describing the wrong reads.
• A cached geometry is verified before use (one entry per batch the footer describes; first and last batches read for real and compared). Any mismatch logs a warning and falls back to the full walk — a stale cache costs time, never correctness.
• This format replaces the earlier .p5i index sidecar; delete any .p5i files and rerun escpod sidecar index.
• escpod index is the old spelling of this command. It still works, as a hidden alias that warns on stderr and forwards here, and will be removed in a later release.
• Format details: The .p5s Sidecar.