fqxv compress¶

Compress one or more FASTQ files into a single .fqxv archive.

Usage¶

fqxv compress <INPUTS>... [-o <OUTPUT>] [OPTIONS]

Give one input for single-end, two for paired-end, or three/four for single-cell (R1/R2/I1[/I2]). Each input's slot label (the R1/I1/2 token before its extension) is recorded in the archive when every input has a distinct one, so decompress --split can restore the original per-slot names rather than renumbering positionally. Multiple inputs are interleaved per spot into one archive and can be split back out with decompress --split. Input order is preserved for the split.

Arguments¶

Options¶

Advanced options¶

Examples¶

# single-end, gzipped input (-o defaults to reads.fqxv)
fqxv compress reads.fastq.gz

# paired-end at higher effort
fqxv compress sample_R1.fq.gz sample_R2.fq.gz -o sample.fqxv --level 7 --threads 16

# 10x single-cell (R1 + R2 + I1 + I2)
fqxv compress R1.fq R2.fq I1.fq I2.fq -o sample.fqxv

# lossy quality (Illumina 8-level binning)
fqxv compress reads.fastq -o reads.fqxv --quality-bin bin8

# lossy quality on long reads (match the table to the platform)
fqxv compress ont_reads.fastq -o ont_reads.fqxv --quality-bin ont
fqxv compress hifi_reads.fastq -o hifi_reads.fqxv --quality-bin hifi

# finer row groups for remote/parallel access (smaller range fetches)
fqxv compress reads.fastq.gz -o reads.fqxv --block-reads 65536

# maximum compression (deepest context + read reordering where it helps)
fqxv compress reads.fastq.gz --max

# verify the archive round-trips before trusting (or deleting) the source
fqxv compress reads.fastq.gz --verify

# estimate the ratio and archive size from a sample, writing nothing
fqxv compress reads.fastq.gz --estimate

# ...the same estimate as a two-line table for scripts
fqxv compress reads.fastq.gz --estimate tsv

Verifying on write¶

--verify closes the window between writing an archive and trusting it. After the archive is written, it is reopened and fully decoded — exercising every block CRC-32C and content digest — and the decoded read count is checked against what was written, all before compress reports success:

fqxv compress reads.fastq.gz --verify

This catches a codec or in-flight memory error that produced a CRC-valid but wrong archive, which the archive's own on-disk checksums cannot detect on their own. It is the check to run before deleting the source FASTQ.

• On any failure — a decode error or a read-count mismatch — the archive is left in place for inspection and the command exits non-zero. A bad archive is never reported as done.
• It adds a full decode pass, so expect roughly double the wall time (the --max / reorder path is the heaviest).
• It validates the bytes the encoder emitted, not long-term on-disk durability; for the latter, re-run fqxv verify against the stored file later.

Estimating compression¶

--estimate tells you how well an input will compress before committing to a full run. It takes a bounded sample of the leading reads (up to 300,000 reads, or one block's worth of bases — whichever binds first, which on long reads is the bases) and predicts the archive without coding it: the real codecs are entropy coders, so a histogram pass measures directly what they converge to — static order-k sequence entropy (blended with a k-mer duplication sketch on the long-read path), order-1 quality entropy, and the real tokenizer for names, each scaled by a small per-platform calibration factor. It then projects the whole-file archive at the chosen --level/--quality-bin and prints the result — no archive is written:

reads.fastq.gz (59.99 MB)  →  estimated fqxv ~37.58 MB  (37% smaller, ~1.60x)

Estimated from a 300,000-read sample (102.26 MB uncompressed FASTQ):
  stream      compressed     share   rate
  names        11.27 KB     0.0%   0.31 bits/read
  sequence      8.92 MB    31.6%   1.663 bits/base
  quality      19.26 MB    68.3%   3.590 bits/base
  vs uncompressed FASTQ (~136.32 MB):  ~3.63x

The top line leads with the outcome: input size(s) → estimated archive size, as a percent reduction and ratio against the input on disk. The per-stream table below shows where the bytes go (names / sequence / quality) for the sample, and the final line gives the ratio against the whole file's uncompressed FASTQ bytes. Multiple inputs are listed individually under a combined headline; a streaming stdin input has no on-disk size, so it reports the sample's own reduction and omits the whole-file projection (pass a file to get one).

For scripting, --estimate tsv prints the same projection as a header line plus one data row:

file    input_bytes est_fqxv_bytes  ratio
reads.fastq.gz  62898962    39405273    1.5962

Accuracy. Because blocks are coded independently and the models are per-block stationary, the sample's ratio is a faithful proxy for the whole file — in practice within ~1–2% of a real run (every projected figure is marked ~). The projection scales the sample by the fraction of on-disk bytes it consumed, so it works for plain and gzipped inputs alike.

Reordering is a lower bound. --order any / --max reordering is not modeled — its cross-read redundancy grows with read count, so a small sample can't capture it. With those flags the estimate is a conservative lower bound (the real archive comes out that size or smaller), and the report says so.

Row-group sizing¶

The archive is a run of independently-coded row groups (blocks); the row group is the unit of parallelism, of coarse random access, and of remote column projection. By default its size comes from --level — higher effort uses larger groups, which train the sequence model on more reads for a better ratio. Pass --block-reads <N> to set it directly and decouple granularity from effort.

The trade-off is real. Smaller groups mean finer random access and more parallelism, but a worse ratio (the order-k sequence model has fewer reads to train on per group) and a slightly larger footer index. Larger groups compress better but make the smallest independently-fetchable unit coarser. When archiving to object storage where clients issue small Range reads — say, fetching just the read names, or just one row group — a smaller --block-reads makes those fetches cheaper; for a write-once/read-sequentially archive the --level default is the right call.

Long reads are sized in bytes, not reads. A long-read file holds so few reads that the read count never binds; blocks are cut by a raw-sequence byte budget instead. For Nanopore input that budget defaults to 64 MiB (other platforms use the 256 MiB cap): on a 576 MB MinION file that is 5 blocks instead of 2, which took full decode from flat-at-any-thread-count to 3.2× faster at 8 threads, and --fasta from 4.6 s to 1.9 s, for +1.08% archive size. --max keeps maximal 256 MiB blocks — its contract is the smallest archive, and the last ~1% lives in the sequence codec's per-block coverage. To trade further in either direction pass --block-reads: with it the read count alone governs the cut (e.g. --block-reads 2000 on ~14 kb reads is ~28 MB groups — more parallelism at ~+3% size; a large N restores maximal blocks at any level).

The per-group and per-stream byte offsets recorded in the footer are what make this projection possible without reading whole blocks — see Container format → Column projection. The reorder path (--order any/--max) clusters globally and does not use this sizing, so --block-reads is ignored there.

Lossy quality binning¶

Quality is lossless by default. --quality-bin maps each quality byte through a fixed table before coding — an explicit, opt-in lossy transform that shrinks the quality stream (usually the largest part of the archive). Sequence and read names are never touched, and a binned read still aligns exactly where the original did.

Match the table to the platform. The tables are not interchangeable: the Illumina bins are absolute-Phred cutpoints that collapse HiFi's narrow high-Q band into a single level, and ont applied to HiFi data folds the Q93 max-quality symbol into the 26+ bin, destroying its application meaning (measured mean |Δ| 42.84, 99.4% of bases changed). hifi keeps Q93 as its own level. On ONT data the ont and hifi tables are byte-identical, since ONT never reaches Q93.

On the ecoli_ont benchmark, --quality-bin ont cuts the quality stream from 165.2 MB to 47.0 MB (3.5×) — the whole archive from 2.96× to 7.05× — at mean |Δ| 3.35. Cutpoints should ultimately be judged by downstream fidelity, not raw ratio — see Long-read support.

Notes¶

• Losslessness: read name + description, sequence, and quality are preserved exactly. The + separator line is normalized to a bare +.
• Grouping: interleaving mates shrinks the archive — near-identical mate names collapse to matches, and a spot's related reads sit together for the sequence model.