fqxv compress¶

Compress one or more FASTQ files into a single .fqxv archive.

Usage¶

fqxv compress <INPUTS>... [-o <OUTPUT>] [OPTIONS]

Give one input for single-end, two for paired-end, or three/four for single-cell (R1/R2/I1[/I2]). Each input's slot label (the R1/I1/2 token before its extension) is recorded in the archive when every input has a distinct one, so decompress --split can restore the original per-slot names rather than renumbering positionally. Multiple inputs are interleaved per spot into one archive and can be split back out with decompress --split. Input order is preserved for the split.

Arguments¶

Options¶

Advanced options¶

Encryption options¶

Examples¶

# single-end, gzipped input (-o defaults to reads.fqxv)
fqxv compress reads.fastq.gz

# paired-end at higher effort
fqxv compress sample_R1.fq.gz sample_R2.fq.gz -o sample.fqxv --level 7 --threads 16

# 10x single-cell (R1 + R2 + I1 + I2)
fqxv compress R1.fq R2.fq I1.fq I2.fq -o sample.fqxv

# lossy quality (Illumina 8-level binning)
fqxv compress reads.fastq -o reads.fqxv --quality-bin bin8

# lossy quality on long reads (match the table to the platform)
fqxv compress ont_reads.fastq -o ont_reads.fqxv --quality-bin ont
fqxv compress hifi_reads.fastq -o hifi_reads.fqxv --quality-bin hifi

# finer row groups for remote/parallel access (smaller range fetches)
fqxv compress reads.fastq.gz -o reads.fqxv --block-reads 65536

# maximum compression (deepest context + read reordering where it helps)
fqxv compress reads.fastq.gz --max

# verify the archive round-trips before trusting (or deleting) the source
fqxv compress reads.fastq.gz --verify

# estimate the ratio and archive size from a sample, writing nothing
fqxv compress reads.fastq.gz --estimate

# ...the same estimate as a two-line table for scripts
fqxv compress reads.fastq.gz --estimate tsv

# encrypt with a passphrase read from a file
fqxv compress reads.fastq.gz -o reads.fqxv --encrypt --password-file secret.txt

# ...or scripted via an environment variable (no file, no prompt)
FQXV_PASSWORD=hunters2 fqxv compress reads.fastq.gz -o reads.fqxv --encrypt

Verifying on write¶

--verify closes the window between writing an archive and trusting it. After the archive is written, it is reopened and fully decoded — exercising every block CRC-32C and content digest — and the decoded read count is checked against what was written, all before compress reports success:

fqxv compress reads.fastq.gz --verify

This catches a codec or in-flight memory error that produced a CRC-valid but wrong archive, which the archive's own on-disk checksums cannot detect on their own. It is the check to run before deleting the source FASTQ.

• On any failure — a decode error or a read-count mismatch — the archive is left in place for inspection and the command exits non-zero. A bad archive is never reported as done.
• It adds a full decode pass, so expect roughly double the wall time (the --max / reorder path is the heaviest).
• It validates the bytes the encoder emitted, not long-term on-disk durability; for the latter, re-run fqxv verify against the stored file later.

Estimating compression¶

--estimate tells you how well an input will compress before committing to a full run. It takes a bounded sample of the leading reads (up to 300,000 reads, or one block's worth of bases — whichever binds first, which on long reads is the bases) and predicts the archive without coding it: the real codecs are entropy coders, so a histogram pass measures directly what they converge to — static order-k sequence entropy (blended with a k-mer duplication sketch on the long-read path), order-1 quality entropy, and the real tokenizer for names, each scaled by a small per-platform calibration factor. It then projects the whole-file archive at the chosen --level/--quality-bin and prints the result — no archive is written:

reads.fastq.gz (59.99 MB)  →  estimated fqxv ~37.58 MB  (37% smaller, ~1.60x)

Estimated from a 300,000-read sample (102.26 MB uncompressed FASTQ):
  stream      compressed     share   rate
  names        11.27 KB     0.0%   0.31 bits/read
  sequence      8.92 MB    31.6%   1.663 bits/base
  quality      19.26 MB    68.3%   3.590 bits/base
  vs uncompressed FASTQ (~136.32 MB):  ~3.63x

The top line leads with the outcome: input size(s) → estimated archive size, as a percent reduction and ratio against the input on disk. The per-stream table below shows where the bytes go (names / sequence / quality) for the sample, and the final line gives the ratio against the whole file's uncompressed FASTQ bytes. Multiple inputs are listed individually under a combined headline; a streaming stdin input has no on-disk size, so it reports the sample's own reduction and omits the whole-file projection (pass a file to get one).

For scripting, --estimate tsv prints the same projection as a header line plus one data row:

file    input_bytes est_fqxv_bytes  ratio
reads.fastq.gz  62898962    39405273    1.5962

Accuracy. Because blocks are coded independently and the models are per-block stationary, the sample's ratio is a faithful proxy for the whole file — in practice within ~1–2% of a real run (every projected figure is marked ~). The projection scales the sample by the fraction of on-disk bytes it consumed, so it works for plain and gzipped inputs alike.

Reordering is a lower bound. --order any / --max reordering is not modeled — its cross-read redundancy grows with read count, so a small sample can't capture it. With those flags the estimate is a conservative lower bound (the real archive comes out that size or smaller), and the report says so.

Row-group sizing¶

The archive is a run of independently-coded row groups (blocks); the row group is the unit of parallelism, of coarse random access, and of remote column projection. By default its size comes from --level — higher effort uses larger groups, which train the sequence model on more reads for a better ratio. Pass --block-reads <N> to set it directly and decouple granularity from effort.

The trade-off is real. Smaller groups mean finer random access and more parallelism, but a worse ratio (the order-k sequence model has fewer reads to train on per group) and a slightly larger footer index. Larger groups compress better but make the smallest independently-fetchable unit coarser. When archiving to object storage where clients issue small Range reads — say, fetching just the read names, or just one row group — a smaller --block-reads makes those fetches cheaper; for a write-once/read-sequentially archive the --level default is the right call.

Long reads are sized in bytes, not reads. A long-read file holds so few reads that the read count never binds; blocks are cut by a raw-sequence byte budget instead. For Nanopore input that budget defaults to 64 MiB (other platforms use the 256 MiB cap): on a 576 MB MinION file that is 5 blocks instead of 2, which took full decode from flat-at-any-thread-count to 3.2× faster at 8 threads, and --fasta from 4.6 s to 1.9 s, for +1.08% archive size. --max keeps maximal 256 MiB blocks — its contract is the smallest archive, and the last ~1% lives in the sequence codec's per-block coverage. To trade further in either direction pass --block-reads: with it the read count alone governs the cut (e.g. --block-reads 2000 on ~14 kb reads is ~28 MB groups — more parallelism at ~+3% size; a large N restores maximal blocks at any level).

The per-group and per-stream byte offsets recorded in the footer are what make this projection possible without reading whole blocks — see Container format → Column projection. The reorder path (--order any/--max) clusters globally and does not use this sizing, so --block-reads is ignored there.

Lossy quality binning¶

Quality is lossless by default. --quality-bin maps each quality byte through a fixed table before coding — an explicit, opt-in lossy transform that shrinks the quality stream (usually the largest part of the archive). Sequence and read names are never touched, and a binned read still aligns exactly where the original did.

Match the table to the platform. The tables are not interchangeable: the Illumina bins are absolute-Phred cutpoints that collapse HiFi's narrow high-Q band into a single level, and ont applied to HiFi data folds the Q93 max-quality symbol into the 26+ bin, destroying its application meaning (measured mean |Δ| 42.84, 99.4% of bases changed). hifi keeps Q93 as its own level. On ONT data the ont and hifi tables are byte-identical, since ONT never reaches Q93.

On the ecoli_ont benchmark, --quality-bin ont cuts the quality stream from 165.2 MB to 47.0 MB (3.5×) — the whole archive from 2.96× to 7.05× — at mean |Δ| 3.35. Cutpoints should ultimately be judged by downstream fidelity, not raw ratio — see Long-read support.

Encryption¶

--encrypt seals the archive with a passphrase (ChaCha20-Poly1305, per-block AEAD via Argon2id key derivation) — natively, inside the container's own framing, rather than wrapping the file with a separate tool. An old fqxv build refuses an encrypted archive cleanly (it recognizes the feature bit and reports "upgrade fqxv") rather than misreading it.

The passphrase is resolved from, in order:

1. --password-file <PATH> — the file's raw bytes (a single trailing newline stripped); keep this file access-controlled, fqxv does not manage its permissions or lifecycle.
2. the FQXV_PASSWORD environment variable — scriptable/CI-friendly, but visible to other processes of the same user (e.g. via /proc/<pid>/environ on Linux).
3. an interactive hidden-terminal prompt, asked twice with confirmation (a typo here is unrecoverable, unlike a wrong guess at decode time, which just fails cleanly and can be retried) — the safest source against shell-history or process-inspection leakage, but needs a real terminal.

There is no recovery without the passphrase. Losing it means losing the archive; there is no back door, and no CLI flag to reset or rotate it in place. fqxv info/fqxv verify on an encrypted archive both work without a password — info reports encrypted: yes and the total ciphertext size but no per-stream breakdown (that can't be known without decrypting every block); verify runs its usual checksum checks and, given the passphrase via --password-file, additionally checks a footer-authentication tag that catches an archive whose trailing blocks were silently dropped — see fqxv verify.

Not supported with --order any/shuffle or --max (which implies --order any) in this release — that layout needs its own encryption design, distinct from the plain layout's per-block scheme. Column-projection random access (fqxv.remote in the Python bindings) is also not supported against an encrypted archive: whole-block encryption means a single stream can't be fetched or authenticated independently of the block that contains it, so fqxv.remote falls back to fetching and decrypting the whole archive instead. Full design and rationale: Encryption.

Notes¶

• Losslessness: read name + description, sequence, and quality are preserved exactly. The + separator line is normalized to a bare +.
• Grouping: interleaving mates shrinks the archive — near-identical mate names collapse to matches, and a spot's related reads sit together for the sequence model.