Proposal
When fgumi sort's input header already declares the order being requested, skip the sort entirely: rewrite the header and copy the input's BGZF blocks to the output. Gate the old behaviour behind --force-sort.
Why
Re-sorting an already-sorted BAM is a real user action — idempotent pipeline steps, re-sorting someone else's output — and today it does the full amount of work. Measured on 1kg-wgs-HG00096 (779,820,469 records, c7g.4xlarge, 8 threads, --max-memory 512M), a coordinate→coordinate sort takes ~567s: read 43 GB, decompress, sort, compress 46.7 GB of spills, write them, read them back, decompress, merge, compress 48.8 GB of output, write.
A copy path replaces all of that with read 43 GB → write 43 GB, roughly 150s — about 3.8× on the whole command, and it removes the pathological merge case described in #709 rather than optimizing it.
Detection already exists
header_declares_order (crates/fgumi-sort/src/lib.rs, added in #709) is exactly the predicate this needs, and it is unit-tested. It is deliberately conservative: SO must match, and when the order defines a sub-sort tag SS must match too — so a queryname input with no SS does not match either queryname flavor, and template-coordinate is distinguished from plain SO:unsorted by its SS. #709 already logs a warning on exactly this condition, so the branch point is in place and observable.
Design notes / open questions
- Headers lie.
--force-sort is the escape hatch. Worth deciding whether the copy path should also verify order as it streams and fail loudly on a violation (cheap — it is comparing adjacent keys it has already parsed) rather than silently producing a mis-labelled file.
- A BAM header can share a BGZF block with the first records, so the copy cannot simply start at a block boundary after the header. The clean form is: emit a fresh header (with the new
@PG) as its own blocks, decompress-and-re-encode only the single boundary block, then copy every subsequent block verbatim. Worth checking how often writers flush after the header, in which case that special case never fires.
--compression-level becomes a no-op when blocks are copied, since the output inherits the input's compression. Either document that, or decline the fast path when the user has explicitly asked for a level.--write-index needs record coordinates, so BAI generation forces decompression. Still no sort and no merge, so most of the win survives, but it is a separate mode. Simplest first cut is to decline the fast path when --write-index is set.- Works on pipes without seeking:
read_header_and_replay (crates/fgumi-bam-io/src/reader.rs:537) tees the compressed bytes and replays the raw BGZF stream, so the copy path has raw blocks available on stdin too.
Relationship to other work
#709 makes the degenerate merge ~9% faster. This makes it unnecessary. They are complementary — this handles the case where the header is honest, while a data-driven approach (detecting sortedness from the spilled runs themselves, no header trust) covers almost-sorted input and lying headers.
Proposal
When
fgumi sort's input header already declares the order being requested, skip the sort entirely: rewrite the header and copy the input's BGZF blocks to the output. Gate the old behaviour behind--force-sort.Why
Re-sorting an already-sorted BAM is a real user action — idempotent pipeline steps, re-sorting someone else's output — and today it does the full amount of work. Measured on
1kg-wgs-HG00096(779,820,469 records, c7g.4xlarge, 8 threads,--max-memory 512M), a coordinate→coordinate sort takes ~567s: read 43 GB, decompress, sort, compress 46.7 GB of spills, write them, read them back, decompress, merge, compress 48.8 GB of output, write.A copy path replaces all of that with read 43 GB → write 43 GB, roughly 150s — about 3.8× on the whole command, and it removes the pathological merge case described in #709 rather than optimizing it.
Detection already exists
header_declares_order(crates/fgumi-sort/src/lib.rs, added in #709) is exactly the predicate this needs, and it is unit-tested. It is deliberately conservative:SOmust match, and when the order defines a sub-sort tagSSmust match too — so aquerynameinput with noSSdoes not match either queryname flavor, and template-coordinate is distinguished from plainSO:unsortedby itsSS. #709 already logs a warning on exactly this condition, so the branch point is in place and observable.Design notes / open questions
--force-sortis the escape hatch. Worth deciding whether the copy path should also verify order as it streams and fail loudly on a violation (cheap — it is comparing adjacent keys it has already parsed) rather than silently producing a mis-labelled file.@PG) as its own blocks, decompress-and-re-encode only the single boundary block, then copy every subsequent block verbatim. Worth checking how often writers flush after the header, in which case that special case never fires.--compression-levelbecomes a no-op when blocks are copied, since the output inherits the input's compression. Either document that, or decline the fast path when the user has explicitly asked for a level.--write-indexneeds record coordinates, so BAI generation forces decompression. Still no sort and no merge, so most of the win survives, but it is a separate mode. Simplest first cut is to decline the fast path when--write-indexis set.read_header_and_replay(crates/fgumi-bam-io/src/reader.rs:537) tees the compressed bytes and replays the raw BGZF stream, so the copy path has raw blocks available on stdin too.Relationship to other work
#709 makes the degenerate merge ~9% faster. This makes it unnecessary. They are complementary — this handles the case where the header is honest, while a data-driven approach (detecting sortedness from the spilled runs themselves, no header trust) covers almost-sorted input and lying headers.