Branch scope: This concerns the arena-based sort front on the feat-runall integration branch (introduced by the #472–#476 stack). main is not affected — it still uses the legacy RawExternalSorter path described below. File/line references are on feat-runall.
Summary
On feat-runall, standalone fgumi sort --max-memory auto allocates its in-memory buffer differently than the legacy path on main: the arena front reserves the full auto-detected budget up front, whereas main starts at min(768 MiB × threads, budget) and grows on demand. The spill trigger (when data goes to disk) is unchanged; this is purely about up-front allocation.
Background
main (legacy RawExternalSorter) caps the initial buffer allocation and grows lazily:
// commands/sort.rs (main)
sorter = sorter.initial_capacity(effective_memory.min(768 MiB × threads));
// external.rs
fn effective_initial_capacity(&self) -> usize {
self.initial_capacity.unwrap_or(self.memory_limit).min(self.memory_limit)
}feat-runall's arena front (ReadBlocks → InflateToArena → FindBoundariesAndSort, added in #474 and wired for standalone sort in #476) sizes and eagerly grows its arena segment to a full run:
// crates/fgumi-pipeline-io/src/sort/arena_ingest.rs (feat-runall)
let run_cap = memory_limit; // = full auto budget
let segment_size = FRONT_REGION + run_cap + MAX_BGZF_BLOCK; // grown eagerly at acquire
A 768 MiB/thread initial clamp was carried into the new builder but was dead code (it clamped a sorter that is dropped on the arena path) and was removed in #476, with a comment explaining why the clamp can't simply be redirected into ReadBlocks::new — its argument is both the arena segment_size and the run_cap spill trigger, so capping it would force premature spills, re-introducing the wall-clock regression ReadBlocks::new's own docs warn about.
Why it may matter
grow_uninit extends length over reserved-but-untouched capacity (set_len, no zero-fill), so resident memory still scales with actual data — pages fault in as records are written. The concrete difference is the allocator reservation / virtual footprint at acquire: on a large-RAM host where auto resolves to tens of GB, sorting a small file requests the entire budget from mimalloc up front instead of ≤ 768 MiB × threads. Whether that becomes real RSS depends on mimalloc/OS lazy-commit behavior for large regions (usually lazy, but not guaranteed).
Proposed direction
Bound what --max-memory auto resolves to for standalone sort (in resolve_memory_budget, or a sort-specific cap) rather than clamping the arena. That caps allocation and the spill trigger together, deliberately — the honest tradeoff — instead of a dead initial-only clamp. Alternatives to weigh:
- A sort
auto cap (e.g. min(detected, 768 MiB × threads × k)), documented as intentional.
- Measure actual RSS under mimalloc for a small-file / big-
auto sort first, to confirm there's a real problem before changing behavior (the reservation may stay largely virtual).
Acceptance criteria
References
Summary
On
feat-runall, standalonefgumi sort --max-memory autoallocates its in-memory buffer differently than the legacy path onmain: the arena front reserves the full auto-detected budget up front, whereasmainstarts atmin(768 MiB × threads, budget)and grows on demand. The spill trigger (when data goes to disk) is unchanged; this is purely about up-front allocation.Background
main(legacyRawExternalSorter) caps the initial buffer allocation and grows lazily:feat-runall's arena front (ReadBlocks → InflateToArena → FindBoundariesAndSort, added in #474 and wired for standalone sort in #476) sizes and eagerly grows its arena segment to a full run:A
768 MiB/threadinitial clamp was carried into the new builder but was dead code (it clamped asorterthat is dropped on the arena path) and was removed in #476, with a comment explaining why the clamp can't simply be redirected intoReadBlocks::new— its argument is both the arenasegment_sizeand therun_capspill trigger, so capping it would force premature spills, re-introducing the wall-clock regressionReadBlocks::new's own docs warn about.Why it may matter
grow_uninitextends length over reserved-but-untouched capacity (set_len, no zero-fill), so resident memory still scales with actual data — pages fault in as records are written. The concrete difference is the allocator reservation / virtual footprint at acquire: on a large-RAM host whereautoresolves to tens of GB, sorting a small file requests the entire budget from mimalloc up front instead of ≤768 MiB × threads. Whether that becomes real RSS depends on mimalloc/OS lazy-commit behavior for large regions (usually lazy, but not guaranteed).Proposed direction
Bound what
--max-memory autoresolves to for standalone sort (inresolve_memory_budget, or a sort-specific cap) rather than clamping the arena. That caps allocation and the spill trigger together, deliberately — the honest tradeoff — instead of a dead initial-only clamp. Alternatives to weigh:autocap (e.g.min(detected, 768 MiB × threads × k)), documented as intentional.autosort first, to confirm there's a real problem before changing behavior (the reservation may stay largely virtual).Acceptance criteria
feat-runall: small file, largeautobudget, arena front vs.main's768 MiB/threadbehavior.autofor standalone sort no longer reserves the full detected budget up front, without forcing earlier spills for inputs that fit the budget.--max-memory.feat-runall(before the stack merges tomain).References
feat-runall(stack fix(bgzf): verify CRC32 + short-fill in decompress_into_slice #472–feat(sort): wire arena engine into fgumi chains + retire legacy sorters #476)crates/fgumi-pipeline-io/src/sort/arena_ingest.rs—ReadBlocks::new,ensure_arenasrc/lib/pipeline/chains/builder.rs— standalone-sort arena wiring