Skip to content

Commit a6f74e9

Browse files
noahgiftclaude
andcommitted
docs(beat): record what the CI host actually measures (0.788/0.805, ~11% headroom)
Validated the new configuration by dispatching beat-speed-nightly twice onto the clean-room pool - the host that produced the 1.044 excursion: apr=265.473ms sklearn=337.106ms ratio=0.788 apr=252.463ms sklearn=313.446ms ratio=0.805 2.2% spread where the old size swung 346% on that same pool, so the stability goal is met. But the LEVEL differs from my local box and the file said so incorrectly: apr is ~1.25x faster there, not ~1.85x, leaving ~11% headroom under the 0.90 ceiling rather than 42%. That host penalises apr more than sklearn (2.3x vs 1.55x slower than lambda-vector). Recorded two things a future reader needs. Enlarging the workload again is NOT the remedy if this fails - apr's advantage shrinks with size, so a bigger problem makes the ratio worse. And the contract's baseline_floor 0.56 describes neither this host nor the old size; whether that is slower CI hardware or a real regression cannot be settled without historical CI data at this size, so it is left open rather than asserted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 2722b94 commit a6f74e9

1 file changed

Lines changed: 25 additions & 0 deletions

File tree

crates/aprender-core/tests/beat_sklearn_linreg_speed.rs

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,31 @@
4040
//! The large workload spans 0.49-0.64 across every locally inducible
4141
//! condition; the small one spanned 8x on CI.
4242
//!
43+
//! VALIDATED ON THE HOST THAT FAILED. Dispatched twice onto the clean-room
44+
//! pool at 200_000x50:
45+
//!
46+
//! ```text
47+
//! apr=265.473ms sklearn=337.106ms ratio=0.788
48+
//! apr=252.463ms sklearn=313.446ms ratio=0.805
49+
//! ```
50+
//!
51+
//! 2.2% spread, where the 10_000x20 configuration swung 0.234 -> 1.044 (346%)
52+
//! on that same pool. The stability goal is met.
53+
//!
54+
//! But read the LEVEL, not just the spread: on that host apr is ~1.25x faster,
55+
//! NOT the ~1.85x measured on lambda-vector, leaving only ~11% headroom under
56+
//! the 0.90 ceiling. apr is penalised more than sklearn by that host (2.3x vs
57+
//! 1.55x slower than lambda-vector). Do NOT "fix" a future failure here by
58+
//! enlarging the workload again - apr's advantage SHRINKS with size
59+
//! (0.26 at 10_000x20 -> 0.54 at 200_000x50 locally), so a bigger problem
60+
//! makes the ratio worse, not better.
61+
//!
62+
//! OPEN, not resolved here: the contract records baseline_floor 0.56, which
63+
//! describes neither this host (0.79-0.81) nor the old size (0.26). Whether
64+
//! that gap is slower CI hardware or a real apr regression cannot be settled
65+
//! without historical CI data at this size, and is deliberately not asserted
66+
//! either way.
67+
//!
4368
//! TWO HONEST CAVEATS. (1) The headline win SHRINKS with size: apr is ~3.6x
4469
//! faster at 10_000x20 but ~1.7x at 200_000x50. The smaller number is the one
4570
//! that can be measured reliably, and a gate that measures reliably is worth

0 commit comments

Comments
 (0)