BEAT Speed (nightly) #66
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # BEAT speed beats — NIGHTLY (PMAT-722) | |
| # | |
| # Runs the timing-based BEAT gates that can't live in per-PR CI (absolute | |
| # wall-clock is flaky across hosts, and they need `uv` + scikit-learn). These | |
| # tests are `#[ignore]`d in the normal suite and run only here, on a single | |
| # self-hosted host so apr and the incumbent are timed on the same machine in the | |
| # same run (the gate is the RELATIVE ratio, which cancels host-speed variance). | |
| # | |
| # Additive: a brand-new workflow that does NOT touch per-PR `ci.yml` / the merge | |
| # gate. A red run here flags a real speed regression; it never blocks PRs. | |
| # | |
| # OBSERVABILITY INVARIANT (added after the 2026-07-27/28 blackout). Steps here | |
| # are siblings, not a pipeline: each beat times a DIFFERENT estimator against a | |
| # DIFFERENT incumbent, so a failure in one says nothing about the next. With | |
| # default fail-fast semantics, though, the first red step aborted the job and | |
| # the remaining nine never ran — and a step that never ran reports no | |
| # measurement, which reads exactly like a step that ran and passed. That is how | |
| # a broken scikit-learn baseline in the LinReg leg (a poisoned `uv` env cache — | |
| # see scripts/check_beat_baseline_env.sh) silently took the ENTIRE Pillar-1 | |
| # speed lane off the air while the GaussianNB beat was actively breaching its | |
| # ceiling (2026-07-03 ratio 0.563 and 07-04 ratio 0.520, both over the 0.50 | |
| # gate) with nobody watching. | |
| # | |
| # So this workflow now enforces two rules: | |
| # 1. Every beat runs, even if an earlier beat failed (`!cancelled()`), so one | |
| # broken leg can never hide the other nine. | |
| # 2. A MISSING measurement is a FAILURE, not a pass. The final step asserts | |
| # that every expected `BEAT-*` marker line actually appeared in the log. | |
| # Silence is treated as red, because silence is what the blackout looked | |
| # like. | |
| name: BEAT Speed (nightly) | |
| on: | |
| schedule: | |
| - cron: "45 5 * * *" # 05:45 UTC, after Nightly Bench (05:30) | |
| workflow_dispatch: | |
| concurrency: | |
| group: beat-speed-${{ github.ref }} | |
| cancel-in-progress: true | |
| jobs: | |
| beat-speed: | |
| runs-on: [self-hosted, X64, Linux, clean-room] | |
| timeout-minutes: 45 | |
| env: | |
| # Every beat step tees here; the final step audits it for completeness. | |
| # | |
| # NOT `runner.temp`: the `runner` context does not exist in a JOB-level env | |
| # block (it is only available to steps), and using it there does not warn - | |
| # it makes the whole workflow unparseable, so `workflow_dispatch` fails with | |
| # "Unrecognized named-value: 'runner'" and the lane cannot run at all. That | |
| # is exactly the blackout this file exists to prevent, reintroduced by the | |
| # fix for it (#2326). `github` IS a valid context at job level. | |
| BEAT_LOG: ${{ github.workspace }}/beat-speed-measurements.log | |
| steps: | |
| - uses: actions/checkout@v7 | |
| - name: Ensure uv is available (for scikit-learn baselines) | |
| run: | | |
| if command -v uv >/dev/null 2>&1; then | |
| echo "uv present: $(uv --version)" | |
| else | |
| echo "Installing uv to ~/.local/bin" | |
| curl -LsSf https://astral.sh/uv/install.sh | sh | |
| echo "$HOME/.local/bin" >> "$GITHUB_PATH" | |
| fi | |
| # A speed beat is a RATIO against the incumbent. If the incumbent will not | |
| # import, the beat measures nothing — so prove it imports before timing | |
| # anything, and self-heal the known poisoned-uv-cache fault in place. | |
| - name: Preflight — scikit-learn/numpy baseline must be importable | |
| id: preflight | |
| run: bash scripts/check_beat_baseline_env.sh | |
| - name: Pillar-1 — apr vs scikit-learn LinearRegression speed beat | |
| if: ${{ !cancelled() && steps.preflight.outcome == 'success' }} | |
| run: | | |
| set -o pipefail | |
| cargo test -p aprender-core --release \ | |
| --test beat_sklearn_linreg_speed -- --ignored --nocapture 2>&1 | tee -a "$BEAT_LOG" | |
| - name: Pillar-1 — apr vs scikit-learn GaussianNB speed beat | |
| if: ${{ !cancelled() && steps.preflight.outcome == 'success' }} | |
| run: | | |
| set -o pipefail | |
| cargo test -p aprender-core --release \ | |
| --test beat_sklearn_gaussiannb_speed -- --ignored --nocapture 2>&1 | tee -a "$BEAT_LOG" | |
| # NOTE: StandardScaler / MinMaxScaler speed beats REMOVED (PMAT-733). Elementwise scalers have | |
| # no algorithmic edge over numpy — they "won" only on a fast dev box; on the canonical Intel CI | |
| # runner (MKL numpy) apr LOSES (StandardScaler measured ratio 1.443 = apr 0.69x). Only beats with | |
| # a genuine algorithmic edge (compute-bound: LinReg, the NB family, GMM) are kept — they hold | |
| # across hosts. See memory: verify beats on the CI host, not just the dev box. | |
| - name: Pillar-1 — apr vs scikit-learn ComplementNB speed beat | |
| if: ${{ !cancelled() && steps.preflight.outcome == 'success' }} | |
| run: | | |
| set -o pipefail | |
| cargo test -p aprender-core --release \ | |
| --test beat_sklearn_complementnb_speed -- --ignored --nocapture 2>&1 | tee -a "$BEAT_LOG" | |
| - name: Pillar-1 — apr vs scikit-learn BernoulliNB speed beat | |
| if: ${{ !cancelled() && steps.preflight.outcome == 'success' }} | |
| run: | | |
| set -o pipefail | |
| cargo test -p aprender-core --release \ | |
| --test beat_sklearn_bernoullinb_speed -- --ignored --nocapture 2>&1 | tee -a "$BEAT_LOG" | |
| - name: Pillar-1 — apr vs scikit-learn MultinomialNB speed beat | |
| if: ${{ !cancelled() && steps.preflight.outcome == 'success' }} | |
| run: | | |
| set -o pipefail | |
| cargo test -p aprender-core --release \ | |
| --test beat_sklearn_multinomialnb_speed -- --ignored --nocapture 2>&1 | tee -a "$BEAT_LOG" | |
| - name: Pillar-1 — apr vs scikit-learn GaussianMixture (GMM) speed beat | |
| if: ${{ !cancelled() && steps.preflight.outcome == 'success' }} | |
| run: | | |
| set -o pipefail | |
| cargo test -p aprender-core --release \ | |
| --test beat_sklearn_gmm_speed -- --ignored --nocapture 2>&1 | tee -a "$BEAT_LOG" | |
| - name: Pillar-2 — apr vs PyTorch one-shot cold-start training speed beat | |
| if: ${{ !cancelled() && steps.preflight.outcome == 'success' }} | |
| run: | | |
| set -o pipefail | |
| cargo test -p aprender-core --release \ | |
| --test beat_pytorch_coldstart_speed -- --ignored --nocapture 2>&1 | tee -a "$BEAT_LOG" | |
| - name: Pillar-3 — apr vs Unsloth one-shot cold-start LoRA-adapter speed beat | |
| if: ${{ !cancelled() && steps.preflight.outcome == 'success' }} | |
| run: | | |
| set -o pipefail | |
| cargo test -p aprender-train --release \ | |
| --test beat_unsloth_coldstart_speed -- --ignored --nocapture 2>&1 | tee -a "$BEAT_LOG" | |
| - name: Pillar-1 — apr vs scikit-learn one-shot cold-start speed beat | |
| if: ${{ !cancelled() && steps.preflight.outcome == 'success' }} | |
| run: | | |
| set -o pipefail | |
| cargo test -p aprender-core --release \ | |
| --test beat_sklearn_coldstart_speed -- --ignored --nocapture 2>&1 | tee -a "$BEAT_LOG" | |
| - name: Pillar-4 — apr vs HuggingFace transformers one-shot inference cold-start speed beat | |
| if: ${{ !cancelled() && steps.preflight.outcome == 'success' }} | |
| run: | | |
| set -o pipefail | |
| cargo test -p aprender-core --release \ | |
| --test beat_hf_inference_coldstart_speed -- --ignored --nocapture 2>&1 | tee -a "$BEAT_LOG" | |
| # Fail-closed observability: a beat that produced no measurement line did | |
| # not "pass quietly", it went dark. Assert all ten reported. | |
| - name: Every beat must have reported a measurement | |
| if: ${{ !cancelled() }} | |
| run: bash scripts/check_beat_measurements.sh "$BEAT_LOG" |