## Summary

Make Terminal-Bench runner stop conditions scoreable without hiding real
mux failures. Strict goal-mode incomplete exits now leave the workspace
for verifier scoring, and task timeouts are classified through Harbor
instead of GNU `timeout` exit codes.

## Background

Recent full-suite Terminal-Bench runs surfaced mux command exit `124` as
adapter `RuntimeError`s, which made expected task timeouts look like mux
execution failures. Goal-mode runs also needed the adapter to
distinguish semantic incomplete status from process failure so Harbor
can score the final workspace.

## Implementation

- Removed the runner-side GNU `timeout` wrapper while preserving
`MUX_TIMEOUT_MS` as a compatibility env marker.
- Normalized mux strict-goal exit `3` only when `MUX_RUN_AS_GOAL=1` and
only after token extraction.
- Kept unexpected nonzero runner exits strict.
- Preserved Harbor timeout classification in the adapter by mapping
environment timeout failures to `AgentTimeoutError`.
- Added fake-runner smoke tests for scoreable goal stops, fatal exits,
and no inner timeout wrapper.
- Updated Terminal-Bench docs/skill content and regenerated bundled
skill content.
- Fixed the Nix format check temp copy so CI can evaluate the flake
formatter with the lock/package inputs it references.

## Validation

- `make static-check`
- `make fmt`
- `/tmp/mux-tbench-venv312/bin/python -m pytest
benchmarks/terminal_bench/mux_agent_test.py`
- `./scripts/check-bench-agent.sh`
- `shellcheck benchmarks/terminal_bench/mux-run.sh`
- After CI surfaced the Nix formatting issue: `make fmt-check`, `make
static-check`, `/tmp/mux-tbench-venv312/bin/python -m pytest
benchmarks/terminal_bench/mux_agent_test.py`

## Risks

Low-to-medium benchmark-only risk. The changes are isolated to the
Terminal-Bench adapter/runner, docs, and the static-check Nix formatting
helper; unexpected mux process exits remain strict.

---

<details>
<summary>📋 Implementation Plan</summary>

# Plan: Make Terminal-Bench goal/timeout stop conditions scoreable
without hiding real mux failures

## Context and evidence

- The referenced Terminal-Bench workflow run `26294491586` completed the
benchmark step but failed in `Verify agent ran successfully`.
- Run stats observed from GitHub Actions logs:
- workflow: `Terminal-Bench`, run `#123`, `workflow_dispatch`, branch
`main`, SHA `8ed9adf214d4368282817972e2487d1f6f43c529`
- model: `openai/gpt-5.5`; dataset: `terminal-bench@2.0`; concurrency:
`4`; `MUX_RUN_AS_GOAL=1`
- trials: `89`; mean: `0.7303370786516854`; reward `1.0`: `65`; reward
`0.0`: `19`
  - exceptions: `9` total = `5` `RuntimeError` + `4` `AgentTimeoutError`
- tokens/cost: `5,251,596` input, `346,442` output, `$75.80921599999998`
- BigQuery upload still succeeded with `89` rows; artifact upload
succeeded.
- Step 13 failed because `.github/workflows/terminal-bench.yml` now
reads `.stats.n_errored_trials` and subtracts infra-like exception
files. That exposed `5` non-infra `RuntimeError`s:
  - `caffe-cifar-10__8dQMih5`: mux command exit `137`
  - `train-fasttext__Z3H4tiT`: mux command exit `124`
  - `winning-avg-corewars__9AAaszy`: mux command exit `124`
  - `make-mips-interpreter__HQ4sdb7`: mux command exit `1`
  - `dna-assembly__d2t2LqT`: mux command exit `1`
- The goal-specific incomplete exit code (`3`) is a real harness risk,
but this specific run did **not** show exit `3`. The immediate
timeout-related issue is that `benchmarks/terminal_bench/mux-run.sh`
wraps `mux run` in GNU `timeout`, so normal task timeout can become mux
command exit `124` and Harbor records it as `RuntimeError` instead of
`AgentTimeoutError`.
- Relevant files:
- `benchmarks/terminal_bench/mux-run.sh` — invokes `bun src/cli/run.ts`,
wraps with `timeout`, and propagates mux nonzero exit codes.
- `benchmarks/terminal_bench/mux_agent.py` — Harbor adapter raises
`RuntimeError` on any nonzero command return.
- `benchmarks/terminal_bench/mux_agent_test.py` — tests nonzero adapter
behavior and env normalization.
- `.github/workflows/terminal-bench.yml` — validates result JSON and
classifies infra vs non-infra errors.
- `Makefile` `benchmark-terminal` target — passes Harbor timeout and
also exports `MUX_TIMEOUT_MS`.
- `benchmarks/terminal_bench/README.md` / project `tbench` skill —
documents `MUX_RUN_AS_GOAL`, `MUX_RUN_ARGS`, and timeout behavior.

## Goals

1. Let Terminal-Bench score tasks when mux reaches an expected semantic
stop condition, especially strict goal-mode incomplete status.
2. Let Harbor own task timeout classification so timeouts surface as
`AgentTimeoutError`, not adapter `RuntimeError exit 124`.
3. Keep the workflow strict for real mux/process failures (`exit 1`,
unexpected `SIGKILL`/`137`, setup failures, provider crashes) until
artifacts prove they are safe to classify as infra.
4. Preserve token/cost extraction and existing BigQuery/artifact upload
behavior.

## Non-goals

- Do not relax the verifier to blindly ignore all `RuntimeError`s.
- Do not change normal `mux run` CLI exit semantics outside the
Terminal-Bench adapter.
- Do not tune model prompting or task-solving quality in this change.
- Do not submit leaderboard updates.

## Recommended approach (Approach A): normalize only benchmark-harness
stop semantics

**Net product LoC estimate:** ~`+8` to `+18` product LoC, depending on
whether a test-only log-dir override is counted as product-adjacent
shell code. Tests/docs excluded from estimate.

### Phase 1 — Harness changes

1. Update `benchmarks/terminal_bench/mux-run.sh` so Harbor remains the
timeout authority.
- Remove the block that rewrites `cmd=(timeout "${timeout_sec}s"
"${cmd[@]}")` from `MUX_TIMEOUT_MS`.
- Keep `MUX_TIMEOUT_MS` env validation/pass-through for compatibility if
desired, but make `mux-run.sh` not enforce it locally.
- Add/adjust comments to explain that Harbor’s `--agent-kwarg
timeout=<seconds>` should produce `AgentTimeoutError`, while an inner
GNU `timeout` produces adapter `RuntimeError exit 124`.
2. Normalize mux semantic stop codes in
`benchmarks/terminal_bench/mux-run.sh` after token extraction and before
the existing nonzero check.
- If `mux_run_as_goal_enabled=1` and `mux_status=3`, log a warning and
continue with status `0` so Harbor verifier can score the task
workspace.
- Also consider treating `mux_status=2` as scoreable when `MUX_RUN_ARGS`
contains a session `--budget`, because budget exhaustion is a configured
stop condition for benchmarks. Keep this narrow and documented; if not
included in the first patch, explicitly add a test TODO/follow-up.
- Keep exits `1`, `125+`, `137`, and other unexpected codes as failures.
3. Keep `benchmarks/terminal_bench/mux_agent.py` strict: it should still
raise `RuntimeError` when the runner returns nonzero.
- The shell runner is the right layer to distinguish mux CLI semantic
stop codes from process failures because it already knows goal mode and
extracts usage before returning.

### Phase 2 — Tests

1. Update `benchmarks/terminal_bench/mux_agent_test.py` so the generic
nonzero-return test does not use `return_code=3`; use an unmistakably
fatal code such as `7`.
2. Add narrow tests for `mux-run.sh` behavior.
- Preferred: add a temp-log override such as
`MUX_LOG_DIR=${MUX_LOG_DIR:-/logs/agent/command-0}` to make the script
testable without writing `/logs`.
- Use a fake `bun` on `PATH` that records args, emits minimal JSONL
including `run-complete`, and exits with the requested code.
   - Cases:
- goal mode + fake mux exit `3` returns overall shell exit `0`, writes a
warning to stderr, and still writes `/tmp/mux-tokens.json` (or an
override if one is added).
     - non-goal mode + fake mux exit `3` still returns `3`.
     - fake mux exit `1` still returns `1`.
- with `MUX_TIMEOUT_MS` set and a fake `timeout` earlier on `PATH`, the
script still reaches fake `bun`; this proves the inner timeout wrapper
is gone.
- if budget exit `2` is normalized, cover both the positive
budget-configured case and a non-budget case.
3. Run targeted tests:
- `uvx pytest benchmarks/terminal_bench/mux_agent_test.py` for the
Python adapter/script tests.
- `bash -n benchmarks/terminal_bench/mux-run.sh` and
`./scripts/check-bench-agent.sh` for shell syntax and benchmark CLI
packaging/import checks.
- `make test` or the smallest existing suite that includes CLI/run
behavior if any TypeScript CLI code is touched (not expected for
Approach A).
- `make typecheck` if any TypeScript files are touched (not expected for
Approach A).

### Phase 3 — Workflow verifier hardening without broad allowlisting

1. Leave `.github/workflows/terminal-bench.yml` strict for non-infra
`RuntimeError`s.
2. Optionally improve the step output for faster diagnosis:
   - print the exception class counts from `result.json` before failing;
- print per-task `return-code.txt` for non-infra exceptions when
present.
3. Do **not** classify `RuntimeError exit 124` as infra if Phase 1
removes the inner timeout. If a future artifact still shows exit `124`,
treat that as new evidence that a different layer is imposing timeout.
4. Do **not** classify `exit 137` as infra without inspecting task
artifacts. `137` may be OOM/SIGKILL, but it can also hide real
runtime/process bugs.

**Net product LoC estimate for optional verifier diagnostics:** ~`+8` to
`+15` YAML shell LoC.

## Alternative approaches considered

### Approach B — Workflow allowlist only

Change `.github/workflows/terminal-bench.yml` to treat `RuntimeError:
mux agent command failed ... exit 124` and/or `exit 3` as
infra/expected.

- **Net product LoC estimate:** ~`+6` to `+12` YAML shell LoC.
- **Pros:** Fastest patch; fixes the observed CI failure mode for
existing artifacts.
- **Cons:** Hides the symptom after Harbor has already marked a task as
an exception; still prevents clean scoring for semantic goal stops;
risks masking real wrapper/process failures. Not recommended except as a
short-lived emergency unblock.

### Approach C — Disable goal mode for full Terminal-Bench runs

Default `mux_run_as_goal` to false for scheduled/full-suite runs and
reserve strict goal mode for smoke tests.

- **Net product LoC estimate:** `0` to `+5` YAML/default config LoC.
- **Pros:** Avoids goal-incomplete exit issues immediately.
- **Cons:** Loses the signal we are trying to measure for goal-mode
benchmarking; does not address inner timeout exit `124`. Not recommended
as the main fix.

## Artifact investigation follow-up

Before widening any allowlist beyond semantic stops, inspect the
uploaded artifact from run `26294491586` for the three remaining
categories:

1. `exit 1` tasks: `make-mips-interpreter`, `dna-assembly`
- Inspect `agent/command-0/stderr.txt`, `stdout.txt`, `return-code.txt`,
and per-task `result.json`.
- Decide whether these are mux bugs/provider failures or expected task
failure states.
2. `exit 137` task: `caffe-cifar-10`
   - Look for OOM, sandbox kill, or process-killed messages.
- Only classify as infra if logs clearly show external kill/OOM rather
than mux crash.
3. `exit 124` tasks: `train-fasttext`, `winning-avg-corewars`
- After removing the inner timeout, rerun a smoke with timeout-shortened
tasks to confirm future timeouts become Harbor `AgentTimeoutError`.

This investigation can run in parallel with Approach A, but should not
block the minimal harness fix.

## Quality gates and validation

1. Static/script checks:
- `shellcheck benchmarks/terminal_bench/mux-run.sh` if available,
otherwise manually validate `bash -n
benchmarks/terminal_bench/mux-run.sh`.
- Targeted Python/benchmark tests for `mux_agent.py` and `mux-run.sh`
behavior.
2. Local smoke tests:
- Fake `bun` exit `3` in goal mode should return `0` and preserve token
extraction.
- Fake `bun` exit `1` should return `1` and still preserve token
extraction.
   - Fake `timeout` should not be invoked when `MUX_TIMEOUT_MS` is set.
3. CI smoke:
- Trigger `terminal-bench.yml` with a tiny task set,
`mux_run_as_goal=true`, and a low `goal_turns`/budget to exercise
goal-incomplete scoring without a full-suite spend.
- Confirm the `Run Terminal-Bench`, `Verify agent ran successfully`,
BigQuery upload (if applicable), and artifact upload steps behave as
expected.
4. Full-suite confidence:
   - Run a full `terminal-bench@2.0` job only after the smoke passes.
- Compare exception categories against the previous run; expected
improvement is fewer/no `RuntimeError exit 124` and no `RuntimeError
exit 3` if goal stops occur.

## Dogfooding / reviewer evidence

Because this is CLI/CI behavior, dogfooding evidence should include
terminal recordings plus GitHub Actions screenshots:

1. Local terminal recording:
   - Record the fake-runner smoke tests with `script` or `asciinema`.
- Capture the command, the goal-mode exit `3` normalization, the fatal
exit `1` preservation, and the no-inner-timeout check.
- Save the recording path and attach it in the final implementation
summary if the environment supports attachments.
2. CI/browser evidence:
   - Use `agent-browser` to open the smoke GitHub Actions run.
   - Capture screenshots of:
- the workflow inputs showing `mux_run_as_goal=true` and the small task
set;
     - the `Run Terminal-Bench` step summary;
     - the `Verify agent ran successfully` step output;
     - uploaded artifact presence.
- If visual verification is important, record a short browser video
navigating from the run summary to step 13.
3. Artifact evidence:
- Attach or summarize relevant snippets from `return-code.txt`,
`stderr.txt`, and result JSON for the smoke tasks.
- Confirm semantic stops are represented as scored results instead of
adapter exceptions.

## Acceptance criteria

- `mux-run.sh` no longer wraps mux with GNU `timeout`; Harbor timeout is
the only task-level timeout authority.
- Strict goal-mode mux exit `3` is treated as a benchmark-scoreable stop
in the Terminal-Bench runner, not as an adapter failure.
- Unexpected mux/process exits still fail the task and surface through
`MuxAgent.run()` as `RuntimeError`.
- Token/cost extraction still runs for both scoreable semantic stops and
fatal exits.
- Tests cover goal exit normalization, fatal exit preservation, and no
inner timeout invocation.
- CI smoke evidence includes screenshots and a terminal/video recording
as described above.
- The full workflow verifier remains strict enough to catch genuine
non-infra mux execution failures.

</details>

---

_Generated with `mux` • Model: `openai:gpt-5.5` • Thinking: `xhigh` •
Cost: `$25.52`_

<!-- mux-attribution: model=openai:gpt-5.5 thinking=xhigh costs=25.52
-->