Skip to content

Name the callee zero test cannot execute in test failures - #429

Open
guygrigsby wants to merge 1 commit into
vercel-labs:mainfrom
guygrigsby:test-runner-unknown-callee
Open

Name the callee zero test cannot execute in test failures#429
guygrigsby wants to merge 1 commit into
vercel-labs:mainfrom
guygrigsby:test-runner-unknown-callee

Conversation

@guygrigsby

Copy link
Copy Markdown

Makes zero test failures name the callee the runner could not execute instead of appending only the test label. Addresses the diagnostic confusion in #428; the underlying limitation (the test interpreters execute only std.testing helpers and pure zerolang functions) is unchanged.

Before:

zero test unknown function: my test

After, for a callee that does not resolve:

zero test unknown function 'myHelper': my test

After, for a real stdlib function the runner does not execute, matching the existing unsupported-construct wording:

zero graph test runner does not implement 'std.mem.span' yet; tests can call std.testing helpers and pure zerolang functions; use live smoke tests for other std calls: std call coverage

Changes:

  • Thread the callee name and call node into test_eval_function and pgt_eval_function so the existing null-function guard is the single place the message is produced, in both interpreters. Failures now anchor to the unresolved call's location instead of the test function's.
  • std.-prefixed callees get the not-implemented framing; other unresolved names keep unknown function '<name>'.
  • Rewrite the skill-data/testing.md paragraph that read this failure as a malformed test label and advised deleting the test; regenerated embedded_skills.inc is included.
  • Add a command contract pinning the message and failure.sourceFile for a package test reaching std.mem.*, and adjust two file line budgets in scripts/compiler-metrics.mts.

Verified locally on darwin-arm64: make -C native/zero-c, pnpm run command-contracts:local, pnpm run conformance:local, pnpm run native:test:local, pnpm run native:sanitize, pnpm run workspace:checks all pass. The :sandbox variants were not run locally (they require VERCEL_OIDC_TOKEN); CI covers those paths.

Both test interpreters failed with only the test label appended, e.g.
"zero test unknown function: my test", which reads as a broken label
and hides the actual missing function. Thread the callee name and call
node into test_eval_function and pgt_eval_function so the existing
null-function guard reports what failed and where:

  zero test unknown function 'myHelper': my test

std.* callees outside std.testing get not-implemented framing instead,
matching the runner's existing unsupported wording, since those are
real stdlib functions the test interpreters do not execute:

  zero graph test runner does not implement 'std.mem.span' yet; ...

Update the stale testing skill guidance that misread this failure as a
malformed test label, add a command contract pinning the message and
location, and adjust two file line budgets.
@vercel

vercel Bot commented Jul 11, 2026

Copy link
Copy Markdown

@guygrigsby is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant