Skip to content

Add Codex plugin metadata and Svix Skill evals#11

Open
rforgeon wants to merge 2 commits into
svix:mainfrom
rforgeon:codex/add-plugin-evals
Open

Add Codex plugin metadata and Svix Skill evals#11
rforgeon wants to merge 2 commits into
svix:mainfrom
rforgeon:codex/add-plugin-evals

Conversation

@rforgeon

Copy link
Copy Markdown

Summary

  • add Codex-compatible plugin metadata for the existing Svix Skills package
  • document the plugin vs Skill distinction in the README
  • add JSONL eval cases for Dispatch planning, webhook receiver review, and CLI quickstart verification

The plugin is the installable package for agent workspaces. Each Svix Skill remains a focused SKILL.md workflow inside that plugin.

Privacy / telemetry

The eval docs explicitly avoid prompts, source files, webhook payloads, API keys, signing secrets, tenant identifiers, tool arguments, and model outputs.

Validation

node -e "JSON.parse(require('fs').readFileSync('.codex-plugin/plugin.json','utf8')); const lines=require('fs').readFileSync('evals/svix-agent-skills/cases.jsonl','utf8').trim().split(/\n/); lines.forEach(JSON.parse); console.log('valid plugin json and jsonl lines='+lines.length);"

Output:

valid plugin json and jsonl lines=3

@svix-pluiz svix-pluiz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @rforgeon, thanks for contributing!
Before we merge this, could you address my comments?

Comment thread README.md Outdated
Comment on lines +42 to +52
## Telvine packaging

If this plugin is published through Telvine, publish the repository as the plugin
and treat each Svix Skill as a plugin component:

```bash
npm i -g telvine
telvine login
telvine publish .
```

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we're not using Telvine, could you remove this section?

Suggested change
## Telvine packaging
If this plugin is published through Telvine, publish the repository as the plugin
and treat each Svix Skill as a plugin component:
```bash
npm i -g telvine
telvine login
telvine publish .
```

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you explain where and when this skill eval script is meant to run? It feels like this is out of the scope for this project

@rforgeon

Copy link
Copy Markdown
Author

Thanks for the review. I pushed e151b42 addressing both points:

  • removed the Telvine packaging sections from the README and eval README
  • clarified that the JSONL evals are seed cases for external agent harness validation, not a Svix runtime script or part of the repo's application test suite

I also revalidated the manifest and JSONL locally:

valid plugin json and jsonl lines=3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants