Skip to content

Commit c3e4221

Browse files
committed
Add ZeroGPT
1 parent 8721f24 commit c3e4221

8 files changed

Lines changed: 886 additions & 4 deletions

File tree

EVAL.md

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
# EVAL - Quality Evaluation Framework for HUMAN-AI
22

3+
> **NEW in v3.0:** ZeroGPT external validator integration. Run `scripts/run-benchmark.ps1` with a ZeroGPT API key to get independent AI detection scores.
4+
35
> Load this prompt alongside SKILL.md output to score the humanization result.
46
> The evaluator is a separate prompt to avoid self-assessment bias.
57
@@ -166,3 +168,70 @@ HUMAN-AI OUTPUT:
166168
```
167169

168170
The evaluator must have access to `shared/tone-profiles.md` and `shared/rhythm-tables.md` for accurate scoring.
171+
172+
---
173+
174+
## External Validation — ZeroGPT API
175+
176+
For **independent** validation (not LLM self-assessment), use ZeroGPT's external AI detection API.
177+
178+
### Prerequisites
179+
180+
1. ZeroGPT API key (register at [zerogpt.com](https://www.zerogpt.com) → Dashboard → API)
181+
2. Set environment variable: `ZEROGPT_API_KEY="your-key"`
182+
183+
### Quick single-text check
184+
185+
```bash
186+
# PowerShell
187+
.\scripts\zerogpt-detect.ps1 -Text "Text to analyze"
188+
189+
# Bash
190+
./scripts/zerogpt-detect.sh --text "Text to analyze"
191+
192+
# From file
193+
.\scripts\zerogpt-detect.ps1 -File "path/to/text.md"
194+
```
195+
196+
### Full benchmark run
197+
198+
```bash
199+
# Dry run (see what would be tested)
200+
.\scripts\run-benchmark.ps1 -SkipApi
201+
202+
# Full run with ZeroGPT API
203+
$env:ZEROGPT_API_KEY = "your-key"
204+
.\scripts\run-benchmark.ps1
205+
```
206+
207+
Results are saved to `tests/benchmark/zerogpt-results.json`.
208+
209+
### Interpreting ZeroGPT scores
210+
211+
| ZeroGPT Score | Meaning | Action |
212+
|--------------|---------|--------|
213+
| 0-10% | Human-written | Pre-flight guard should have STOPPED |
214+
| 10-25% | Mostly human | Acceptable output |
215+
| 25-50% | Mixed signals | Some AI patterns remain |
216+
| 50-80% | Likely AI | Re-run pipeline with aggressive cleanup |
217+
| 80-100% | Heavy AI | Pipeline did not work — adjust parameters |
218+
219+
### Combined Evaluation Flow
220+
221+
```
222+
1. Run skill → get self-evaluated [QUALITY: XX/100]
223+
2. Run ZeroGPT on output → get external AI probability
224+
3. Run EVAL.md LLM evaluator → get independent 5-metric score
225+
4. Triangulate: if all three agree → high confidence in quality
226+
```
227+
228+
### Exit codes
229+
230+
| Code | Meaning |
231+
|------|---------|
232+
| 0 | Text classified as human / mixed |
233+
| 10 | Text classified as AI-generated |
234+
| 1 | Input error |
235+
| 2 | Missing API key |
236+
| 3 | API connection error |
237+
| 4 | API returned error |

README.md

Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,18 @@ natural-skill/
9191
│ ├── specificity-ladder.md ← Abstraction → concrete framework
9292
│ ├── rhythm-tables.md ← Sentence flow parameters
9393
│ └── language-template.md ← Template for adding new languages
94+
├── scripts/ ← Automation scripts
95+
│ ├── validate.ps1 ← Integrity checker (PowerShell)
96+
│ ├── validate.sh ← Integrity checker (Bash)
97+
│ ├── zerogpt-detect.ps1 ← ZeroGPT AI detection (PowerShell)
98+
│ ├── zerogpt-detect.sh ← ZeroGPT AI detection (Bash)
99+
│ └── run-benchmark.ps1 ← External benchmark runner
100+
├── tests/benchmark/ ← Evaluation dataset
101+
│ ├── annotations.json
102+
│ ├── zerogpt-results.json ← ZeroGPT external validation results
103+
│ ├── ai-texts/ (9 languages)
104+
│ ├── human-texts/
105+
│ └── edge-cases/
94106
├── scenarios/ ← Task-specific playbooks
95107
│ ├── full-rewrite.md ← Default: all 6 stages
96108
│ ├── blog-post.md ← Blog post humanization
@@ -189,7 +201,35 @@ Here's what the four good ones have in common. And why the other eight failed.
189201
- Any capable LLM with a system prompt / custom instructions field
190202
- For full skill functionality: a skill system that loads files from a folder (OpenCode, Claude Code)
191203
- For standalone: copy `SKILL.md` as system prompt - contains complete pipeline
192-
- No API keys, no tools, no dependencies - pure prompt engineering
204+
- **External validation:** ZeroGPT API key (optional) — for independent AI detection scoring via `scripts/run-benchmark.ps1`
205+
206+
## Quality Validation
207+
208+
HUMAN-AI provides **three layers** of quality verification:
209+
210+
| Layer | Method | Independence | How to run |
211+
|-------|--------|-------------|------------|
212+
| 1. Self-eval | Stage 5.6 Quality Score | LLM self-assessment | Built into pipeline |
213+
| 2. EVAL.md | 5-metric LLM evaluation | Separate LLM call | `EVAL.md` prompt |
214+
| 3. ZeroGPT | External API detection | Fully independent | `scripts/run-benchmark.ps1` |
215+
216+
**ZeroGPT external validation:**
217+
218+
```bash
219+
# Set API key
220+
$env:ZEROGPT_API_KEY = "your-key"
221+
222+
# Single text check
223+
.\scripts\zerogpt-detect.ps1 -File "output.md"
224+
225+
# Full benchmark run (all test texts)
226+
.\scripts\run-benchmark.ps1 -MaxTexts 10 -DelaySeconds 2
227+
228+
# Dry run (see what would be tested)
229+
.\scripts\run-benchmark.ps1 -SkipApi
230+
```
231+
232+
Results saved to `tests/benchmark/zerogpt-results.json`.
193233

194234
---
195235

SKILL.md

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -367,6 +367,8 @@ If QUALITY SCORE < 75: identify the 2 lowest-scoring checks. Re-run ONLY the rel
367367

368368
After the single re-loop, output the final score regardless. Stop after 2 passes total.
369369

370+
**External validation:** For independent verification, run ZeroGPT detection via `scripts/zerogpt-detect.ps1` or `scripts/run-benchmark.ps1`. See `EVAL.md` — External Validation section for details. The self-evaluation score is an LLM self-assessment. ZeroGPT provides an independent external check.
371+
370372
---
371373

372374
## WHEN NOT TO APPLY
@@ -495,11 +497,15 @@ natural-skill/
495497
├── .gitignore
496498
├── scripts/
497499
│ ├── validate.ps1 ← Integrity checker (PowerShell)
498-
│ └── validate.sh ← Integrity checker (Bash)
500+
│ ├── validate.sh ← Integrity checker (Bash)
501+
│ ├── zerogpt-detect.ps1 ← ZeroGPT AI detection (PowerShell)
502+
│ ├── zerogpt-detect.sh ← ZeroGPT AI detection (Bash)
503+
│ └── run-benchmark.ps1 ← External benchmark runner (ZeroGPT)
499504
├── .github/workflows/
500505
│ └── validate.yml ← CI pipeline
501506
├── tests/benchmark/ ← Evaluation dataset
502507
│ ├── annotations.json
508+
│ ├── zerogpt-results.json ← ZeroGPT external validation results
503509
│ ├── ai-texts/ (9 languages)
504510
│ ├── human-texts/
505511
│ └── edge-cases/

0 commit comments

Comments
 (0)