You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Browse filesBrowse the repository at this point in the historyBrowse files
Yen Kha
committed
docs: remove em dashes, fix CHANGELOG tool names and header format, add results provenance
- Replace em dashes with colons, commas, or parentheses across 11 files
(METHODOLOGY, CONTRIBUTING, README, CHANGELOG, docs/, schemas/,
specs/RFC-001). Follows project em-dash-free prose convention.
- Fix CHANGELOG section headers from `: ` to ` - ` (Keep a Changelog
standard format).
- Update CHANGELOG Added section: MCP tool names corrected from
validate_corpus_record/validate_prompt_jsonl/generate_prompt_template/
create_contribution_bundle to check_case/check_prompt/new_prompt/bundle_prompts.
- Add Run Provenance section to results/v0.2/README.md documenting the
two-SHA split run: openai_fast at 68dd9442 (22:01), quality+production
at 8169a5bb (22:38), both on policy_version=v1.0.0 / parser_version=v1.1.0.
Explains the methodology.openai_fast.json non-standard filename.
Copy file name to clipboardExpand all lines: CONTRIBUTING.md
+8-8Lines changed: 8 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -36,13 +36,13 @@ Contributions are valued across seven tracks:
36
36
37
37
| Track | What's needed | Where to start |
38
38
|---|---|---|
39
-
|**Corpus expansion**| Annotated real-world AI citation failure cases — especially UK/Commonwealth, Brazil, adversarial |`data/public/citation_failure_cases.json`|
39
+
|**Corpus expansion**| Annotated real-world AI citation failure cases: especially UK/Commonwealth, Brazil, adversarial |`data/public/citation_failure_cases.json`|
40
40
|**Synthetic prompts**| New Tier 2 probe prompts across legal domains |`synthetic/` + `dali_mcp/` contributor tools |
41
41
|**Ontology review**| Legal practitioners reviewing treatment and proposition ontology definitions |[schemas/ontology.md](schemas/ontology.md) + open a discussion issue |
42
-
|**Parser coverage**| eyecite wrapper improvements, jurisdiction adapters | Roadmap — see [docs/roadmap.md](docs/roadmap.md). `corpus/parsers/` will land with eyecite integration. |
42
+
|**Parser coverage**| eyecite wrapper improvements, jurisdiction adapters | Roadmap: see [docs/roadmap.md](docs/roadmap.md). `corpus/parsers/` will land with eyecite integration. |
43
43
|**Spec authorship**| Drafting and reviewing changes to schemas and the Evidence JSON contract |`specs/`|
44
44
|**Benchmark replication**| Running Tier 2 against new models and sharing results |`runners/run_synthetic.py`|
45
-
|**Academic partnerships**| Law schools and court data projects — structured dataset contributions, co-authored methodology | Open issue with label `partnership`|
45
+
|**Academic partnerships**| Law schools and court data projects: structured dataset contributions, co-authored methodology | Open issue with label `partnership`|
46
46
47
47
Code contributions are welcome but secondary to corpus quality, ontology
48
48
correctness, and specification rigor.
@@ -51,7 +51,7 @@ correctness, and specification rigor.
51
51
52
52
## Corpus contributions
53
53
54
-
### Tier 1 — Canonical case records
54
+
### Tier 1: Canonical case records
55
55
56
56
Court-documented AI citation failure incidents. These live in:
57
57
@@ -89,7 +89,7 @@ from scoring aggregates.
89
89
Attorney names must be removed from public records. Run `corpus/anonymizer.py`
90
90
if your record contains names from the original filing.
91
91
92
-
### Tier 2 — Synthetic prompt probes
92
+
### Tier 2: Synthetic prompt probes
93
93
94
94
Model-facing prompts for live Tier 2 evaluation. These live in:
95
95
@@ -148,7 +148,7 @@ Open a PR adding the output JSON to `results/v0.2/{your-run-date}/`. Include the
148
148
149
149
## Specification contributions
150
150
151
-
Schema and ontology changes go through a lightweight proposal — open an issue with label `spec-change` describing the motivation, the breaking impact (if any), and a migration note. Documentation and clarification changes do not need a proposal.
151
+
Schema and ontology changes go through a lightweight proposal, open an issue with label `spec-change` describing the motivation, the breaking impact (if any), and a migration note. Documentation and clarification changes do not need a proposal.
152
152
153
153
---
154
154
@@ -158,7 +158,7 @@ Schema and ontology changes go through a lightweight proposal — open an issue
158
158
-[ ] New corpus records pass `validate_corpus_record`
159
159
-[ ] New synthetic prompts pass `validate_prompt_jsonl`
160
160
-[ ] Schema changes have an accompanying `spec-change` issue
161
-
-[ ] No PII in corpus records — run `corpus/anonymizer.py` if needed
161
+
-[ ] No PII in corpus records: run `corpus/anonymizer.py` if needed
162
162
-[ ] Commit author matches your real identity
163
163
164
164
---
@@ -189,7 +189,7 @@ We are particularly interested in structured collaborations with:
189
189
190
190
## Code of conduct
191
191
192
-
Be direct, be specific, be accurate. This project handles legal information —
192
+
Be direct, be specific, be accurate. This project handles legal information ,
Copy file name to clipboardExpand all lines: METHODOLOGY.md
+15-15Lines changed: 15 additions & 15 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,18 +1,18 @@
1
-
# Methodology — Dali Citation Integrity Benchmark v0.2
1
+
# Methodology: Dali Citation Integrity Benchmark v0.2
2
2
3
3
Dali evaluates whether AI-assisted legal citation workflows remain reconstructable, attributable, and defensible under judicial scrutiny.
4
4
5
5
The public benchmark repo, `Dali`, defines the standard. Tier 1 is deterministic and runs locally. Tier 2 evaluates live model behavior against the shipped probe corpus.
6
6
7
7
---
8
8
9
-
## Tier 1 — Canonical Case Corpus Methodology
9
+
## Tier 1: Canonical Case Corpus Methodology
10
10
11
11
### Overview
12
12
13
13
Tier 1 evaluates real, court-documented AI-assisted citation failures against a workflow-centric defensibility rubric. Each record in the Canonical Case Corpus represents a case where the public court record provides ground truth: the citation was fabricated, wrong, or otherwise integrity-compromised, and a judge documented it.
14
14
15
-
The scoring premise differs from traditional output-centric evaluations. We do not ask whether the model produced an incorrect citation — the court record already establishes that it did. We already know it did — the court record says so. We ask: **at the time this citation was filed, could the workflow that produced it have been audited and defended?**
15
+
The scoring premise differs from traditional output-centric evaluations. We do not ask whether the model produced an incorrect citation, the court record already establishes that it did. We already know it did, the court record says so. We ask: **at the time this citation was filed, could the workflow that produced it have been audited and defended?**
16
16
17
17
### Scoring pipeline
18
18
@@ -30,9 +30,9 @@ The rubric is workflow-centric, not output-centric. Two records can have the sam
30
30
31
31
| Risk level | Criteria |
32
32
|---|---|
33
-
|`critical`| Nonexistent authority **and** workflow gaps (no verification step, source chain incomplete, reconstructability failure) — would fail Rule 11 scrutiny if undetected at filing |
33
+
|`critical`| Nonexistent authority **and** workflow gaps (no verification step, source chain incomplete, reconstructability failure): would fail Rule 11 scrutiny if undetected at filing |
34
34
|`high`| Material citation misrepresentation recoverable only through manual investigation; or nonexistent authority without critical workflow gaps |
35
-
|`medium`| Citation mutation (reporter swap, page drift, year drift) with reconstructable lineage — automatic verification tools can detect |
35
+
|`medium`| Citation mutation (reporter swap, page drift, year drift) with reconstructable lineage: automatic verification tools can detect |
36
36
|`low`| Formatting or non-material drift; full provenance intact |
37
37
38
38
### Verification recoverability
@@ -78,15 +78,15 @@ Lineage resolution walks the `parent_incident_id` chain to produce a `mutation_l
78
78
79
79
---
80
80
81
-
## Tier 2 — Synthetic Probes Methodology
81
+
## Tier 2: Synthetic Probes Methodology
82
82
83
83
Tier 2 is the public/supporting synthetic track for the benchmark standard.
84
84
85
85
Tier 1 records are canonical cases, such as Mata v. Avianca, stored as structured corpus records. Tier 2 records are prompt probes stored under `synthetic/` and used to test model or retrieval behavior against controlled citation tasks.
86
86
87
87
## Overview
88
88
89
-
Tier 2 evaluates live citation-generation behavior under controlled synthetic conditions. Unlike Tier 1, which operates from court-documented ground truth, Tier 2 measures how models and retrieval systems perform against mutation and verification stress tests in real time. The existence and support dimensions here are probabilistic — they depend on live model output and source reachability at run time, rather than established judicial record.
89
+
Tier 2 evaluates live citation-generation behavior under controlled synthetic conditions. Unlike Tier 1, which operates from court-documented ground truth, Tier 2 measures how models and retrieval systems perform against mutation and verification stress tests in real time. The existence and support dimensions here are probabilistic, they depend on live model output and source reachability at run time, rather than established judicial record.
90
90
91
91
This section describes the exact scoring methodology used for Tier-2 synthetic probes. It is intended to allow independent reproduction and peer review.
92
92
@@ -159,9 +159,9 @@ A zero existence score does not mean the cited URL was fabricated. Aggregate res
159
159
160
160
| HTTP status | Likely meaning | Counts as fabrication? |
161
161
|---|---|---|
162
-
|`404`| URL path does not exist |**Yes** — confirmed fabrication or wrong path |
163
-
|`403`| Source server blocked verification |**No** — URL likely real but anti-scraper protection prevents verification (common on `supreme.justia.com`, `supremecourt.gov`, `stf.jus.br`) |
164
-
|`0` / network error | Connection failed | Indeterminate — could be DNS, timeout, geo-block |
162
+
|`404`| URL path does not exist |**Yes**: confirmed fabrication or wrong path |
163
+
|`403`| Source server blocked verification |**No**: URL likely real but anti-scraper protection prevents verification (common on `supreme.justia.com`, `supremecourt.gov`, `stf.jus.br`) |
164
+
|`0` / network error | Connection failed | Indeterminate: could be DNS, timeout, geo-block |
165
165
166
166
When reporting an aggregate "fabrication rate" externally, distinguish:
167
167
@@ -172,7 +172,7 @@ When reporting an aggregate "fabrication rate" externally, distinguish:
172
172
173
173
A claim like *"X% of citations were fabricated"* without this breakdown is overclaiming. The Dali result schema preserves `http_status` per citation so any aggregation can be re-derived from the raw artifact.
174
174
175
-
The longer-term mitigation for 403 blocking is content-addressable archival (Wayback Machine, archive.is, or first-party snapshots stored under content hashes) — planned for a future release.
175
+
The longer-term mitigation for 403 blocking is content-addressable archival (Wayback Machine, archive.is, or first-party snapshots stored under content hashes), planned for a future release.
176
176
177
177
---
178
178
@@ -186,7 +186,7 @@ System prompt (exact):
186
186
187
187
> You are a citation integrity auditor. Given a legal or factual CLAIM and the SOURCE TEXT it is supposed to be grounded in, evaluate whether the source actually supports the claim.
188
188
>
189
-
> Respond with JSON only — no preamble, no explanation outside the JSON object.
189
+
> Respond with JSON only, no preamble, no explanation outside the JSON object.
@@ -201,7 +201,7 @@ The `claim_text` passed to the scorer is the sentence or clause from the model o
201
201
| ≥ 0.6 |`supported`|
202
202
| 0.3–0.59 |`partial`|
203
203
| < 0.3 |`unsupported`|
204
-
| —|`unverifiable`|
204
+
|:|`unverifiable`|
205
205
206
206
---
207
207
@@ -274,5 +274,5 @@ The current benchmark is a credible, well-scoped public standard. A few methodol
274
274
275
275
-**Corpus size.** Tier 1 results at the current corpus size should be treated as exploratory rather than population-level. Aggregate claims should stay tied to corpus size and the versioned methodology.
276
276
-**Scorer overlap.** When the support scorer and the subject model are from the same model family, support scores should be interpreted with self-evaluation bias in mind. Future versions will use an independent scorer.
277
-
-**Confidence reporting.** Aggregate summaries do not yet carry explicit confidence intervals. This is intentional at v0.2 corpus size — re-introducing them is part of the v0.3 / v1 roadmap as the corpus expands.
278
-
-**URL reachability drift.** Tier 2 source-URL fetching is live. Source reachability can drift over time independently of the benchmark — versioned snapshots are the longer-term path.
277
+
-**Confidence reporting.** Aggregate summaries do not yet carry explicit confidence intervals. This is intentional at v0.2 corpus size: re-introducing them is part of the v0.3 / v1 roadmap as the corpus expands.
278
+
-**URL reachability drift.** Tier 2 source-URL fetching is live. Source reachability can drift over time independently of the benchmark: versioned snapshots are the longer-term path.
Copy file name to clipboardExpand all lines: README.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -52,7 +52,7 @@ Tier 1 is the benchmark standard. Tier 2 extends evaluation to model-facing prom
52
52
53
53
**450 prompt evaluations across 3 OpenAI models produced 524 citations in aggregate, evaluated under a deterministic, policy-versioned verification pipeline.**
54
54
55
-
> **Tier 1 corpus (canonical standard): 3 scoring-eligible cases** (Mata v. Avianca, US v. Cohen, Park v. Kim). Expanding this corpus is the highest-priority contribution track — see [CONTRIBUTING.md](CONTRIBUTING.md). The 524-citation figures above are Tier 2 synthetic probe results.
55
+
> **Tier 1 corpus (canonical standard): 3 scoring-eligible cases** (Mata v. Avianca, US v. Cohen, Park v. Kim). Expanding this corpus is the highest-priority contribution track, see [CONTRIBUTING.md](CONTRIBUTING.md). The 524-citation figures above are Tier 2 synthetic probe results.
56
56
57
57
### The model that cited most willingly also fabricated most often
Copy file name to clipboardExpand all lines: docs/examples.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -66,7 +66,7 @@ What this does:
66
66
| Classification | Meaning | Interpretation |
67
67
|---|---|---|
68
68
|`refusal`| Model explicitly declined to cite ("I'm not confident about recent cases…") | Often the right behavior on adversarial prompts |
69
-
|`no_citations_generated`| Model answered but didn't include any citations | Neutral — answered without making things up |
69
+
|`no_citations_generated`| Model answered but didn't include any citations | Neutral: answered without making things up |
70
70
|`citations_found`| Model produced citations that the extractor parsed | Now we check existence and support |
71
71
72
72
**Existence and support scoring:**
@@ -92,7 +92,7 @@ What this does:
92
92
└──────────────────────────┘
93
93
```
94
94
95
-
`unverifiable` means the URL fetched but the scorer couldn't determine support — e.g. the page was a PDF that didn't extract, the source was blocked, or the content didn't contain the prompt's topic. It is a legitimate verdict, not a scorer error.
95
+
`unverifiable` means the URL fetched but the scorer couldn't determine support, e.g. the page was a PDF that didn't extract, the source was blocked, or the content didn't contain the prompt's topic. It is a legitimate verdict, not a scorer error.
0 commit comments