The three court-documented incidents that anchor the Tier 1 canonical corpus. Each is a real legal filing in which AI-generated citations failed under judicial scrutiny.
The narrative below is the human-readable version. The machine-readable evidence artifact for each case lives in data/benchmark/tier1/corpus/citation_failure_cases.json.
Case ID: mata-v-avianca-2023
Source: CourtListener docket 63107798
Failure class: nonexistent_authority, reconstructability_failure
Counsel for the plaintiff submitted a brief opposing a motion to dismiss in a personal-injury matter. The brief cited multiple federal appellate decisions, including Varghese v. China Southern Airlines Co., 925 F.3d 1339 (11th Cir. 2019). When opposing counsel attempted to locate the cited authority, they could not. The decisions did not exist.
Counsel had used ChatGPT to draft portions of the brief and had not independently verified the citations. The model had fabricated case captions, reporter citations, and even apparent quotations from the non-existent opinions. When pressed, ChatGPT confirmed the fabricated citations existed — they did not.
The court issued sanctions under Federal Rule 11.
The failure pattern is not "the citation looked wrong." It is "the workflow that produced the citation could not be reconstructed, audited, or defended." That is the property Dali measures.
The Tier 1 record encodes:
- The fabricated citation string
- The actual status (
nonexistent_authority) - The failure class taxonomy entry
- The source URL to the sanctions order
- The retrieval date and policy version
Run python -m dali.runners.run_integrity --corpus data/benchmark/tier1/corpus/citation_failure_cases.json and you will get a deterministic CitationIntegrityResult artifact for this case, hash-sealed and replayable.
A citation can be syntactically perfect, reporter-conformant, and quotation-rich — and still be entirely fabricated. Pre-AI citation-checking tooling does not catch this failure mode because it assumes the citation refers to some real authority. Dali assumes nothing.
Case ID: us-v-cohen-2023
Source: CourtListener docket 8009608
Failure class: nonexistent_authority, provenance_gap
Michael Cohen's attorney submitted a motion for early termination of supervised release. The motion cited three federal decisions in support of the request. None of the three existed.
The fabricated citations had been generated by Google Bard (now Gemini) and passed to the attorney by Cohen himself, with the attorney filing the brief without independent verification.
The court documented the incident in a published order and required affidavits explaining the provenance of the citations.
It is the canonical example of provenance gap: the workflow that produced the citation could not be reconstructed even after the fact. The attorney did not know which model generated which citation, what prompt was used, whether any verification step occurred, or who in the workflow was responsible.
This is the failure mode Dali was designed for. A defensibility_risk: critical verdict means precisely this: no audit trail exists, no reconstructability path is available, no Rule 11 defense is plausible.
Citation integrity is not a model problem. It is a workflow problem. Even if the underlying model is improved, the absence of a reconstructable provenance chain means the next incident is unavoidable. Dali measures the workflow gap, not just the model gap.
Case ID: park-v-kim-2024
Source: Second Circuit opinion documenting the citation failure
Failure class: nonexistent_authority, reconstructability_failure
In a federal appellate matter, counsel cited a non-existent decision in a reply brief. The Second Circuit identified the citation as fabricated and referred counsel for disciplinary action. The court's opinion included an extended discussion of the obligations of counsel using generative AI.
Notably, this case post-dates Mata by over a year. The pattern was already well-publicized in the legal press. The fabrication still made it into a filed appellate brief.
It demonstrates that awareness of the failure mode is not sufficient to prevent recurrence. Practitioners knew about Mata. They were sanctioned anyway. This is what makes the case useful as benchmark evidence: it falsifies the hypothesis that the problem resolves itself through publicity.
The Dali artifact for Park v. Kim carries the same deterministic verdict as Mata: verification: FAILED, recoverability: infeasible, risk: critical.
The failure pattern is structural. It will not resolve through education, advisories, or even prior sanctions. It resolves only when (a) the workflow is reconstructable end-to-end, and (b) verification is enforced as a non-skippable step rather than a recommended best practice.
Dali is the open infrastructure for measuring whether that structural change is happening.
The Tier 1 corpus currently has 3 scoring-eligible cases. The public record contains many more — sanctions orders, judicial findings, disciplinary referrals, court-documented retrieval failures across U.S. state and federal courts and increasingly internationally. Damien Charlotin's AI hallucination tracker maintains the canonical public list.
Corpus expansion is the highest-priority contribution track in the project. If you know of a court-documented incident that meets the Tier 1 sourcing standard — verifiable source URL, retrieval timestamp, reproducible verification path, stable citation metadata — see docs/for-legal-practitioners.md for the 30-minute contribution path.
Every additional case strengthens the benchmark's evidentiary thesis. The corpus is the network effect.