Accuracy on the 20-question LegalMCQA benchmark (single-answer, 4 options). Because n=20, always report the 95% confidence interval — point accuracies alone are not meaningful at this scale.
The rows below are the reference baselines shipped with the toolkit. They are
produced by python scripts/run_benchmark.py and are exactly reproducible.
There are no model results here that we did not run ourselves — submit
yours via pull request (see below).
| Rank | Method | Type | Accuracy | 95% CI (Wilson) | Reproduce |
|---|---|---|---|---|---|
| 1 | tfidf_cooccurrence | unsupervised lexical | 0.450 | [0.258, 0.658] | legalmcqa.evaluate baseline --name tfidf_cooccurrence |
| 1 | word_overlap | unsupervised lexical | 0.450 | [0.258, 0.658] | legalmcqa.evaluate baseline --name word_overlap |
| 3 | first_option | positional heuristic | 0.350 | [0.181, 0.567] | legalmcqa.evaluate baseline --name first_option |
| 4 | longest_option | length heuristic | 0.300 | [0.145, 0.519] | legalmcqa.evaluate baseline --name longest_option |
| - | random | chance | 0.247 | ~0.25 (analytic) | empirical over 2000 seeds |
These standard methods are not filled in because we have not run them in this repository — submitting them is a great first contribution:
- Sentence-Transformer (e.g.
all-MiniLM-L6-v2) cosine baseline - BERT / Legal-BERT zero-shot and fine-tuned (note: no train split — use as encoder)
- Longformer / long-context encoders
- Retrieval-augmented (BM25 + reader)
- LLM zero-shot and few-shot (script:
scripts/llm_baseline.py)
- Generate predictions as
.jsonl({"id", "prediction", "confidence"}). - Evaluate:
python -m legalmcqa.evaluate predict --file my_preds.jsonl --json my_result.json. - Open a PR adding a row with method name, type, accuracy, 95% CI, and a link to code that reproduces it. Self-reported numbers must be reproducible.