A small, rigorously documented open benchmark for legal multiple-choice question answering (legal MCQA), with a reproducible evaluation toolkit, baselines, and explainability support.
LegalMCQA accompanies the paper Multiple Choice Question Answering in the Legal Domain Using Reinforced Co-occurrence (DEXA 2019). This repository extends the original dataset into a small but fully reproducible legal QA benchmark: a labelled dataset in multiple formats, an installable evaluation toolkit, real unsupervised baselines, auto-generated figures and reports, and a leaderboard.
Scope, stated honestly. This is an evaluation-only benchmark of 20 questions. It is deliberately not split into train/validation/test, and we do not publish fabricated baseline numbers or significance claims. At n=20, always read the confidence intervals the toolkit reports — point accuracies alone are not meaningful. It is a high-signal probe and teaching resource, not a large-scale training corpus.
- What problem does this solve?
- Why legal MCQA matters
- Coverage
- Install
- Load the dataset
- Evaluate a model
- Reproduce the benchmark
- Baseline results
- Explainability
- Add new questions
- Cite
- License
- Research that cites this work
Legal question answering systems are hard to compare because legal text is specialised, jurisdiction-dependent, and full of fine distinctions. LegalMCQA provides a clean, labelled, multiple-choice target where the correct answer is unambiguous to score (single correct option) while the content remains genuinely legal. It lets you measure, with one command, how well a method — lexical, embedding-based, retrieval-augmented, BERT/Legal-BERT, or an LLM — answers legal questions, and where it fails (by jurisdiction, topic, and difficulty).
Multiple-choice questions are how law students and professionals are actually examined, so legal MCQA is a realistic, high-stakes proxy for legal knowledge. It is also a clean evaluation format for legal NLP: scoring is objective, distractors probe genuine misconceptions, and per-question metadata enables fine-grained error analysis and explainable legal QA. This makes it useful for benchmarking Legal-BERT, retrieval-augmented legal reasoning, and LLM legal AI evaluation, and for classroom use in NLP, legal informatics, and information retrieval courses.
20 questions · 4 options each · exactly one correct answer · English.
Jurisdiction
| Jurisdiction | Questions |
|---|---|
| United Kingdom | 13 |
| European Union | 2 |
| Roman law (historical) | 2 |
| Council of Europe | 1 |
| Public international law | 1 |
| General / comparative | 1 |
Difficulty (heuristic — see caveat): easy 7 · medium 5 · hard 5 · expert 3.
Topics span 17 distinct legal areas, including civil and criminal procedure, contract law, company law, employment & restitution law, constitutional law, EU institutions, human rights, public international law, and Roman legal history (the Digest).
Full per-question metadata and an auto-generated audit live in
reports/validation_report.md. Distribution
figures are in reports/figures/.
git clone https://github.com/jorge-martinez-gil/lmcqa.git
cd lmcqa
pip install -e ".[dev]" # core is NumPy-only; figures use matplotlibNo GPU and no model downloads are required for the dataset or the baselines.
Three equivalent formats are provided (all generated from one source):
dataset.json, data/legal_mcqa.jsonl, data/legal_mcqa.csv. See
data/schema.md for every field.
# With the bundled toolkit
from legalmcqa import load_dataset
questions = load_dataset()
q = questions[0]
print(q.question, "->", q.answer, "|", q.jurisdiction, q.difficulty)# With HuggingFace datasets
from datasets import load_dataset
ds = load_dataset("json", data_files="data/legal_mcqa.jsonl", split="train")Produce a predictions file (.jsonl), one object per line. prediction may be
a letter, a 0-based index, or the exact option text; confidence is optional
(used for calibration metrics):
{"id": 1, "prediction": "C", "confidence": 0.81}
{"id": 2, "prediction": 0}python -m legalmcqa.evaluate predict --file my_preds.jsonlYou get accuracy with 95% Wilson and bootstrap confidence intervals, macro and weighted F1, calibration error (ECE) when confidences are supplied, and breakdowns by difficulty, jurisdiction, and question type.
LLM baselines (zero-/few-shot) use your own API key:
export ANTHROPIC_API_KEY=... # or OPENAI_API_KEY
python scripts/llm_baseline.py --provider anthropic --model <model> --shots 0 --out llm_preds.jsonl
python -m legalmcqa.evaluate predict --file llm_preds.jsonlEverything (formats, baselines, figures, tables, reports) regenerates with one command — no numbers are hand-entered:
python scripts/run_benchmark.py
# or
make benchmarkOutputs land in reports/ (benchmark_report.md, validation_report.md,
results.md, results.tex) and reports/figures/. A Docker image is provided
for a pinned environment:
docker build -t legalmcqa . && docker run --rm legalmcqaUnsupervised reference baselines on all 20 questions (95% Wilson CI). These are produced by the toolkit and are exactly reproducible. There are no fabricated or third-party numbers here.
| Baseline | Accuracy | 95% CI (Wilson) | Macro-F1 |
|---|---|---|---|
| tfidf_cooccurrence | 0.450 | [0.258, 0.658] | 0.278 |
| word_overlap | 0.450 | [0.258, 0.658] | 0.270 |
| first_option | 0.350 | [0.181, 0.567] | 0.130 |
| longest_option | 0.300 | [0.145, 0.519] | 0.277 |
| random (empirical) | 0.247 | ~0.25 (analytic) | — |
The lexical co-occurrence baseline — in the spirit of the original paper — clears chance, but the wide intervals show why larger evaluations and the leaderboard matter. Open baseline slots (Sentence-Transformer, BERT/Legal-BERT, Longformer, retrieval-augmented, LLM) await submissions.
For every question the toolkit can return the selected answer, a confidence, a
full ranking of the alternatives, and the dataset's rationale. See section 5
of the quickstart notebook. Model outputs are
designed to carry a confidence so low-confidence answers can be surfaced as
uncertainty warnings and scored for calibration.
The dataset's single source of truth is scripts/build_dataset.py. Edit the
metadata there, regenerate, validate, and test:
python scripts/build_dataset.py
python -m legalmcqa.evaluate validate
python -m pytest -qPlease read CONTRIBUTING.md first — no fabricated legal
content, cite verifiable sources, and flag contestable items with
needs_review. Use the issue and PR templates under .github/.
If you use this dataset or toolkit, please cite the paper:
@inproceedings{GilFT19,
title = {Multiple Choice Question Answering in the Legal Domain Using Reinforced Co-occurrence},
author = {Jorge Martinez-Gil and Bernhard Freudenthaler and A Min Tjoa},
booktitle = {Database and Expert Systems Applications - 30th International Conference, {DEXA} 2019, Linz, Austria, August 26-29, 2019, Proceedings, Part {I}},
year = {2019},
publisher = {Springer},
pages = {138--148},
series = {Lecture Notes in Computer Science},
volume = {11706},
doi = {10.1007/978-3-030-27615-7_10},
url = {https://doi.org/10.1007/978-3-030-27615-7_10}
}A machine-readable citation is in CITATION.cff.
Code (the legalmcqa package, scripts, tests): MIT — see LICENSE.
Dataset (dataset.json, data/): CC BY 4.0 — see LICENSE-DATA.
Disclaimer. This dataset is for research and education only and is not legal advice. Legal rules vary by jurisdiction and change over time; verify any legal content against primary sources.
- Exploring the State of the Art in Legal QA Systems — A. Abdallah, B. Piryani, A. Jatowt. Journal of Big Data, 2023 (Springer).
- A Survey on Legal Question–Answering Systems — J. Martinez-Gil. Computer Science Review, 2023 (Elsevier).
- BERT-CNN Based Evidence Retrieval and Aggregation for Chinese Legal Multi-Choice Question Answering — Y. Li, J. Wu, X. Luo. Neural Computing and Applications, 2024 (Springer).
- cLegal-QA: A Chinese Legal Question Answering with Natural Language Generation Methods — Y. Wang, X. Shen, Z. Huang, L. Niu, S. Ou. Complex & Intelligent Systems, 2025 (Springer).
- Language models for information quality: methods and applications — JG Mathew. Doctoral thesis, 2025.
- Natural Language-Based Analysis of SQuAD: An Analytical Approach for BERT — Z.A. Guven, M.O. Unalir. Expert Systems with Applications, 2022 (Elsevier).
- NLP-Based Management of Large Multiple-Choice Test Item Repositories — V. Albano, D. Firmani, L. Laura, J.G. Mathew. Journal of Learning, 2023 (ERIC).
- Paragraph Similarity Matches for Generating Multiple-Choice Test Items — H. Maslak, R. Mitkov. RANLP Student Research Workshop, 2021 (ACL Anthology).
- Managing Large Multiple-Choice Test Item Repositories — V. Albano, D. Firmani, L. Laura. IEEE ICETA, 2022.
- A General Framework for Multiple-Choice Question Answering Based on Mutual Information and Reinforced Co-Occurrence — J. Martinez-Gil, B. Freudenthaler, A.M. Tjoa. TLDKS, 2019 (Springer).