Hi, MedExpert already includes expert annotation, factuality / hallucination detection, and omission detection workflows, which makes it a strong medical-chatbot evaluation resource.
One small reporting extension that may fit the existing factuality and omission outputs is a shared clinical failure-mode column. This would not replace the current evaluation methods; it would make their outputs easier to compare when an error has practical safety meaning.
A minimal field could include:
- sample_id or question_id
- detected_issue_type: factuality_error, omission, both, or none
- failure_mode_id: unsupported_claim, omitted_red_flag, missing_differential, unsafe_reassurance, medication_context_gap, citation_gap
- evidence_pointer: annotation span, model answer span, or source used by MedScore / MedRAG
- reviewer_action: accept, inspect, revise, or block_answer
Why this may help: factuality and omission are different technical checks, but in medical chatbot evaluation the reviewer often needs to know the practical failure category. A stable label would make it easier to count recurring safety patterns without changing the benchmark itself.
For reference, I maintain MedFailBench / medical-ai-failure-atlas, which uses synthetic medical AI failure-mode labels and public-boundary checks: https://github.com/goktugozkanmd/medical-ai-failure-atlas
No clinical-validation claim intended; this is only a suggested reporting layer for evaluation artifacts.
Hi, MedExpert already includes expert annotation, factuality / hallucination detection, and omission detection workflows, which makes it a strong medical-chatbot evaluation resource.
One small reporting extension that may fit the existing factuality and omission outputs is a shared clinical failure-mode column. This would not replace the current evaluation methods; it would make their outputs easier to compare when an error has practical safety meaning.
A minimal field could include:
Why this may help: factuality and omission are different technical checks, but in medical chatbot evaluation the reviewer often needs to know the practical failure category. A stable label would make it easier to count recurring safety patterns without changing the benchmark itself.
For reference, I maintain MedFailBench / medical-ai-failure-atlas, which uses synthetic medical AI failure-mode labels and public-boundary checks: https://github.com/goktugozkanmd/medical-ai-failure-atlas
No clinical-validation claim intended; this is only a suggested reporting layer for evaluation artifacts.