Catch AI transcripts that failed silently — the ones that read fluently and are wrong.
When speech-to-text mangles specialist audio, it doesn't crash or garble. It substitutes: the hard technical words get replaced by ordinary words that sound the same, and the result reads like a normal sentence. This tool checks a transcript for the vocabulary its topic says should be there, and raises the alarm when it isn't.
Automatic captions on specialist content don't crash. On a psychopharmacology lecture they cheerfully produce:
"Today we cover close a pin. The feared risks are a granular site process and my old card artist…"
for audio that said clozapine, agranulocytosis, and myocarditis. The output is grammatical, confident, and useless — and because it's fluent, nothing downstream flags it. I hit this pulling lecture transcripts for study notes: the caption engine had quietly replaced most of the clinical vocabulary with phonetic soundalikes, and every summary built on top of it inherited the garbage.
Specialist content predicts its own vocabulary. A transcript of a clozapine lecture that contains almost no clozapine-lecture words is broken, no matter how clean it reads:
$ python -m captioncanary transcript.txt terms.txt
FAILED: only 0% of expected terms present; phonetic substitutions detected:
{'clozapine': 'close a pin', 'seizure threshold': 'see sure threshold'}
— transcript is likely fluent nonsense for this topic
Two checks:
- Vocabulary coverage — fraction of expected domain terms present. Under 50% → suspicious; under 20% → failed. Conservative on purpose.
- Phonetic-substitution detection — the signature of silent caption failure is that a term's letter material survives, split across adjacent common words. Positional string comparison misses this ("closeapin" vs "clozapine" differs at 6 of 9 positions), so the matcher uses sequence similarity over squashed word runs and recovers exactly the garbled span.
Comparison mode scores two transcripts of the same audio (platform captions vs a local Whisper run) and reports whether the coverage gap is big enough to mean one of them silently failed:
$ python -m captioncanary autocaptions.txt terms.txt --compare whisper.txt
This is a general pattern for AI-output QC: fluency is not evidence of correctness, and the absence of expected domain signal is a measurable red flag. The same trick — score generated output against the vocabulary its context predicts — applies to summaries, extractions, and translations.
Companion essay: When Not to Use a Model — this tool is the third of three cases on when a deterministic check beats a better model.
pip install pytest && python -m pytest tests/ -v
(for when above too many word)
ROBOT LISTEN. ROBOT WRITE DOWN WHAT HEAR. WORDS COME OUT SMOOTH.
SMOOTH NOT MEAN RIGHT — ROBOT SWAP HARD WORD FOR EASY WORD THAT SOUND SAME. SNEAKY.
THIS TOOL = CANARY. CANARY KNOW WHAT WORD BELONG.
RIGHT WORD GONE, FAKE WORD SNEAK IN → CANARY SING. CANARY QUIET → PROBABLY FINE.
EYEBALL TIRED. EYEBALL TRUST SMOOTH.
CANARY NOT TRUST SMOOTH — CANARY LISTEN FOR WORD THAT SHOULD BE THERE. NOT THERE = DANGER.
🪨 Caveman voice borrowed, with thanks, from caveman by Julius Brussee — a Claude Code skill that makes agents talk like this to cut ~75% of output tokens. Credit to him for the style; go star it.
No dependencies beyond the standard library. Test fixtures are synthetic reconstructions of the failure mode. MIT license.