What would you like to be added?
An optional L2 semantic tier for the semantic-cache polarity guard, layered
on top of the L1 lexical guard shipped for #2691. When enabled, a candidate
cache hit that clears the bi-encoder similarity_threshold and passes the L1
lexical guard is verified once with the router's in-repo NLI model
(candle.ClassifyNLI, tasksource/ModernBERT-base-nli, added in #1865): if the
incoming query and the matched entry query are classified contradiction
above a configurable threshold, the hit is rejected.
Proposed config (opt-in, default keeps L1-only behaviour):
semantic_cache:
polarity_guard:
mode: lexical | nli | lexical+nli # default: lexical (L1 only)
nli:
model_id: models/ModernBERT-base-nli
contradiction_threshold: 0.5
Design constraints:
- L1 stays the unconditional, zero-dependency floor. L2 never replaces it;
it only adds coverage when the model is configured.
- NLI runs at most once per lookup, on the single best candidate right
before serving — never per-entry inside the scan loop (that would multiply the
model cost by the cache size).
- Opt-in, not mandatory. Enabling
mode: nli|lexical+nli binds the NLI
model; leaving the default lexical requires no model and does not regress
existing cache users. Model absence with an NLI mode selected should fail at
config-load (validate early), not at runtime.
- Reuse the existing
SetGroundingBackends-style function injection so pkg/cache
does not import pkg/classification (avoids an import cycle).
Why is this needed?
The L1 lexical guard (#2691) is a cue/antonym-table guard: it deterministically
catches single-token negation and known antonym flips, but by design misses
cue-less semantic negation, ordering-only polarity changes, and non-English
queries. The router already ships an NLI model that closes exactly this gap.
Measured on tasksource/ModernBERT-base-nli via the production candle path,
on the same polarity pairs from #2691:
| pair (question form) |
contradiction |
| enable / disable dark mode |
0.983 |
| "reset my password" + inserted not |
0.967 |
| open / close the file |
0.993 |
| start / stop the server |
0.997 |
| add / remove a user |
0.987 |
| genuine paraphrase (reset ⇄ reset) |
0.001 |
| synonym (delete ⇄ remove) |
0.001 |
Discrimination margin is ~0.96 vs ≤0.002 — a fixed 0.5 threshold separates
cleanly with no tuning. It also correctly treats delete/remove as a synonym
(entailment), a case the L1 antonym table does not contain. Latency is
~70 ms/call single-thread CPU (p99 ~97 ms), one-time init ~330 ms; materially
lower on GPU. Since a cache hit avoids a full multi-second LLM generation, a
single ~70 ms verification before serving is net-positive.
Additional context
What would you like to be added?
An optional L2 semantic tier for the semantic-cache polarity guard, layered
on top of the L1 lexical guard shipped for #2691. When enabled, a candidate
cache hit that clears the bi-encoder
similarity_thresholdand passes the L1lexical guard is verified once with the router's in-repo NLI model
(
candle.ClassifyNLI,tasksource/ModernBERT-base-nli, added in #1865): if theincoming query and the matched entry query are classified contradiction
above a configurable threshold, the hit is rejected.
Proposed config (opt-in, default keeps L1-only behaviour):
Design constraints:
it only adds coverage when the model is configured.
before serving — never per-entry inside the scan loop (that would multiply the
model cost by the cache size).
mode: nli|lexical+nlibinds the NLImodel; leaving the default
lexicalrequires no model and does not regressexisting cache users. Model absence with an NLI mode selected should fail at
config-load (validate early), not at runtime.
SetGroundingBackends-style function injection sopkg/cachedoes not import
pkg/classification(avoids an import cycle).Why is this needed?
The L1 lexical guard (#2691) is a cue/antonym-table guard: it deterministically
catches single-token negation and known antonym flips, but by design misses
cue-less semantic negation, ordering-only polarity changes, and non-English
queries. The router already ships an NLI model that closes exactly this gap.
Measured on
tasksource/ModernBERT-base-nlivia the production candle path,on the same polarity pairs from #2691:
Discrimination margin is ~0.96 vs ≤0.002 — a fixed
0.5threshold separatescleanly with no tuning. It also correctly treats
delete/removeas a synonym(entailment), a case the L1 antonym table does not contain. Latency is
~70 ms/call single-thread CPU (p99 ~97 ms), one-time init ~330 ms; materially
lower on GPU. Since a cache hit avoids a full multi-second LLM generation, a
single ~70 ms verification before serving is net-positive.
Additional context
it does not re-open the bug.
feature: Add model-managed cross-encoder reranking for two-stage retrieval/routing #2247 / feature: optional two-stage cross-encoder rerank as a routing signal #2251 / feat: add rerank endpoint with bi-encoder and cross-encoder backends #2236, which remains deferred; this reuses the lighter,
already-present in-repo NLI head.
context on the same cache surface).