Skip to content

[Router] Optional NLI polarity tier (L2) for in-memory semantic cache negation guard #2751

Description

@WUKUNTAI-0211

What would you like to be added?

An optional L2 semantic tier for the semantic-cache polarity guard, layered
on top of the L1 lexical guard shipped for #2691. When enabled, a candidate
cache hit that clears the bi-encoder similarity_threshold and passes the L1
lexical guard is verified once with the router's in-repo NLI model
(candle.ClassifyNLI, tasksource/ModernBERT-base-nli, added in #1865): if the
incoming query and the matched entry query are classified contradiction
above a configurable threshold, the hit is rejected.

Proposed config (opt-in, default keeps L1-only behaviour):

semantic_cache:
  polarity_guard:
    mode: lexical | nli | lexical+nli   # default: lexical (L1 only)
    nli:
      model_id: models/ModernBERT-base-nli
      contradiction_threshold: 0.5

Design constraints:

  • L1 stays the unconditional, zero-dependency floor. L2 never replaces it;
    it only adds coverage when the model is configured.
  • NLI runs at most once per lookup, on the single best candidate right
    before serving — never per-entry inside the scan loop (that would multiply the
    model cost by the cache size).
  • Opt-in, not mandatory. Enabling mode: nli|lexical+nli binds the NLI
    model; leaving the default lexical requires no model and does not regress
    existing cache users. Model absence with an NLI mode selected should fail at
    config-load (validate early), not at runtime.
  • Reuse the existing SetGroundingBackends-style function injection so pkg/cache
    does not import pkg/classification (avoids an import cycle).

Why is this needed?

The L1 lexical guard (#2691) is a cue/antonym-table guard: it deterministically
catches single-token negation and known antonym flips, but by design misses
cue-less semantic negation, ordering-only polarity changes, and non-English
queries. The router already ships an NLI model that closes exactly this gap.

Measured on tasksource/ModernBERT-base-nli via the production candle path,
on the same polarity pairs from #2691:

pair (question form) contradiction
enable / disable dark mode 0.983
"reset my password" + inserted not 0.967
open / close the file 0.993
start / stop the server 0.997
add / remove a user 0.987
genuine paraphrase (reset ⇄ reset) 0.001
synonym (delete ⇄ remove) 0.001

Discrimination margin is ~0.96 vs ≤0.002 — a fixed 0.5 threshold separates
cleanly with no tuning. It also correctly treats delete/remove as a synonym
(entailment), a case the L1 antonym table does not contain. Latency is
~70 ms/call single-thread CPU (p99 ~97 ms), one-time init ~330 ms; materially
lower on GPU. Since a cache hit avoids a full multi-second LLM generation, a
single ~70 ms verification before serving is net-positive.

Additional context

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

Status
Backlog

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions