fix(router): invert IP vector distance before comparing against a similarity threshold - #2716
Merged
Merged
Conversation
✅ Deploy Preview for vllm-semantic-router ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
github-actions
Bot
assigned
AayushSaini101,
drivebyer,
FAUST-BENCHOU,
ramkrishs,
shraderdm and
WUKUNTAI-0211
Contributor
👥 vLLM Semantic Team NotificationThe following members have been identified for the changed files in this PR and have been automatically assigned when their GitHub accounts are assignable in this repository: 📁
|
Contributor
✅ Supply Chain Security Report — All Clear
Scanned at |
Contributor
Performance Benchmark ResultsComponent benchmarks completed with no regressions beyond thresholds. Summary
DetailsSee attached benchmark artifacts for detailed results and profiles. Performance testing powered by vLLM Semantic Router |
…ity threshold
RediSearch and valkey-search report vector_distance as a distance for all
three metrics: IP is defined as d = 1 - u*v, so smaller means closer. Four
copies of the conversion returned that value unchanged for IP while their
callers kept the result on `similarity >= threshold`, which runs the
comparison backwards. An identical query yields d = 0 and scored 0.0, below
every threshold, while an unrelated one yields d ~ 1 and scored ~1.0. The
semantic cache never reused a cached response, agentic memory never recalled
a stored memory, and the vector store dropped exact-match chunks.
Collapse the four copies into pkg/utils/valkey, whose package doc already
covers the vector store, semantic cache and agentic memory, and fix IP there
once. Normalize the Redis metric type at construction as the other three
backends already do, so the index that gets built and the formula used to
read its distances back cannot disagree.
COSINE and L2 are unaffected: their branches already inverted the ordering.
Three unit tests asserted the passthrough and are replaced by a shared table
plus a monotonicity guard that fails for any branch where a larger distance
scores as more similar. The Valkey integration tests for the IP and L2
metrics were written as `if hit { assert.NotNil(...) }`, passing whether or
not the cache hit; they now require the hit. Tests that assert which entry
came back get their own index, because the shared "doc:" keyspace is never
cleaned up and a TopK=1 search ranks against every document earlier tests
left behind.
Closes #2715
Signed-off-by: drivebyer <wuyangmuc@gmail.com>
drivebyer
force-pushed
the
fix/semantic-cache-inverted-hit-criterion
branch
from
July 28, 2026 14:19
40c3276 to
88eae76
Compare
drivebyer
changed the title
…corder Deduplicate the request/response body capture blocks in AddRecord into a single helper, keeping the semantics introduced in #2749 unchanged: capture off drops the body, oversized bodies are truncated, and the caller's truncated flag is preserved within the limit. Follow-up to a post-merge review suggestion on #2749. Signed-off-by: drivebyer <wuyangmuc@gmail.com>
drivebyer
force-pushed
the
fix/semantic-cache-inverted-hit-criterion
branch
from
August 4, 2026 14:46
d92143a to
17c7fd8
Compare
Xunzhuo
approved these changes
Aug 5, 2026
Contributor
Merge Queue Status
This pull request spent 3 hours 27 minutes 57 seconds in the queue, with no time running CI. Waiting for
All conditions
ReasonThe merge conditions cannot be satisfied due to failing checks Failing checks: HintYou may have to fix your CI before adding the pull request to the queue again. Requeued — the merge queue status continues in this comment ↓. |
Contributor
Merge Queue Status
This pull request spent 3 minutes 32 seconds in the queue, including 2 seconds running CI. Required conditions to merge
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2715
Purpose
RediSearch and valkey-search return
vector_distanceas a distance for all three metrics. ForIPit is defined asd = 1 - u·v, so smaller means closer (Redis docs, valkey FT.CREATE). Four copies of the distance-to-similarity conversion returned that value unchanged forIP, while their callers keep results onsimilarity >= threshold. The comparison ran backwards: an identical query givesd = 0and scored0.0, an unrelated one givesd ≈ 1and scored≈ 1.0.Three paths were affected: the semantic cache (Redis and Valkey), agentic memory retrieval, and vector-store/RAG retrieval.
The conversion now lives in
pkg/utils/valkey, whose package doc already covers all three components, and the four call sites use it. The Redis backend also gains the uppercase metric-type normalization the other three already had, so the index that gets built and the formula that reads its distances back cannot disagree.COSINEandL2already ordered correctly and are unchanged.Module:
RouterTest Plan
make agent-lint CHANGED_FILES="<the changed files>" SKIP_VALKEY_TESTS=false VALKEY_HOST=localhost VALKEY_PORT=6380 make test-semantic-routerThree unit tests asserted the passthrough and are replaced by one shared table plus
TestDistanceToSimilarityIsDecreasing, which fails for any branch where a larger distance scores as more similar. The Valkey integration tests forIPandL2were written asif hit { assert.NotNil(...) }and passed either way. They now require the hit and assert the returned body. CI runs them withSKIP_VALKEY_TESTS: false, which is the regression gate for this change. No E2E case is added: everye2e/config/*.yamlusesbackend_type: memory, and the integration tests cover this instead.Test Result
make agent-lint: exit 0main, which brought inpartitionedKNNQueryfrom [Router][Dashboard] Add first-class Mixture-of-Models recipes #2741IPbranch toreturn distancefailsTestDistanceToSimilarityIsDecreasing/IPandTestValkeyCacheIntegration_IPMetricTypewithsimilarity=0.0000, threshold=0.5Known gaps:
TestValkeyCacheIntegration_UpdateWithResponsestill shares thedoc:keyspace and can miss on a dirty Valkey (newIsolatedValkeyCacheadded here is available if someone converts it later). An unsetmetric_typestill builds a COSINE index read back as1-d; unchanged here and deferred to the COSINE-scale discussion.