[Router][CLI][Dashboard] Harden recursive serving and recipe runtime - #2770
Open
Xunzhuo wants to merge 9 commits into
Open
[Router][CLI][Dashboard] Harden recursive serving and recipe runtime#2770Xunzhuo wants to merge 9 commits into
Xunzhuo wants to merge 9 commits into
Conversation
Allow complete local configurations to bootstrap their first administrator safely and resolve recursive routing through the effective split-runtime listener. Signed-off-by: Xunzhuo Liu <xunzhuo.liu@amd.com>
Authenticate internal routing context, normalize bounded ReMoM responses, and preserve recipe identity through durable replay storage. Signed-off-by: Xunzhuo Liu <xunzhuo.liu@amd.com>
Discover transitive model consumers across reachable recipes while separating default API readiness from aggregate runtime inventory. Signed-off-by: Xunzhuo Liu <xunzhuo.liu@amd.com>
Keep explicit reasoning evidence above terse preferences and align the maintained recipe with executable backend capabilities. Signed-off-by: Xunzhuo Liu <xunzhuo.liu@amd.com>
Preserve new policy-signal behavior while retaining default API and aggregate recipe readiness semantics. Signed-off-by: Xunzhuo Liu <xunzhuo.liu@amd.com>
Match the calibrated probe manifest to the plural preference signal family emitted by eval responses. Signed-off-by: Xunzhuo Liu <xunzhuo.liu@amd.com>
✅ Deploy Preview for vllm-semantic-router ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Contributor
There was a problem hiding this comment.
Pull request overview
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
This PR tightens split-runtime/router internal re-entry, improves ReMoM runtime behavior (non-streaming subrequests, completion limits, and empty-response handling), and makes recipe scoping durable across replay storage and model/classifier readiness reporting.
Changes:
- Add recipe-scoped internal-request authentication + recipe propagation for looper re-entry and replay isolation.
- Extend ReMoM with
max_completion_tokens, enforce non-streaming internal calls, and prevent “successful” empty final responses. - Rework model/classifier “needs” + readiness to account for request-reachable named recipes and truthful lifecycle metadata (local vs endpoint).
Reviewed changes
Copilot reviewed 88 out of 88 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| website/docs/tutorials/algorithm/looper/remom.md | Document non-streaming ReMoM subrequests and max_completion_tokens. |
| website/docs/proposals/unified-config-contract-v0-3.md | Add unified contract note for remom.max_completion_tokens. |
| website/docs/installation/configuration.md | Document split-runtime looper endpoint rewrite and ReMoM completion budget. |
| src/vllm-sr/tests/test_runtime_support.py | Add tests for dashboard bootstrap policy defaults and looper endpoint rewrite. |
| src/vllm-sr/tests/test_algorithm_config.py | Add ReMoM max_completion_tokens validation coverage (Pydantic). |
| src/vllm-sr/cli/container_start.py | Default dashboard open-bootstrap policy when admin creds aren’t fully present. |
| src/vllm-sr/cli/commands/runtime_support.py | Apply local looper endpoint rewrite in effective-config generation. |
| src/vllm-sr/cli/commands/runtime_looper.py | New helper to rewrite loopback/missing looper endpoint to stack-local Envoy. |
| src/vllm-sr/cli/algorithms.py | Add max_completion_tokens to CLI ReMoM config model. |
| src/semantic-router/pkg/services/classification_readiness.go | Add aggregate “Any*” readiness methods (reachable-recipe inventory). |
| src/semantic-router/pkg/routerreplay/store/postgres_record_row.go | Add replay recipe column wiring into Record decode/insert/select. |
| src/semantic-router/pkg/routerreplay/store/postgres_recipe_test.go | New tests to guard recipe column alignment & round-trip behavior. |
| src/semantic-router/pkg/routerreplay/store/postgres.go | Add recipe TEXT column + migration and include in INSERT placeholders. |
| src/semantic-router/pkg/modeldownload/named_recipe_test.go | New tests for named-recipe model dependency discovery and reachability. |
| src/semantic-router/pkg/modeldownload/feature_gates.go | Update optional model gates to reflect API vs routing vs backend needs. |
| src/semantic-router/pkg/modeldownload/config_parser_test.go | Expand test fixture to ensure fact-check model is included when used. |
| src/semantic-router/pkg/modeldownload/config_parser.go | Provision only shared/default + request-reachable recipe model paths. |
| src/semantic-router/pkg/looper/routing_recipe_context.go | New context key for routing recipe propagation through looper calls. |
| src/semantic-router/pkg/looper/remom_test.go | Add ReMoM tests for usability counting, token limits, empty outputs, reasoning-only behavior. |
| src/semantic-router/pkg/looper/remom_response.go | New helpers to select final response and format JSON/SSE with stricter “non-empty” guarantees. |
| src/semantic-router/pkg/looper/remom.go | Enforce non-streaming internal calls, apply completion-token override, count only usable responses, fallback behavior for empty final. |
| src/semantic-router/pkg/looper/looper.go | Add RecipeName to looper Request for propagation across internal calls. |
| src/semantic-router/pkg/looper/execute_latency.go | Attach recipe to context before Execute so client headers can include it. |
| src/semantic-router/pkg/looper/client_response_normalization_test.go | New tests for reasoning/content separation in streaming and non-streaming parsing. |
| src/semantic-router/pkg/looper/client_recipe_header_test.go | New test validating recipe header propagation + internal auth on internal calls. |
| src/semantic-router/pkg/looper/client_internal_context.go | Add authenticated internal headers (token, recipe, decision, iteration). |
| src/semantic-router/pkg/looper/client.go | Centralize internal header setting; parse SSE reasoning fields; normalize reasoning extraction helpers. |
| src/semantic-router/pkg/internalauth/token_test.go | New tests for token generation/authentication behavior. |
| src/semantic-router/pkg/internalauth/token.go | New process-local internal auth token implementation. |
| src/semantic-router/pkg/headers/headers_test.go | Add header-constant coverage for x-vsr-internal-auth. |
| src/semantic-router/pkg/headers/headers.go | Define VSRInternalAuth header constant. |
| src/semantic-router/pkg/extproc/request_context.go | Clarify LooperRequest means token-authenticated internal requests. |
| src/semantic-router/pkg/extproc/req_filter_looper_test.go | Ensure selected-recipe header is stripped before backend dispatch. |
| src/semantic-router/pkg/extproc/req_filter_looper_routing_context.go | Restore recipe boundary for authenticated looper re-entry and decision lookup. |
| src/semantic-router/pkg/extproc/req_filter_looper_replay_test.go | New tests for recipe-scoped replay recorder selection and spoofing defense. |
| src/semantic-router/pkg/extproc/req_filter_looper_reasoning_alias_test.go | Ensure reasoning capability checks use physical model metadata, not request alias. |
| src/semantic-router/pkg/extproc/req_filter_looper_internal.go | Remove internal headers before backend; hydrate looper recipe context before decision resolution. |
| src/semantic-router/pkg/extproc/req_filter_looper.go | Propagate selected recipe into looper request object. |
| src/semantic-router/pkg/extproc/recorder_recipe_lifecycle_test.go | Ensure replay records persist resolved recipe scope. |
| src/semantic-router/pkg/extproc/processor_req_header_test.go | Require internal token for looper detection; ensure internal headers are removed/mutated. |
| src/semantic-router/pkg/extproc/processor_req_header_internal.go | New shared internal-header list, token authentication, and removal mutation helper. |
| src/semantic-router/pkg/extproc/processor_req_header.go | Always remove looper internal headers; authenticate looper context after capture. |
| src/semantic-router/pkg/dsl/dsl_test.go | Add DSL compile/decompile + validation tests for max_completion_tokens. |
| src/semantic-router/pkg/dsl/decompiler_algorithms.go | Include max_completion_tokens in ReMoM decompile output. |
| src/semantic-router/pkg/dsl/compiler_algorithms.go | Validate ReMoM config after compile; parse max_completion_tokens. |
| src/semantic-router/pkg/config/testdata/generic-multi-recipe-model-needs.yaml | Add named-recipe fixture for model needs across recipes, projections, plugins. |
| src/semantic-router/pkg/config/routing_signal_usage_test.go | Add tests for projection dependency traversal, cycle handling, reachability, and model needs. |
| src/semantic-router/pkg/config/routing_signal_usage.go | Add reachable-routing signal usage; add new model-need helpers; traverse projection deps with cycle protection. |
| src/semantic-router/pkg/config/remom_config_test.go | Add validation tests for ReMoM max_completion_tokens. |
| src/semantic-router/pkg/config/remom_config.go | Validate max_completion_tokens is positive when set. |
| src/semantic-router/pkg/config/recipes_registry_test.go | Add tests for reachable recipe inventory and auto-alias disable behavior. |
| src/semantic-router/pkg/config/recipes.go | Add reachability inventory (ReachableRoutingRecipes, IsRecipeReachableForRouting). |
| src/semantic-router/pkg/config/multi_objective_asset_test.go | Update multi-objective tests for reasoning-family contract & plugin availability. |
| src/semantic-router/pkg/config/module_runtime_helpers.go | Redefine “enabled” helpers to reflect API vs routing consumers and reachability. |
| src/semantic-router/pkg/config/decision_config.go | Add max_completion_tokens to ReMoM YAML config struct. |
| src/semantic-router/pkg/config/config_test.go | Update tests for new “enabled” semantics requiring actual consumers. |
| src/semantic-router/pkg/classification/unified_classifier_stub.go | Extend native backend capabilities shape for hallucination support flags. |
| src/semantic-router/pkg/classification/unified_classifier_cgo_onnx.go | Extend native backend capabilities shape for hallucination support flags. |
| src/semantic-router/pkg/classification/unified_classifier_cgo_candle.go | Advertise candle local hallucination + NLI capabilities. |
| src/semantic-router/pkg/classification/recipe_classifiers_test.go | Add tests for default API readiness vs inventory readiness, and lifecycle ordering. |
| src/semantic-router/pkg/classification/recipe_classifiers.go | Track runtime vs routing lifecycle orders; add default vs aggregate readiness helpers. |
| src/semantic-router/pkg/classification/native_capabilities_test.go | Update tests for new capabilities fields. |
| src/semantic-router/pkg/classification/native_capabilities.go | Add capabilities fields for local hallucination + NLI support. |
| src/semantic-router/pkg/classification/hallucination_test.go | Update classifier tests for new consumer-based enablement logic. |
| src/semantic-router/pkg/classification/classifier_signal_context.go | Require initialized models for fact-check/feedback signal readiness. |
| src/semantic-router/pkg/classification/classifier_lifecycle_test.go | Add lifecycle tests for capability degradation, endpoint backend behavior, and consumer ownership. |
| src/semantic-router/pkg/classification/classifier_lifecycle.go | Add default-API-only runtime initialization path and use reachable-routing checks. |
| src/semantic-router/pkg/classification/classifier_feedback_lifecycle.go | Separate API enablement from routing/runtime needs for feedback. |
| src/semantic-router/pkg/classification/classifier_fact_hallucination_lifecycle.go | Separate API enablement from routing/runtime needs; gate local hallucination by native capabilities. |
| src/semantic-router/pkg/apiserver/runtime_service.go | Add live service wrappers for aggregate readiness methods. |
| src/semantic-router/pkg/apiserver/route_model_info_test.go | Add test for endpoint hallucination backend lifecycle truthfulness. |
| src/semantic-router/pkg/apiserver/route_model_info_runtime.go | Avoid resolving local registry/path metadata for “external lifecycle” models. |
| src/semantic-router/pkg/apiserver/route_model_info_classifiers.go | Use aggregate readiness when available; annotate hallucination models with truthful lifecycle/backend metadata. |
| e2e/testcases/router_replay_recipe_filter.go | Add E2E test validating replay list filtering by recipe. |
| e2e/profiles/router-replay/values.yaml | Add entrypoints + recipes to support recipe-scoped replay E2E. |
| e2e/profiles/router-replay/profile.go | Register new replay recipe filter E2E case. |
| dashboard/frontend/src/pages/builderPageStructuredFieldControls.test.tsx | Add UI test for enforcing ReMoM min completion tokens. |
| dashboard/frontend/src/pages/builderPageFieldControls.tsx | Apply min constraint to numeric inputs. |
| dashboard/frontend/src/lib/dslSchemas.ts | Extend schema type with optional min. |
| dashboard/frontend/src/lib/dslAlgorithmSchemas.ts | Add ReMoM max_completion_tokens field schema with min constraint. |
| dashboard/frontend/e2e/auth-flow.spec.ts | Add E2E test for bootstrap registration flow in complete configuration. |
| config/recipes/multi-objective/recipe.dsl | Adjust projection weights; remove hallucination plugin usage for unsupported runtime. |
| config/recipes/multi-objective/probes.yaml | Add probes and update expectations for new weight calibration and plugin availability. |
| config/recipes/multi-objective/config.yaml | Align reasoning-family metadata to shared backend contract; remove hallucination plugins; adjust weights. |
| config/recipes/multi-objective/README.md | Document maintained capability assumptions, probe count update, and plugin exclusion rationale. |
| config/config.yaml | Update canonical config with ReMoM max_completion_tokens. |
| config/algorithm/looper/remom.yaml | Update ReMoM fragment with max_completion_tokens. |
| config/README.md | Document ReMoM fragment’s max_completion_tokens. |
Suppressed comments (2)
src/vllm-sr/cli/container_start.py:1
DASHBOARD_ALLOW_OPEN_BOOTSTRAPis copied based on presence inos.environ, even if the value is empty/whitespace. That can result in an empty string being emitted to the container environment and may bypass the intended defaulting logic. Consider treating empty/whitespace values as “unset” (consistent with the earlierif value:behavior), and optionally normalizing/validating accepted values (e.g., only "true"/"false") before exporting.
src/semantic-router/pkg/config/routing_signal_usage.go:1- When
value_sourceisconfidence,dependency = sourceByOutput[dependency]does not check whether the mapping exists. If the output name is missing fromsourceByOutput,dependencybecomes the empty string and the recursion can mark""as visited/computed, potentially masking later lookups and producing incorrect “does this routing depend on X” results. Add anokcheck (and treat missing mappings as non-matching / terminal) before recursing; also ensure the key-normalization strategy used forsourceByOutputmatches the normalization used at lookup sites.
Comment on lines
+106
to
+113
| func (s *RecipeClassifiers) routingRecipe(recipeName config.RecipeName) bool { | ||
| for _, candidate := range s.routingLifecycleOrder() { | ||
| if candidate == recipeName { | ||
| return true | ||
| } | ||
| } | ||
| return false | ||
| } |
github-actions
Bot
assigned
AayushSaini101,
drivebyer,
FAUST-BENCHOU,
ramkrishs,
shraderdm and
WUKUNTAI-0211
Contributor
👥 vLLM Semantic Team NotificationThe following members have been identified for the changed files in this PR and have been automatically assigned when their GitHub accounts are assignable in this repository: 📁
|
Contributor
Performance Benchmark ResultsComponent benchmarks completed with no regressions beyond thresholds. Summary
DetailsSee attached benchmark artifacts for detailed results and profiles. Performance testing powered by vLLM Semantic Router |
Contributor
✅ Supply Chain Security Report — All Clear
Scanned at |
Keep the refreshed API reference aligned with the repository's executable documentation contract. Signed-off-by: Xunzhuo Liu <xunzhuo.liu@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related #2331
Related #2375
Summary
Test plan
Notes