Skip to content

test(rayline): prove multi-router transactional consistency #2756

Description

@chilang

Problem

Rayline ARC now has a bounded retained-encoder replica contract with Redis-backed episode ownership, versioned membership, explicit close fanout, and one status-gated encoder remap. The current live dynamic-membership evidence exercises a single Semantic Router process. Before claiming router-tier high availability, we need to prove that the same episode remains transactional when successive turns are handled by different Semantic Router replicas.

This work is intentionally deferred while the current program focuses on single-instance end-to-end serving performance.

Proposed scope

  • Run at least two Semantic Router processes against the same Redis episode store and encoder-membership source.
  • Alternate or round-robin successive turns from one episode across router processes.
  • Exercise membership register, drain, removal, router restart, and exact final-turn close while requests are active.
  • Reuse the existing Rayline ARC artifact and retained-encoder request contract rather than adding another routing or state service.
  • Keep telemetry aggregate-only.

Acceptance criteria

  • Exactly one fenced commit occurs for each successful logical turn.
  • Concurrent preparation for the same episode cannot produce two committed transitions.
  • Encoder ownership and visited-owner state remain sticky and reconstructible across router processes.
  • Membership revisions are adopted monotonically; a stale router cannot overwrite or commit against invalid ownership.
  • A draining encoder receives no new episode affinity while existing owners remain valid until the documented removal boundary.
  • Final-turn close reaches every visited owner idempotently and resident sessions converge to zero.
  • Restarting one router during an active episode does not corrupt committed state or leak a lease.
  • Tests cover Redis interruption and privacy scans without logging prompts, raw episode IDs, credentials, or replica endpoints.

Out of scope

  • Kubernetes fleet provisioning or autoscaling.
  • Shared encoder KV storage.
  • The current single-instance OpenRouter end-to-end performance benchmark.
  • General multi-region consensus or an unbounded retry policy.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status
    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions