[BUG]Concurrent streaming requests wrongly queued as mid-turn → 202 breaks standard SDK clients

[wzy-del]

Description

Mid-turn steering wrongly queues concurrent independent streams. When two streaming /v1/messages requests share the same model + system prompt and arrive concurrently (no x-headroom-session-id header), the proxy misclassifies the second as a "mid-turn message", returns 202 {"event":"headroom_queued"}, and never forwards it upstream. A standard Anthropic SDK client that made a streaming call receives a non-SSE 202 → empty event stream → AssertionError, and fails after retries.

To Reproduce

1. Run the proxy in cache mode: headroom proxy --port 8799 --mode cache
2. Point a stock Anthropic SDK at it and fire two concurrent messages.stream() calls with identical model + system and no x-headroom-session-id header (see Code Sample).
3. The second (in-flight) request returns 202; the SDK's get_final_message() raises AssertionError.

Expected Behavior

Both concurrent streams are forwarded upstream and return normal SSE responses. Mid-turn steering should only apply to clients that explicitly opt in via x-headroom-session-id, since queued messages are only drained back through the custom headroom_pending_messages SSE event that a standard SDK doesn't understand.

Actual Behavior

The second request is queued and answered with a non-SSE 202 {"event":"headroom_queued"} and is never sent upstream. The streaming SDK accumulates zero events, so self.__final_message_snapshot stays None and the final-message assertion fails. Proxy logs show status=202 with upstream_connect: null.

Root cause: without an x-headroom-session-id header, _get_session_key() falls back to md5(model + system[:500]) (mirrors prefix_tracker.compute_session_id). That key is too coarse to distinguish concurrent independent streams sharing a model + system prompt, so the second stream hits session_key in self._active_streams and gets queued. Note: sending a unique header per request is not a workaround — the same header also drives prefix_tracker.compute_session_id(), so unique-per-stream ids break prompt caching while a shared id keeps colliding.

Code Sample

import anthropic, threading

client = anthropic.Anthropic(base_url="http://127.0.0.1:8799", api_key="…")

def call():
    with client.messages.stream(
        model="claude-sonnet-4-20250514",
        system="same system prompt for both",
        max_tokens=256,
        messages=[{"role": "user", "content": "hello"}],
    ) as s:
        s.get_final_message()

# Fire both while the first is still streaming
t1 = threading.Thread(target=call); t2 = threading.Thread(target=call)
t1.start(); t2.start(); t1.join(); t2.join()

Error Output

AssertionError
  File ".../anthropic/lib/streaming/_messages.py", line 94, in get_final_message
    assert self.__final_message_snapshot is not None

Proxy log for the failing request:

event=proxy_inbound_response method=POST path=/v1/messages status=202  (upstream_connect: null — never forwarded)

Environment

• Headroom version: 0.30.0 (also present on main)
• Python version: 3.11
• OS: macOS
• LLM Provider: Anthropic (streaming, messages.stream); proxy mode cache

Additional Context

Suggested fix — only engage mid-turn steering when the client explicitly opts in via the header (never from the coarse md5 fallback):

explicit = request.headers.get("x-headroom-session-id")
session_key = self._get_session_key(body, session_header=explicit)
if explicit and session_key in self._active_streams:
    return JSONResponse(content=self._queue_mid_turn_message(session_key, body), status_code=202)

All existing test_mid_turn_steering.py cases pass x-headroom-session-id explicitly, so they're unaffected; prefix-tracker / cache behavior is untouched. A larger alternative is to track _active_streams by a per-stream instance token rather than by session id, so mid-turn steering can coexist with concurrent independent streams even for header-bearing clients.