[Bug]: delegated child background-process completion can resolve a foreign session key after inner executor context loss

[LinLin00000000]

Bug Description

A delegated child can lose the parent session ContextVar when execution crosses an inner ThreadPoolExecutor boundary. The child then resolves HERMES_SESSION_KEY through the process environment, where the value may belong to another concurrently active session. If the child starts a background terminal process, that foreign key is persisted in ProcessSession and in the completion event. The completion can consequently enter the wrong session.

This is a narrow, implementation-level report about the delegated-child producer path. It is not a claim that every wrong-session notification has the same cause, and it does not include private logs or debug-upload data.

Summary

The currently affected runtime checkout has two bare executor submissions in tools/delegate_tool.py: the per-child timeout executor and the multi-child batch executor. The outer async-delegation workers already propagate context, but these inner boundaries do not. A standard-library probe shows that a bare submission reads a foreign process-environment value while contextvars.copy_context().run(...) preserves the parent value.

The upstream main ref checked during this review (91937a6dc3ffbbe2f3be91a500f0ecf962c4cf53) already contains copy_context().run at the corresponding timeout and batch sites. The affected runtime checkout is older/different, so this report should be treated as an affected-checkout/release-regression and regression-coverage follow-up rather than as a claim that current upstream main is still unfixed.

Observed Behavior

In a multi-session run, a background completion from one session was observed entering another session. The source path explains how this can happen:

1. A delegated child crosses an unwrapped executor boundary.
2. The child context has no value for the session-key ContextVar.
3. Session-key resolution falls back to os.environ["HERMES_SESSION_KEY"].
4. The resulting key is stored in ProcessSession.session_key and copied into the completion event.
5. A consumer that maps the key to a live session can deliver the completion or wake-up to the wrong session.

Expected Behavior

A delegated child must retain the true initiating session context across both inner executor boundaries. Any background process it creates should store the true parent session key in ProcessSession and in its completion event. Completion output and any follow-up delivery should be accepted only by that initiating session.

Minimal Reproduction (executor mechanism)

This platform-independent probe uses only the Python standard library and deliberately sets a different process-environment value:

import contextvars
import os
from concurrent.futures import ThreadPoolExecutor

session_key = contextvars.ContextVar("session_key", default=None)
session_key.set("true-parent")
os.environ["HERMES_SESSION_KEY"] = "env-foreign"

def resolve_session_key():
    value = session_key.get()
    return value if value is not None else os.environ["HERMES_SESSION_KEY"]

with ThreadPoolExecutor(max_workers=1) as executor:
    lost = executor.submit(resolve_session_key).result()
    parent_context = contextvars.copy_context()
    propagated = executor.submit(parent_context.run, resolve_session_key).result()

print({
    "parent": session_key.get(),
    "lost_plain_submit": lost,
    "propagated_wrapper": propagated,
    "env": os.environ["HERMES_SESSION_KEY"],
})

Observed output:

{'parent': 'true-parent', 'lost_plain_submit': 'env-foreign',
 'propagated_wrapper': 'true-parent', 'env': 'env-foreign'}

Impact

• A background completion can contaminate an unrelated conversation.
• Completion output can be visible to, or cause a follow-up wake-up in, the wrong live session.
• The core persists the wrong route datum at process creation, so a downstream consumer cannot reliably reconstruct the true parent from session_key alone.
• Concurrent sessions can therefore experience cross-session delivery and potentially act on unrelated process results.

Environment

• Hermes Agent: v0.19.0; affected checkout HEAD=651c0931debf86e75129b4c781f2669e41fdfd42
• Upstream comparison: main=91937a6dc3ffbbe2f3be91a500f0ecf962c4cf53 (contains the two context wrappers; not the affected checkout)
• OS: Linux 6.8.0-136-generic
• Python: 3.11.15
• Surface: multi-session execution with delegated children and background terminal completion notifications
• Affected components: Agent Core / delegation, Tools / terminal and process registry, Gateway session context
• Messaging platform: N/A; the executor mechanism is platform-independent

Debug Report

No debug-upload link is included. This is a source-level isolation defect with a self-contained standard-library reproduction; no private logs, credentials, or session data are required for the mechanism probe.

Source Pointers

On the affected checkout:

• tools/delegate_tool.py:2006-2017: _run_with_thread_capture enters the child conversation, then :2017 calls _timeout_executor.submit(_run_with_thread_capture) without a context wrapper.
• tools/delegate_tool.py:2646-2679: the batch path calls executor.submit(_run_single_child, ...) at :2673-2679 without a context wrapper. A single child still reaches the unwrapped per-child timeout executor.
• tools/approval.py:202-214: get_current_session_key() checks approval context and then calls get_session_env.
• gateway/session_context.py:303-326: a bound context value wins, but an unset value falls back to os.environ for CLI/cron compatibility.
• tools/terminal_tool.py:2448-2454: the terminal background-process path reads the current session key.
• tools/process_registry.py:90-119: ProcessSession stores session_key; :717-724 writes the passed key at spawn; :1142-1174 copies it into a type=completion event.

For comparison, upstream main at 91937a6dc3ffbbe2f3be91a500f0ecf962c4cf53 has:

• tools/delegate_tool.py:2188-2192: contextvars.copy_context() followed by _child_context.run for the timeout worker.
• tools/delegate_tool.py:3005-3013: a per-child contextvars.copy_context() followed by child_context.run for the batch worker.

Relationship to Existing Issues and PRs

• #60426 and #52447 are the broad, user-visible wrong-session background-notification trackers. This report is a narrower producer-side implementation follow-up: it identifies the inner delegated-child executor boundary that can create the wrong session_key, rather than proposing another generic queue-consumer routing fix. It is related to both, but is not a duplicate of their symptom descriptions.
• #65407 (and its open #65409) covers residual readers that prefer process-global environment values over already-bound task-local ContextVars. This report is distinct: the inner worker has no copied ContextVar in the first place, so the existing unbound-context fallback is reached. The proposed scope is to propagate the context at the two delegate call sites, not to remove the intentional CLI/cron fallback globally.
• #71472 is the closest overlap. Its requested two-site context propagation is closed as implemented on upstream main, and the current upstream source confirms both wrappers. This draft is not a competing root-cause report or a claim of a new fix; it records that the affected checkout still lacks those sites and asks for release/deployment readback plus call-site regression coverage. If version-specific tracking is not useful to maintainers, this draft should be closed in favor of fix(delegate): propagate session ContextVars across the subagent executor boundary #71472.
• #75856 is merged and addresses delegated-child SESSION_ID clobbering during child construction. That is related session-identity hardening, but it does not replace propagation of the session-key ContextVar across the inner timeout/batch executor submissions or verify ProcessSession.session_key completion routing.

Proposed Fix / Regression Test

The minimal implementation direction is to snapshot the parent context before each inner submit and invoke the target through context.run (or an equivalent context-propagation wrapper), at both the timeout and batch sites. Preserve the executor's non-interactive delegated-child approval callback semantics; do not replace it with a wrapper that can reintroduce prompt callbacks on a worker thread. Do not remove the process-environment fallback wholesale because CLI/cron compatibility intentionally uses it when no session ContextVar is bound.

Please add a call-site regression matrix that:

1. Sets a true parent session ContextVar and a deliberately different HERMES_SESSION_KEY environment value.
2. Exercises a single delegated child that reaches the timeout executor, creates a background process, and asserts ProcessSession.session_key and the completion event equal the true parent.
3. Exercises the batch executor with at least two children and makes the same assertion for every child; cover the batch executor and each child's timeout executor.
4. Runs two concurrent parent contexts with deliberately swapped environment values and asserts that no child or completion event crosses parents.
5. Verifies that approval callback installation/cleanup and timeout/cancellation behavior remain unchanged.

Evidence and Limitations

• The affected checkout source shows both bare submissions and the fallback chain described above.
• The standard-library probe was executed and produced the output shown above; the propagated wrapper retained true-parent while the bare worker resolved env-foreign.
• GitHub REST API readback confirmed the issue/PR states above and confirmed that upstream main contains both copy_context().run sites.
• This is not a full UI or end-to-end cross-session replay. The probe proves the executor/context mechanism, while the source chain explains how it reaches ProcessSession and completion routing. It does not by itself prove that every wrong-session incident has this exact cause.
• No private paths, credentials, debug links, profile names, or incident-specific session data are included.