Cedar authz with embedded auth server silently denies everything when the upstream IdP issues JWT access tokens without the referenced identity claims (e.g. `email`)

[alex-feel]

Bug description

When a workload uses the embedded auth server, the Cedar authorization middleware sources the principal's claims from the raw upstream IdP access token instead of the AS-issued JWT that the client actually presented. If the upstream provider's access tokens are JWT-shaped but do not carry the identity claims that policies reference (many OIDC providers put email only in the id_token, not the access token — JetBrains Hub is a concrete public example), every Cedar policy referencing those claims silently denies all requests, while the audit middleware keeps logging the correct user identity from the AS-issued JWT.

Code chain (file:line at v0.34.0, commit b084be45; byte-identical in the relevant functions at v0.37.0 and at main = 193a7f29d40f03e8c5276f7abc7da89f3342e758, checked 2026-07-22):

1. pkg/runner/middleware.go:192 → injectUpstreamProviderIfNeeded (pkg/runner/middleware.go:468) sets the Cedar authorizer's PrimaryUpstreamProvider to the first upstream whenever config.EmbeddedAuthServerConfig != nil. There is no opt-out.
2. cedar.InjectUpstreamProvider (pkg/authz/authorizers/cedar/core.go:79) overwrites any user-supplied value unconditionally (core.go:94). The CRD field primaryUpstreamProvider is documented as "silently ignored" on MCPServer and MCPRemoteProxy (cmd/thv-operator/api/v1beta1/mcpexternalauthconfig_types.go:420, docs/operator/crd-api.md), so on those kinds a user cannot select the claim source at all (only VirtualMCPServer gained an explicit field via Expose explicit primaryUpstreamProvider for Cedar authz on VirtualMCPServer #5197).
3. resolveClaims (pkg/authz/authorizers/cedar/core.go:486) then reads identity.UpstreamTokens[<provider>] (core.go:489), parses it without signature verification (parseUpstreamJWTClaims, core.go:559, jwt.Parser.ParseUnverified), and uses those claims as the Cedar principal attributes (prefixed claim_ by preprocessClaims, core.go:585-590).
4. The fallback to the AS-issued JWT's claims added in Fall back to request-token claims for opaque upstream tokens #5147 triggers only when the upstream token is not JWT-shaped: looksLikeJWT (core.go:537) checks for exactly two dots, and the fallback branch is gated on !looksLikeJWT(upstreamToken) (core.go:512-520). A JWT-shaped upstream access token that parses fine but lacks the referenced claims never falls back. (A missing upstream token is likewise a hard deny at core.go:489-494 — same silent-deny class.)

The frustrating part: the identity data the policy needs is already available in identity.Claims. The embedded auth server mirrors the upstream OIDC sub/name/email into its issued access token (pkg/authserver/server/session/session.go:133-137), and the audit middleware reads exactly that identity (pkg/audit/auditor.go:463-473, :493), which is why audit logs show the correct user while authorization denies the same request.

Steps to reproduce

1. Run a workload with the embedded auth server and a single OIDC upstreamProviders entry whose access tokens are JWTs that do not contain email (JetBrains Hub behaves this way: the access token is a JWT without an email claim; the email is present only in the id_token). Reproducible on Kubernetes with MCPServer/MCPRemoteProxy, and the same code path is shared by the CLI runner.
2. Attach a Cedar authorization config with a domain gate plus a tool permit — note the forbid policy is even authored defensively with has, as recommended by the code comment at core.go:508-511:

authzConfig:
  inline:
    policies:
      - |
        forbid(principal, action, resource)
        unless { principal has claim_email && principal.claim_email like "*@example.com" };
      - |
        permit(principal, action == Action::"call_tool", resource);

3. Complete the OAuth flow through the embedded auth server with a user whose upstream email matches the gated domain, then issue tools/list and tools/call with the AS-issued bearer token.

Expected behavior

The domain gate is evaluated against the user's email. The AS-issued JWT presented by the client carries email (mirrored from the upstream id_token by the embedded auth server), so the user sees the tool list and can call tools.

Actual behavior

Everything is denied, silently and for every user:

• tools/list returns {"tools":[]} — the middleware forwards list requests and filters each item against the call_tool policy, and every item fails the gate.
• tools/call returns HTTP 403 with body {"Result":null,"Error":{"code":403,"message":"Unauthorized"},"ID":{}} (the forbid policy applies cleanly because principal has claim_email is false against the upstream access token's claims, so there is no Cedar diagnostic error and no hint in the response).
• The audit middleware logs the correct user for the same denied requests (audit reads the AS-issued JWT identity), so audit and authz visibly diverge, which is confusing to operate.
• No warning is logged: the "upstream token is not a JWT; falling back to request-token claims" warning fires only for opaque tokens.

Why opaque-token providers mask this

With upstreams whose access tokens are opaque (Google ya29.*, GitHub gho_*), parseUpstreamJWTClaims fails, looksLikeJWT is false, and the authorizer falls back to the AS-issued JWT's claims — which do carry the mirrored email. So claim_email policies appear to work with Google/GitHub upstreams and break only when the upstream happens to issue JWT-shaped access tokens, which makes the behavior look provider-specific and hard to diagnose.

Environment (if relevant)

• ToolHive: observed live at v0.34.0 (operator-managed Kubernetes deployment); the code chain above is unchanged at v0.37.0 (git diff v0.34.0 v0.37.0 -- pkg/authz pkg/runner/middleware.go pkg/auth shows no change to claim resolution) and at main@193a7f29d40f03e8c5276f7abc7da89f3342e758.
• Upstream IdP: any OIDC provider issuing JWT-shaped access tokens without email (JetBrains Hub as a public example).

Additional context

Related prior work, none of which covers this case:

• Enforce Cedar Policies on Upstream IDP Token #4408 / PR Enforce Cedar policies on upstream IDP token claims #4448 introduced sourcing Cedar claims from the upstream IdP access token when the embedded auth server is active; Cedar policies cannot access upstream IDP claims on VirtualMCPServer with embedded auth server #4997 / PR Fix Cedar upstream-claim evaluation on VirtualMCPServer #5002 extended it to the Kubernetes path.
• VirtualMCPServer Cedar authorization rejects all requests when upstream OIDC provider issues opaque access tokens #5146 / PR Fall back to request-token claims for opaque upstream tokens #5147 fixed the deny-all for opaque upstream access tokens by falling back to request-token claims — but only for tokens that are not JWT-shaped, leaving this residual case: a well-formed upstream JWT that legitimately lacks the referenced identity claims.
• Expose explicit primaryUpstreamProvider for Cedar authz on VirtualMCPServer #5197 / PR Expose explicit primaryUpstreamProvider for Cedar authz on VirtualMCPServer #5199 added an explicit primaryUpstreamProvider (including the leave-empty opt-out) for VirtualMCPServer only; on MCPServer/MCPRemoteProxy the injection remains forced and the CRD field is documented "silently ignored".

Suggested fix directions (any one of these resolves the report; the first addresses the root cause):

1. Merge instead of replace: evaluate the upstream token's claims merged with the AS-issued JWT's claims (for example upstream claims win per key, AS-JWT claims fill the gaps). The AS-JWT already mirrors the upstream sub/email/name (pkg/authserver/server/session/session.go), so standard OIDC identity policies would keep working for every upstream token shape while upstream-only claims (groups, custom namespaced claims) remain available.
2. Extend the Fall back to request-token claims for opaque upstream tokens #5147 fallback: when a JWT-shaped upstream token parses successfully but lacks the standard identity claims (or lacks the attributes the policy set references), fall back to — or supplement with — the request-token claims instead of hard-committing to the upstream token.
3. Make the claim source selectable: honor primaryUpstreamProvider on MCPServer/MCPRemoteProxy including an explicit way to opt out of upstream-token evaluation (parity with what Expose explicit primaryUpstreamProvider for Cedar authz on VirtualMCPServer #5197 delivered for VirtualMCPServer), instead of force-injecting the first upstream.

[alex-feel]

Some runtime evidence from a v0.34.0 deployment, in case it helps triage: the branch this issue is about is directly observable in the proxy log, and the observable signal is inverted from what you would want.

Two endpoints on the same build, differing only in the upstream IdP's access-token format.

Opaque upstream access token (Google). A signed-in tools/call succeeds under a claim_email policy, and the proxy emits exactly one line:

{"level":"WARN","msg":"upstream token is not a JWT; falling back to request-token claims for Cedar evaluation","provider":"google"}

That is pkg/authz/authorizers/cedar/core.go:516, inside if !looksLikeJWT(upstreamToken) and wrapped in the claimKeyLog.Do(...) guard, so it appears once per process as intended.

JWT upstream access token (JetBrains Hub). The same policy shape denies everything, and that line never appears in the proxy log — the endpoint is on the other branch, where the upstream token's claims are used verbatim and email is simply not among them.

So the presence of that WARN is a reliable, zero-cost indicator that an endpoint is on the working fallback branch, and its absence is the only externally visible sign that a claim_email policy is being evaluated against a token that cannot satisfy it. Nothing is logged on the JWT branch to say which referenced claims were absent.

If a full fix is not imminent, a cheap intermediate improvement would be to log once on the JWT branch too, naming the claims the policy references that the upstream token does not carry. That would turn a silent deny-all into something diagnosable from logs alone. Today the only symptoms are tools/list returning an empty list and tools/call returning 403, alongside an audit trail that shows the correct user (audit reads the AS-issued JWT), which actively points an investigation away from authorization.