Skip to content

Commit 1fe7758

Browse files
committed
feat(llm): player-facing error text separate from diagnostics; v6.9.0
A failed LLM call has two audiences and used to get one string. CoreAiChatPanel pasted `exception.Message` into the transcript, so a player read `❌ HTTP error 403: {"error":{"message":"..."}}` — transport prefix plus a JSON blob, truncated at the exception cap — while the log got that same short string and never saw the provider body. LlmErrorPresentation (portable core) splits them: - ToUserMessage(ex) — one sentence for the bubble. A message the BACKEND authored for the player wins, parsed out of LlmClientException.ProviderErrorBody: a gateway that already says "the teacher is unavailable, try again in a minute" knows the product better than this library. Otherwise a phrase per LlmErrorCode, with the provider's retry window folded into the RateLimited text. - ToDiagnosticText(ex) — typed code, HTTP status, retry hint and the raw body, for the log line next to the exception itself. Guardrails: JSON dumps, stack traces and bodies over 400 chars are diagnostics, not UI text, and fall back to the typed phrase; a 401 body reaches neither the player nor the log, because providers echo the submitted key back in invalid-credentials responses (same redaction the HTTP adapters already apply). CoreAiChatPanel now logs before it renders, and exposes ResolveErrorMessage / ResolveStreamErrorMessage as protected virtual hooks next to ResolveTimeoutMessage, so a game can localize the text without being able to hide diagnostics. Errors arriving inside a stream chunk are logged too and lose the "HTTP error 500:" prefix. Verified: LlmErrorPresentationEditModeTests 8/8 (run standalone against the built CoreAI.Core, since the editor holds the project lock); dotnet build gates on a clean worktree — CoreAI.Core, CoreAI.Source and CoreAI.Core.Tests, 0 errors. Full EditMode suite runs on next editor start. All six packages bumped in lockstep.
1 parent 7f8f37e commit 1fe7758

15 files changed

Lines changed: 470 additions & 22 deletions

File tree

Assets/CoreAI/CHANGELOG.md

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,29 @@
11
# Changelog
22

3+
## [6.9.0] - 2026-07-29
4+
5+
One failure, two audiences: a sentence for the player, everything for the log.
6+
7+
### Added
8+
9+
- **`LlmErrorPresentation`** (portable core, no Unity types). Turns a failed LLM call into the two
10+
strings it always needed:
11+
- `ToUserMessage(exception)` — one readable sentence for a chat bubble. A message the BACKEND
12+
authored for the player wins (parsed from `error.message` in `LlmClientException.ProviderErrorBody`),
13+
because a gateway that already says "the teacher is unavailable, try again in a minute" knows the
14+
product better than this library. Otherwise a phrase per `LlmErrorCode` is used — with the
15+
provider's retry window folded into the `RateLimited` text.
16+
- `ToDiagnosticText(exception)` — typed code, HTTP status, retry hint and the raw provider body,
17+
for the log line next to the exception itself.
18+
- `ExtractProviderMessage` / `StripHttpErrorPrefix` are public: hosts that render errors themselves
19+
reuse the same parsing instead of string-matching `"HTTP error "`.
20+
- Guardrails, all covered by `LlmErrorPresentationEditModeTests`:
21+
- JSON dumps, stack traces and bodies longer than `MaxUserMessageLength` (400) are diagnostics, not
22+
UI text — they fall back to the typed phrase, so a player never reads `{"error":{...}}`.
23+
- **A 401 body reaches neither the player nor the log.** Providers echo the submitted key back in
24+
invalid-credentials responses, so the redaction the HTTP adapters already apply to log lines now
25+
applies here too (`body=[redacted auth error body]`; the bubble shows the "sign in again" phrase).
26+
327
## [6.8.3] - 2026-07-29
428

529
The real fix for "mods spawn nothing in a build", and a correction: 6.8.2 blamed the wrong cause.
Lines changed: 225 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,225 @@
1+
using System;
2+
using Newtonsoft.Json.Linq;
3+
4+
namespace CoreAI.Ai
5+
{
6+
/// <summary>
7+
/// Turns an LLM failure into two different strings: one for the PLAYER and one for the LOG.
8+
///
9+
/// WHY: chat UIs used to paste <c>exception.Message</c> straight into the transcript, so a player
10+
/// saw <c>HTTP error 403: {"error":{"message":"..."}}</c> — technical, often truncated, and useless
11+
/// to them — while the log got the same short string and lost the provider body. This type splits
12+
/// the two audiences:
13+
/// <list type="bullet">
14+
/// <item><see cref="ToUserMessage(Exception, string)"/> — one readable sentence. A server-authored
15+
/// message wins (a gateway that already says "the teacher is unavailable, try again in a minute"
16+
/// knows the product better than this library); otherwise a per-<see cref="LlmErrorCode"/>
17+
/// phrase is used.</item>
18+
/// <item><see cref="ToDiagnosticText"/> — everything worth keeping: error code, HTTP status,
19+
/// retry hint and the raw provider body.</item>
20+
/// </list>
21+
/// Portable core: no Unity types, so the same mapping is available to any host or headless test.
22+
/// </summary>
23+
public static class LlmErrorPresentation
24+
{
25+
/// <summary>Fallback shown when nothing more specific is known.</summary>
26+
public const string DefaultUserMessage = "The assistant is unavailable right now. Please try again in a moment.";
27+
28+
/// <summary>Longest server-authored message that is shown to the player as-is.</summary>
29+
public const int MaxUserMessageLength = 400;
30+
31+
/// <summary>One readable sentence for the chat bubble. Never returns null or empty.</summary>
32+
public static string ToUserMessage(Exception exception, string fallback = null)
33+
{
34+
if (exception is LlmClientException llmException)
35+
{
36+
return ToUserMessage(llmException, fallback);
37+
}
38+
39+
if (exception is OperationCanceledException)
40+
{
41+
return ForErrorCode(LlmErrorCode.Cancelled);
42+
}
43+
44+
return Coalesce(fallback, DefaultUserMessage);
45+
}
46+
47+
/// <summary>One readable sentence for the chat bubble, from a typed LLM failure.</summary>
48+
public static string ToUserMessage(LlmClientException exception, string fallback = null)
49+
{
50+
if (exception == null)
51+
{
52+
return Coalesce(fallback, DefaultUserMessage);
53+
}
54+
55+
// WHY: a 401 body can echo the submitted key/token back (providers do this), so its text
56+
// never reaches the transcript — the player gets the "sign in again" phrase instead.
57+
// Same rule as the redaction in the HTTP adapters; see MeaiOpenAiChatClient.BuildHttpException.
58+
if (IsAuthFailure(exception))
59+
{
60+
return Coalesce(fallback, ForErrorCode(LlmErrorCode.AuthExpired));
61+
}
62+
63+
// A gateway/provider message aimed at the player wins over any built-in phrase.
64+
string authored = ExtractProviderMessage(exception.ProviderErrorBody);
65+
if (string.IsNullOrWhiteSpace(authored))
66+
{
67+
authored = StripHttpErrorPrefix(exception.Message);
68+
}
69+
70+
if (IsPresentableToPlayer(authored))
71+
{
72+
return authored.Trim();
73+
}
74+
75+
return Coalesce(fallback, ForErrorCode(exception.ErrorCode, exception.RetryAfterSeconds));
76+
}
77+
78+
/// <summary>Built-in phrase for a failure category; used when nobody authored a better one.</summary>
79+
public static string ForErrorCode(LlmErrorCode errorCode, int? retryAfterSeconds = null)
80+
{
81+
string retryHint = retryAfterSeconds.HasValue && retryAfterSeconds.Value > 0
82+
? $" Try again in {retryAfterSeconds.Value} s."
83+
: "";
84+
85+
switch (errorCode)
86+
{
87+
case LlmErrorCode.Timeout:
88+
return "The assistant took too long to answer. Please try again.";
89+
case LlmErrorCode.Cancelled:
90+
return "The request was stopped.";
91+
case LlmErrorCode.EmptyResponse:
92+
return "The assistant returned an empty answer. Please try again.";
93+
case LlmErrorCode.AuthExpired:
94+
return "The session has expired. Please sign in again.";
95+
case LlmErrorCode.QuotaExceeded:
96+
return "The assistant quota for this account is used up.";
97+
case LlmErrorCode.RateLimited:
98+
return "Too many requests to the assistant right now." + retryHint;
99+
case LlmErrorCode.BackendUnavailable:
100+
return "The assistant is unavailable right now. Please try again in a moment.";
101+
case LlmErrorCode.InvalidRequest:
102+
return "The assistant could not process this request.";
103+
case LlmErrorCode.RoutingError:
104+
return "No assistant backend is configured. Please tell the maintainer.";
105+
case LlmErrorCode.ContextLengthExceeded:
106+
return "The conversation got too long for the model. Start a new chat or shorten the message.";
107+
default:
108+
return DefaultUserMessage;
109+
}
110+
}
111+
112+
/// <summary>
113+
/// Everything worth writing to the log: category, HTTP status, retry hint and the raw provider
114+
/// body. Callers should log this ALONGSIDE the exception itself (which carries the stack trace).
115+
/// </summary>
116+
public static string ToDiagnosticText(Exception exception)
117+
{
118+
if (!(exception is LlmClientException llmException))
119+
{
120+
return exception == null ? "" : exception.ToString();
121+
}
122+
123+
string status = llmException.HttpStatus.HasValue ? $" http={llmException.HttpStatus.Value}" : "";
124+
string retry = llmException.RetryAfterSeconds.HasValue
125+
? $" retryAfter={llmException.RetryAfterSeconds.Value}s"
126+
: "";
127+
// Same redaction rule as the HTTP adapters: an auth-failure body can contain the key
128+
// that was just sent, and logs travel further than the process.
129+
string body;
130+
if (string.IsNullOrWhiteSpace(llmException.ProviderErrorBody))
131+
{
132+
body = "";
133+
}
134+
else if (IsAuthFailure(llmException))
135+
{
136+
body = " body=[redacted auth error body]";
137+
}
138+
else
139+
{
140+
body = $" body={llmException.ProviderErrorBody}";
141+
}
142+
143+
return $"code={llmException.ErrorCode}{status}{retry} message={llmException.Message}{body}";
144+
}
145+
146+
/// <summary>Reads <c>error.message</c> (OpenAI-compatible shape) out of a raw provider body.</summary>
147+
public static string ExtractProviderMessage(string providerErrorBody)
148+
{
149+
if (string.IsNullOrWhiteSpace(providerErrorBody))
150+
{
151+
return "";
152+
}
153+
154+
try
155+
{
156+
JObject parsed = JObject.Parse(providerErrorBody);
157+
string message = parsed["error"]?["message"]?.ToString();
158+
if (string.IsNullOrWhiteSpace(message))
159+
{
160+
message = parsed["message"]?.ToString() ?? parsed["detail"]?.ToString();
161+
}
162+
163+
return string.IsNullOrWhiteSpace(message) ? "" : message.Trim();
164+
}
165+
catch (Exception)
166+
{
167+
// Not JSON (HTML error page, proxy text, truncated body) — the caller falls back to
168+
// the exception message, so a parse failure must never surface as a second error.
169+
return "";
170+
}
171+
}
172+
173+
/// <summary>
174+
/// Drops the <c>HTTP error 403: </c> prefix that HTTP adapters put in front of the provider text,
175+
/// so a message authored for the player is not shown with transport noise in front of it.
176+
/// </summary>
177+
public static string StripHttpErrorPrefix(string message)
178+
{
179+
if (string.IsNullOrWhiteSpace(message))
180+
{
181+
return "";
182+
}
183+
184+
const string marker = "HTTP error ";
185+
if (!message.StartsWith(marker, StringComparison.OrdinalIgnoreCase))
186+
{
187+
return message.Trim();
188+
}
189+
190+
int separator = message.IndexOf(':', marker.Length);
191+
return separator < 0 || separator + 1 >= message.Length
192+
? message.Trim()
193+
: message.Substring(separator + 1).Trim();
194+
}
195+
196+
/// <summary>
197+
/// Is this string something a player should read? JSON dumps, stack traces and novel-length
198+
/// bodies are diagnostics, not UI text — those fall back to the built-in phrase.
199+
/// </summary>
200+
private static bool IsPresentableToPlayer(string message)
201+
{
202+
if (string.IsNullOrWhiteSpace(message) || message.Length > MaxUserMessageLength)
203+
{
204+
return false;
205+
}
206+
207+
string trimmed = message.Trim();
208+
char first = trimmed[0];
209+
if (first == '{' || first == '[' || first == '<')
210+
{
211+
return false;
212+
}
213+
214+
return !trimmed.Contains("\n at ", StringComparison.Ordinal)
215+
&& !trimmed.Contains("Exception:", StringComparison.Ordinal);
216+
}
217+
218+
/// <summary>401-class failure: its body is treated as secret-bearing everywhere.</summary>
219+
private static bool IsAuthFailure(LlmClientException exception) =>
220+
exception.HttpStatus == 401 || exception.ErrorCode == LlmErrorCode.AuthExpired;
221+
222+
private static string Coalesce(string preferred, string fallback) =>
223+
string.IsNullOrWhiteSpace(preferred) ? fallback : preferred.Trim();
224+
}
225+
}

Assets/CoreAI/Runtime/Core/Features/Llm/LlmErrorPresentation.cs.meta

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
using System;
2+
using CoreAI.Ai;
3+
using NUnit.Framework;
4+
5+
namespace CoreAI.Core.Tests.EditMode
6+
{
7+
/// <summary>
8+
/// Guards the split between the string a PLAYER reads and the string a LOG keeps
9+
/// (<see cref="LlmErrorPresentation"/>): a backend-authored message must survive to the chat
10+
/// bubble, raw JSON/stack noise must not, and a 401 body must never leave the log — it can echo
11+
/// the key that was just submitted.
12+
/// </summary>
13+
public sealed class LlmErrorPresentationEditModeTests
14+
{
15+
[Test]
16+
public void UserMessage_PrefersBackendAuthoredMessageFromProviderBody()
17+
{
18+
LlmClientException exception = new(
19+
"HTTP error 403: Teacher is unavailable: AI service answered 403.",
20+
LlmErrorCode.ProviderError,
21+
403,
22+
null,
23+
"{\"error\":{\"code\":\"ai_upstream_error\",\"message\":\"Teacher is unavailable, try again in a minute.\"}}");
24+
25+
Assert.AreEqual(
26+
"Teacher is unavailable, try again in a minute.",
27+
LlmErrorPresentation.ToUserMessage(exception));
28+
}
29+
30+
[Test]
31+
public void UserMessage_StripsHttpPrefixWhenBodyIsNotJson()
32+
{
33+
LlmClientException exception = new(
34+
"HTTP error 502: Upstream gateway is down.",
35+
LlmErrorCode.BackendUnavailable,
36+
502,
37+
null,
38+
"<html>502 Bad Gateway</html>");
39+
40+
Assert.AreEqual("Upstream gateway is down.", LlmErrorPresentation.ToUserMessage(exception));
41+
}
42+
43+
[Test]
44+
public void UserMessage_FallsBackToTypedPhraseWhenTextIsDiagnosticNoise()
45+
{
46+
LlmClientException exception = new(
47+
"HTTP error 500: {\"error\":{\"metadata\":{\"raw\":\"...\"}}}",
48+
LlmErrorCode.BackendUnavailable,
49+
500,
50+
null,
51+
"{\"error\":{\"metadata\":{\"raw\":\"...\"}}}");
52+
53+
string message = LlmErrorPresentation.ToUserMessage(exception);
54+
55+
Assert.AreEqual(LlmErrorPresentation.ForErrorCode(LlmErrorCode.BackendUnavailable), message);
56+
StringAssert.DoesNotContain("{", message);
57+
}
58+
59+
[Test]
60+
public void UserMessage_NeverEchoesAuthFailureBody()
61+
{
62+
LlmClientException exception = new(
63+
"HTTP error 401: invalid api key sk-secret-value",
64+
LlmErrorCode.AuthExpired,
65+
401,
66+
null,
67+
"{\"error\":{\"message\":\"Invalid API key sk-secret-value\"}}");
68+
69+
string message = LlmErrorPresentation.ToUserMessage(exception);
70+
71+
Assert.AreEqual(LlmErrorPresentation.ForErrorCode(LlmErrorCode.AuthExpired), message);
72+
StringAssert.DoesNotContain("sk-secret-value", message);
73+
}
74+
75+
[Test]
76+
public void UserMessage_UsesRetryHintWhenProviderSuppliedOne()
77+
{
78+
LlmClientException exception = new(
79+
"HTTP error 429: {\"error\":{}}",
80+
LlmErrorCode.RateLimited,
81+
429,
82+
14,
83+
"{\"error\":{}}");
84+
85+
StringAssert.Contains("14", LlmErrorPresentation.ToUserMessage(exception));
86+
}
87+
88+
[Test]
89+
public void UserMessage_HandlesPlainExceptionsAndCancellation()
90+
{
91+
Assert.AreEqual(
92+
LlmErrorPresentation.DefaultUserMessage,
93+
LlmErrorPresentation.ToUserMessage(new InvalidOperationException("boom")));
94+
Assert.AreEqual(
95+
LlmErrorPresentation.ForErrorCode(LlmErrorCode.Cancelled),
96+
LlmErrorPresentation.ToUserMessage(new OperationCanceledException()));
97+
Assert.AreEqual("Custom fallback.", LlmErrorPresentation.ToUserMessage(null, "Custom fallback."));
98+
}
99+
100+
[Test]
101+
public void DiagnosticText_KeepsEverythingExceptAuthBodies()
102+
{
103+
LlmClientException providerError = new(
104+
"HTTP error 403: blocked",
105+
LlmErrorCode.ProviderError,
106+
403,
107+
7,
108+
"{\"error\":{\"message\":\"blocked\"}}");
109+
110+
string diagnostics = LlmErrorPresentation.ToDiagnosticText(providerError);
111+
112+
StringAssert.Contains("code=ProviderError", diagnostics);
113+
StringAssert.Contains("http=403", diagnostics);
114+
StringAssert.Contains("retryAfter=7s", diagnostics);
115+
StringAssert.Contains("\"message\":\"blocked\"", diagnostics);
116+
117+
LlmClientException authError = new(
118+
"HTTP error 401: invalid key",
119+
LlmErrorCode.AuthExpired,
120+
401,
121+
null,
122+
"{\"error\":{\"message\":\"Invalid API key sk-secret-value\"}}");
123+
124+
string redacted = LlmErrorPresentation.ToDiagnosticText(authError);
125+
126+
StringAssert.Contains("[redacted auth error body]", redacted);
127+
StringAssert.DoesNotContain("sk-secret-value", redacted);
128+
}
129+
130+
[Test]
131+
public void StripHttpErrorPrefix_LeavesOrdinaryTextAlone()
132+
{
133+
Assert.AreEqual("Model is overloaded.", LlmErrorPresentation.StripHttpErrorPrefix("Model is overloaded."));
134+
Assert.AreEqual("Model is overloaded.", LlmErrorPresentation.StripHttpErrorPrefix("HTTP error 503: Model is overloaded."));
135+
Assert.AreEqual("", LlmErrorPresentation.StripHttpErrorPrefix(null));
136+
}
137+
}
138+
}

0 commit comments

Comments
 (0)