Summary
When exporting translations to JSON, supplementary-plane (astral) characters — most modern emoji (👋 😀 🎉 🚀 …) — are emitted as \uXXXX\uXXXX surrogate-pair escape sequences instead of raw UTF-8. BMP characters (e.g. ⭐ U+2B50) and all of YAML export are written raw.
We want JSON export to escape only characters that actually require escaping in JSON (", \, control chars U+0000–U+001F), and pass everything else — including astral emoji — through as raw UTF-8.
Reproduction
Export keys with these values:
| key |
value |
code point |
| star |
⭐ |
U+2B50 (BMP, single UTF-16 unit) |
| wave |
👋 |
U+1F44B (supplementary, surrogate pair) |
JSON output (wave wrongly escaped):
{
"star": "⭐",
"wave": "\uD83D\uDC4B"
}
YAML output (correct):
Behavior is identical with ICU placeholder support on or off — it is not the message/ICU conversion.
Root cause
GenericStructuredFileExporter serializes JSON via objectMapper.writer(...).writeValueAsBytes(...). The byte path uses Jackson's UTF8JsonGenerator, which escapes supplementary-plane characters by default. The Writer/String path does not:
writeValueAsString (WriterBasedJsonGenerator) -> {"wave":"👋"} raw
writeValueAsBytes (UTF8JsonGenerator) -> {"wave":"\uD83D\uDC4B"} escaped
backend/data/src/main/kotlin/io/tolgee/formats/genericStructuredFile/out/GenericStructuredFileExporter.kt
YAML is unaffected because it goes through SnakeYAML (allowUnicode defaults to true), not UTF8JsonGenerator. ESCAPE_NON_ASCII is not enabled (BMP non-ASCII like ⭐ comes out raw).
Impact
- Output is valid JSON and round-trips (
\uD83D\uDC4B decodes back to 👋), so this is cosmetic, not data loss.
- But it is surprising and inconsistent with YAML, bloats output, and hurts readability/diffing of exported files. Reported by a customer.
Proposed fix
Make JSON export escape only JSON-required characters and emit astral chars raw. Options:
- Serialize via the Writer/String path (then UTF-8 encode) instead of
writeValueAsBytes, or
- Configure the JSON generator /
JsonFactory so the byte generator does not escape supplementary characters.
Keep YAML behavior unchanged. Add a regression test covering a BMP char, an astral emoji, and the JSON-required escapes (", \, control char) for both JSON and YAML.
Summary
When exporting translations to JSON, supplementary-plane (astral) characters — most modern emoji (👋 😀 🎉 🚀 …) — are emitted as
\uXXXX\uXXXXsurrogate-pair escape sequences instead of raw UTF-8. BMP characters (e.g. ⭐U+2B50) and all of YAML export are written raw.We want JSON export to escape only characters that actually require escaping in JSON (
",\, control charsU+0000–U+001F), and pass everything else — including astral emoji — through as raw UTF-8.Reproduction
Export keys with these values:
U+2B50(BMP, single UTF-16 unit)U+1F44B(supplementary, surrogate pair)JSON output (
wavewrongly escaped):YAML output (correct):
Behavior is identical with ICU placeholder support on or off — it is not the message/ICU conversion.
Root cause
GenericStructuredFileExporterserializes JSON viaobjectMapper.writer(...).writeValueAsBytes(...). The byte path uses Jackson'sUTF8JsonGenerator, which escapes supplementary-plane characters by default. The Writer/String path does not:backend/data/src/main/kotlin/io/tolgee/formats/genericStructuredFile/out/GenericStructuredFileExporter.ktYAML is unaffected because it goes through SnakeYAML (
allowUnicodedefaults to true), notUTF8JsonGenerator.ESCAPE_NON_ASCIIis not enabled (BMP non-ASCII like ⭐ comes out raw).Impact
\uD83D\uDC4Bdecodes back to 👋), so this is cosmetic, not data loss.Proposed fix
Make JSON export escape only JSON-required characters and emit astral chars raw. Options:
writeValueAsBytes, orJsonFactoryso the byte generator does not escape supplementary characters.Keep YAML behavior unchanged. Add a regression test covering a BMP char, an astral emoji, and the JSON-required escapes (
",\, control char) for both JSON and YAML.