incorrect character set mapping for some escape sequences

[GorillaSapiens]

refer to https://llvm-mos.org/wiki/Character_set

and also code in mos-platform/c64/charset.h and similar.

C17 §5.1.1.2 and §6.4.5 require that ordinary characters are mapped via the source character set, while escape sequences are converted directly to execution character set members, whose values are implementation-defined; therefore "A\x41" may produce two different characters.

this applies to all hex and octal escape sequences. when a programmer puts \x41, they really want that exact byte, not an 'A'.

the method on the Character_set wiki page simply will not work, because by the time those operator"" get called, the "\x41" has already been translated to "A".

the correct way to handle this is to use raw strings, and translate the escape sequences ourselves in our operator""_foo functions.

the correct wiki examples would then be RU"HELLO"_foo

and additional code would be needed in

./mos-platform/atari8-common/charset.h
./mos-platform/cx16/charset.h
./mos-platform/pet/charset.h
./mos-platform/vic20/charset.h
./mos-platform/c64/charset.h

[mysterymath]

We explicitly take the POV in llvm-mos that the source and execution character sets are always UTF-8. This is because LLVM simply does not support non-UTF-8 execution character sets. IIRC there's some tolerance for line endings, but broadly, the source is also expected to be UTF-8. There has been a lot of upstream discussion by IBM about adding in the GCC options for this, but the LLVM community has been very resistant to opening that can of worms, and I'm similarly resistant to introducing that complexity into our fork.

Accordingly, until the above changes, we can be guaranteed that "A\x41" always produces two identical characters.

[GorillaSapiens]

note that string concatenation won't work around this.

U"Hello"_foo "\x20" U"World!"_foo; // won't work

conceptually: U"Hello" "\x20" U"World!" concatenates into one U"Hello\x20World!"

then the suffix gets applied: U"Hello\x20World!"_foo

then one operator""_foo(const char32_t*, size_t) gets called

[GorillaSapiens]

the way i'd like to address this is leave the existing operator""_foo() functions "as is" to handle U""_foo constructs, and add additional operator""_rawfoo() functions to handle UR""_rawfoo constructs. the raw functions would decode but not translate the hex and octal escape sequences.

[mysterymath]

the way i'd like to address this is leave the existing operator""_foo() functions "as is" to handle U""_foo constructs, and add additional operator""_rawfoo() functions to handle RU""_rawfoo constructs. the raw functions would decode but not translate the hex and octal escape sequences.

Can you clarify what the actual issue is you're addressing, given that source and executions character sets are both always UTF-8?

[GorillaSapiens]

Can you clarify what the actual issue is you're addressing, given that source and executions character sets are both always UTF-8?

That assumption isn’t correct for the target I’m discussing.

While the host environment (editor, compiler frontend, repo) may be UTF-8, the execution character set of the target machine is not. On a PETSCII-based system, source character literals like 'A' and "A" are mapped into the target’s execution character set during compilation, whereas \xNN is a numeric byte escape that explicitly inserts the byte value 0xNN into the object code.

Those two only coincide if the execution character set happens to encode A as 0x41, which is not something I want to assume or guarantee. On PETSCII systems, it is both historically accurate and practically useful for character literals and numeric byte escapes to live in different abstraction layers.

So the issue being addressed is precisely this:
\xNN must mean “byte value NN”, not “the character whose encoding is NN”.
If you collapse those meanings under the assumption of UTF-8 everywhere, you lose the ability to safely mix textual data and raw protocol/control bytes in string literals… something PETSCII-era systems did constantly.

In short, UTF-8 may describe the source encoding, but it does not define the execution character set, and that distinction matters here.

[mysterymath]

Source and execution character sets are terms from the C/C++ standard, and the nature of those character sets are largely implementation-defined. We are the implementation, so we must define them.

We don't have an arbitrary choice of how to define them, because clang, and much of LLVM's middle-end, defines them both to be UTF-8. Since we're an optimizing compiler, to change this, we'd have to contend with every optimization/transform that assumes that they are UTF-8. This is a task that IBM with its resources has not had a very good time with, and our project doesn't really have a prayer. At the end of the day, we're a fork of LLVM, and we have only so many "weirdness points" to spend. Maybe someday, who knows.

Accordingly, we've taken another out. We consider the execution character set to be UTF-8, but we do Windows-style translation for text streams at runtime to the actual target character set. The C++ libraries are provided for use with raw interfaces to avoid that overhead, but they still operate within a regime where the source and character sets are UTF-8. Namely, they take UTF as input, and output wholly uninterpreted bytes from the language's point of view.