Built-in character type questions

[josh11b]

There are a number of questions:

• Is there a built-in character type?
• How big is it?
• Is it signed?
• What does it represent?
• How do you name it?
• How does it correspond to C++ char?
• Are there other character types with different sizes, signedness, or meaning?

[josh11b]

Proposed answers:

• Yes, there is a built-in character type.
• It is 8 bits.
• It is unsigned.
• It conventionally represents a UTF-8 code unit, but there is no enforcement that it has a valid value.
• This type would be named Core.Char and char would be a built-in keyword alias.
• It would be the same type as Cpp.char. The Carbon toolchain would use unsigned C++ char by default. If the user opted into signed C++ char, the signedness would be different for the same type on the C++ and Carbon sides of the interop boundary.
• Carbon would support Cpp.char8_t, Cpp.char16_t, and Cpp.char32_t as names for the corresponding C++ types.

It is future work to decide if there would be any other character types (like Core.Char16 or Core.Char32) in Carbon outside of the Cpp namespace.

[chandlerc]

Marking as decided -- I think this is a good extraction of the current char type consensus among leads.

Rationale (me summarizing what I heard in discussion):

• Minimal contentious decisions at this stage
• Strongly reflects expected reality of char data such as C-strings and std::string contents
• Reasonable interop story for signedness, while preserving significantly preferred semantics in Carbon
• Leaves doors open for us to nail down exactly how other types and encodings work

[geoffromer]

It is future work to decide if there would be any other character types (like Core.Char16 or Core.Char32) in Carbon outside of the Cpp namespace.

Does this decision intentionally override the decision in proposal #1964 that Carbon would have Core.Char8, Core.Char16, and Core.Char32 types? If so, is it undoing the rest of that proposal as well? Surely the intent isn't that we support code point literal constants, but haven't decided whether to support code points at run-time?

[burakemir]

"character type" evokes three possible meanings to me:

1. a type (or set of types) used for character literals.
2. a type to represent Unicode characters - or parts of Unicode characters - in isolation, outside of a string.
3. something that corresponds to C++ "char" for the purpose of interop

#1964 talks about 1. and a bit about 2. It is not actually clear to me if "builtin character type" in this issue talks about "type backing character literals" or not. So I second Geoff's request for clarification.

Since signedness of Cpp.char is discussed and "Core.Char*" suggested to be separate, I believe this issue is more about 3.

Point 3 is subtle as char, unsigned char and signed char are different types in C++. This mismatch creates problems for Rust C++ interop as observed during work on Crubit project.

The cpp reference says ...

| The signedness of char depends on the compiler and the target platform: the defaults for ARM and PowerPC are typically unsigned, the defaults for x86 and x64 are typically signed.

... so, IMHO it might be prudent to make char configurable through an -fsigned-char flag.

Fixing it to be signed or unsigned creates interop problems whenever an existing C++ (or C) code base did the opposite in the desire to follow the platform constraint. In other words, fixing the signedness may create unintended effects when code that has been designed to follow platform-choices (in particular, code that has to interface with old C system code.)

[josh11b]

@burakemir I disagree that Carbon signedness of chars needs to match C++, as long as the binary representation of each character is the same. Further, it is important that Carbon not replicate the unnecessary complexity of C++ wherever possible. I think mandating that Carbon's Char is unsigned is an example of precisely the things we should be doing.

[burakemir]

About signedness:

It is less about C++ directly but C interoperability, e.g. calling libc functions to interact with the operating system. These use char* and const char* in a lot of places. If we are certain that our spec can force our interpretation of what char signedness in the place where these headers are parsed - presumably by an existing C/C++ parser - then I don't see much of an issue with fixing char to be unsigned.

For a clang based implementation, I supposed this amounts to either failing when encountering -fsigned-char or somehow working around the -fsigned-char setting.

orthogonality:
Why not use (the Carbon equivalent of) uint8_t for those places where C / C++ code was using "char"?

A well-designed language should have primitives types for those purposes that cannot be addressed using other, existing primitive types or user-defined types.

About "character type":

Core.Char should be different from Cpp.Char.

IMHO, something called a "character" type should be connected to character literals and the string API. I am assuming there is a desire to support Unicode, in both source encoding and actual standard library.

In a world of Unicode characters, a character literal (= the textual representation of what the language defines as character type) can reasonably be expected to support Unicode characters, and strings can reasonably be expected to support Unicode. This has of course a few far-reaching implications.

When we have a string literal "こんにちは" (konnichiwa) and want to get the first character, it seems reasonable to expect that:

• this returns an instance of a type that can represent any Unicode scalar value (up to 4 bytes)
• that this value is an instance of a character type, and
• that this character type is actually the built-in character type, because it is a type for representing characters

When we have a character literal '😤' (U+1F624 Face with Steam From Nose), we may expect this to be

• to be represented by a built-in type, since any literal has to correspond to a built-in type
• that this built-in type is actually a character type, because it is a character literal
• that this character type is actually the built-in character type, because of orthogonality

Rust uses std::char for this purpose, it is always 4 bytes wide.

Sometimes we want to work with Unicode scalar values. Then one can just cast the value of the character type to a suitable integer type.

Converstion in the other direction is dangerous due to invalid values. Instead of permitting casts, which may lead to invalid characters and invalid strings and safety issues, there has to be a fallible conversion function that accounts for number values that are not legal characters, see e.g. std::char::from_u32 in Rust https://doc.rust-lang.org/std/char/fn.from_u32.html

With the expectations above, the Rust answers are unsurprising and worth replicating IMHO:

  |     let ch = 1 + '😤';
  |                ^ no implementation for `{integer} + char`

let ch : u32 = 1 + '😤' as u32;
passes, prints 128549

let ch : char = (1 + '😤' as u32) as char;
error[E0604]: only `u8` can be cast as `char`, not `u32`
 --> src/main.rs:4:21
  |
4 |     let ch : char = (1 + '😤' as u32) as char;
  |                     ^^^^^^^^^^^^^^^^^^^^^^^^^ invalid cast

Rust playground link: https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=2143089b0ac102ea98d7873b4ba44960

Note that when there is a desire for static safety guarantees, then the need for proper Unicode support cannot be handwaived away by saying "not enforced conventions". Any modern language will offer a library to deal with manipulation of Unicode characters and strings, and these Unicode character- and string-manipulating libraries will need to assume valid bit sequences.

If that is all left to convention instead of types, there are two choices:

• either the implementations have to be very defensive (which costs a lot of performance) or they can exhibit undefined behavior, at which point there is no safety guarantee left.
• or they are not defensive, at which point there is usually undefined behavior can there is no longer any safety guarantee.

[jonmeow]

It is future work to decide if there would be any other character types (like Core.Char16 or Core.Char32) in Carbon outside of the Cpp namespace.

Does this decision intentionally override the decision in proposal #1964 that Carbon would have Core.Char8, Core.Char16, and Core.Char32 types? If so, is it undoing the rest of that proposal as well? Surely the intent isn't that we support code point literal constants, but haven't decided whether to support code points at run-time?

@geoffromer My understanding at present is that a lot of #1964 needs to be redone, and it'd probably be good to decide in the context of std::char8_t, std::char16_t, and std::char32_t what design is really best for Unicode. Right now though I'm focused on char, and the issues found as part of implementing #5901 (with an eye on unblocking interop). I was planning on a proposal to clean up some of these issues, but #5922 seems important for that.

@burakemir I'll note in particular regarding your orthogonality comments:

Why not use (the Carbon equivalent of) uint8_t for those places where C / C++ code was using "char"?

This is really because we would like type safety. Consider code such as:

var indent: Cpp.string;
indent.append(79, ' ');

Here, ' ' is being turned into a char. If char is its own type, this seems fine because ' ' is implicitly converting from a character literal to a character. If char is u8, then we either are allowing character literals to implicitly convert to 8-bit integers (meaning var i: u8 = '0' is also valid), or require an explicit ' ' as u8 there. Using a distinct type is helpful because we can define different operations for it, as Josh was noting on #5922.

(by the way, note this is done without making char a primitive type -- it's currently class Char { adapt u8; }, a user-defined type)

Similarly for string literals, when it comes to something like Cpp.printf("foo") we might rather have the Carbon string literal allow implicit conversion to a char* but not a u8*.

Also, this follows on to your comments about casting:

let ch : char = (1 + '😤' as u32) as char;

The conversion to u32 is actually part of the problem here, you're changing the space to valid u32. Suppose Carbon let you write something like:

let ch: Core.ValidatedChar32 = '😤' + 1;

We can compute at compile time that this'll just be U+1F625, which is a valid UTF-32 character.

I think leads still want to think through how things like std::char8_t, std::char16_t, and std::char32_t should work, and be represented in Carbon, and how that should interoperate. However, I think there's actually space here for type-specific operations to offer better behaviors, which are validated.

[geoffromer]

It makes complete sense to me that for interop purposes we'll need a type that's more or less a drop-in replacement for C++ char, meaning that it's character-oriented but equivocates about whether it's a code point, a code unit, or not unicode at all. And it makes sense that we'll try to file off some of the rough edges where we can, like putting some limits on the arithmetic, and picking a definite signedness.

The main thing I don't understand is the decision to name this type Core.Char, and make char an alias for it: if we named it Cpp.char instead, with no alias, that would immediately set the right expectations about this type (1 byte, supports arithmetic, probably text-ish, most likely UTF-8, etc.), and it would avoid constraining or prejudging the eventual design of the native Carbon character and string APIs.

[jonmeow]

The main thing I don't understand is the decision to name this type Core.Char, and make char an alias for it: if we named it Cpp.char instead, with no alias, that would immediately set the right expectations about this type (1 byte, supports arithmetic, probably text-ish, most likely UTF-8, etc.), and it would avoid constraining or prejudging the eventual design of the native Carbon character and string APIs.

The best I can offer is that on #5969 leads are discussing names of strings, and the decision stated during the summit was to have both str and str_buf be names of C++-equivalent types. I think char is consistent with that, since char is a character in str and str_buf.

[burakemir]

OK, I don't want to be difficult : ) "Interop" is very broad, but I see the argument of migratability, being close to C++.

A predictable, consistent design choice would IMHO be one of these two (nothing in the middle):

• either Carbon's character and string handling is unapologetically aligned with C++. That means byte strings. Then everything related to unicode is "extra". Things like permitting unicode string literals and unicode characters and adding compile time checks are then intended for convenience, but they are special cases and the default is byte strings.

• or (=exclusive-or) Carbon's character and string handling is unapologetically Unicode-first and affords explicit conversions where needed for interop.

The path that seems to be decided is the former, which is not just defensible but also a fine choice. It may IMHO do not leave much room for a unicode-capable 32-bit wide Core.char and Unicode strings by default that would ensure validity of UTF-8 encoding. Rust is of course the latter, with byte strings being repesented by [u8].

One always needs both, Unicode strings and byte strings, anyway (the python experience reported here https://gist.github.com/seanh/0a56cd528714496625662dd9136d0cd3#python-2-vs-python-3 hints at some consequences of having two different types), but only one thing can be default.

IMHO the golang choice is to be absolutely avoided ("conventionally utf-8 but we don't check")

Apologies if this is distracting and I was missing context. I did see proposals that I interpreted to be more leaning on the Unicode-first side, maybe they are outdated.