Built-in character operators

[jonmeow]

Summary of issue:

Following up on #5903, another question about characters: what operators should Char support, and what should their return values be?

Details:

For background, in #1964, it says (summarizing):

• Comparison: <, >, <=, >= ==
• Arithmetic: +, -
• Explicit conversions to integers, no implicit
  • No statement on conversions from integers.

It gives an example using these in rationale:

var b: u8;
b = 'a' + 1;
b + 1 == 'c';

This example implies the two operator behaviors:

• 'a' + 1 returns u8 (because there's no implicit conversion from an integer)
• Comparison between u8 and Char works.

However, other parts of the proposal seem to suggest that neither of those operators should work that way.

[jonmeow]

I'll suggest the following operator support for Char:

• Comparison and equality where both arguments are Char
  • No mixing integers.
• Arithmetic:
  • Addition:
    • Char + integer (signed or unsigned) -> Char
    • integer (signed or unsigned) + Char -> Char
    • Char + Char is unsupported (noted in Character Literals (#1934) #1964)
  • Subtraction:
    • Char - integer (signed or unsigned) -> Char
    • integer (signed or unsigned) - Char is unsupported (noted in Character Literals (#1934) #1964)
    • Char - Char -> i32
      • Behaves as return (lhs as u8) - (rhs as u8);, except for overflow; for return type, see below character literal notes
  • Uses signed integer overflow semantics. (('a' as Char) + 500 is a programming error)
  • edit to add: in general, operations with integer operands behave as return (((character as u8) <op> integer) as u8) as Char.
• Explicit conversion to/from integers equivalent to what u8 supports.
  • Continuing no implicit conversions.

Character literals (CharLiteral):

• Comparison and equality either both or mixed with Char
• Arithmetic:
  • Mostly as with Char, with matching return types (i.e., addition with CharLiteral returns CharLiteral, not Char)
  • Char - CharLiteral behaves as c - (literal as Char); can be reversed
  • CharLiteral - CharLiteral is a unicode code point delta.
    • Because it's a unicode code point delta, i32 seems like appropriate storage.
      • Edit to add: working in reverse, this is why I suggest "Char - Char" returns i32, to have a consistent return type.
    • Note this works similarly to (lhs as Char) - (rhs as Char) for values that fit inside Char, but also should yield a reasonable result for larger values.
• Explicit conversion to/from integers equivalent to what i32 supports, plus u32.
  • Edit to add: I was thinking to 32-bit and larger integer sizes; explicit casts to types that i32 implicitly casts to. 'a' as i16 I suggest is invalid; ('a' as Char) as i16 gives an unambiguous result. Put differently, CharLiteral behaves similarly to how a UTF-32 character type might.

[burakemir]

I would suggest to not support arithmetic on types that represent characters (e.g. Unicode characters), at all.

A number may not correspond to a legal character (e.g. code points above U+10FFFF, surrogate code points (U+D800 to U+DFFF))

Instead, anyone who needs such operations should cast to scalar value, do their arithmetic there, and convert back. The conversion should then be fallible.

I would further suggest to not have a specific CharacterLiteral type. I could not find such a type being mentioned in #1964 and the monospace and wording suggest that there would be such a type. Instead, should a character literals not simply map to an instance of a suitable character type, which is the same as e.g. extracted from a string or read from input?

[josh11b]

@burakemir A significant amount of C & C++ code operates on characters as integers, and doing so can be important to efficiency. I think the proposed resolution sensibly restricts to those operations that are common, meaningful, and useful on character values. The set of sensible operations on characters is different than on integer scalar types, so requiring to convert is not just verbose, but gives up an opportunity to catch problematic constructions.

The CharacterLiteral type is aligned with our resolution of how to handle integer literals. It will be a type with only compile-time values, and can at compile-time validate that any conversion to a specific width runtime type is in range.

I don't think this specific proposal has any issues about illegal Unicode characters, since it only concerns:

• Char which is an 8-bit type that will not reach the illegal character range, and
• CharacterLiteral which is a compile-time type and can reject invalid values during compilation.

[burakemir]

I left some comments regarding "character type" and orthogonality on #5903.

[jonmeow]

@burakemir FWIW, I think we want to support code like var c: Char = 'a';, without requiring an explicit conversion ('a' as Char). That's difficult if 'a' is a UTF-32 Char32, which I would expect to require an explicit conversions to Char (because some conversions are invalid). It's easy if 'a' is its own CharLiteral type, which can have implicit conversions due to the different semantics of its use.

Sidenote, minor edit to the above: for subtraction, c - (literal as u8) -> c - (literal as Char). u8 vs Char is significant here, but this was really just an oversight as I was copying examples.

[burakemir]

My understanding here evolved in that it seems to me we want Carbon to do byte strings first / default, aligned with C++, with unicode being extra. Coming from Rust, I found this surprising, but as I said in #5903, I consider this a fine choice if done consistently.

[josh11b]

@burakemir FWIW, I think we want to support code like var c: Char = 'a';, without requiring an explicit conversion ('a' as Char). That's difficult if 'a' is a UTF-32 Char32, which I would expect to require an explicit conversions to Char (because some conversions are invalid). It's easy if 'a' is its own CharLiteral type, which can have implicit conversions due to the different semantics of its use.

Plan is for the eventual forms and compile-time designs to support "implicit conversion of this value known at compile time only if it is in a specific range that is representable by the target type."

As a side note: I support the proposed resolution, and resolving this issue is a requirement for improvements to sample code that appears in the newsletter and example code (1, 2).

[chandlerc]

FWIW, I really like the balance in @jonmeow's proposed direction.

I think it gives us excellent utility for fundamentally code-unit operations such as the distance between ASCII characters, and other common patterns. It also does a good job of provided simple and consistent types and return types for operations. And lastly, it avoids the operations that are genuinely problematic.

I do understand some hesitancy around having this at all, but I think both for efficiency reasons and given the existing idioms of C++, this integer-ish view of Char as the UTF-8 code unit is important to include.

[zygoloid]

Leads decision: we will follow the design presented in @jonmeow's comment on this issue.