[Tracking] UnicodeScalar: Does any platform's case conversion routine allow for full folding?

[GrabYourPitchforks]

This is a tracking issue for the comment at dotnet/coreclr#20935 (comment).

Throughout string, ROS<char>, and UnicodeScalar, we assume that the underlying globalization libraries only allow for simple case folding, not full case folding. In a simple folding system, the number of scalars will never change when performing a case conversion for any string value. Furthermore, case folding will never cause values to jump between the BMP and any supplementary plane. The end result of all of this is that UTF-16 strings will not change length when changing case.

Tarek's comment is that this might not be a safe assumption - that the underlying libraries might actually use full folding under the covers. We need to verify this one way or another so that we can update our internal logic accordingly.

[GrabYourPitchforks]

Spoke with @tarekgh regarding this. The current behavior is that all built-in case conversion routines which return string-like data back to the developer use simple folding mechanisms, which will never alter the length of the UTF-16 data.

The framework might still use full case folding semantics internally for things like comparisons (e.g., String.Equals("ß", "SS", StringComparison.InvariantCultureIgnoreCase)), but that doesn't actually return any string-like data back to the caller, so the caller never observes that the UTF-16 data might be changing length.

If we do want to expose a mechanism to allow full folding semantics and return string-like data back to the caller, it should probably be a different API (or at least a different overload) than today's ToUpper / ToLower.