StringInfo.ParseCombiningCharacters returns incorrect values on Linux

[leppie]

var c = StringInfo.ParseCombiningCharacters("लो");
Console.WriteLine(c.Length);

The following prints 2 on Windows, but 1 on Linux (Ubuntu 19.04 x64). (Mono also prints 1)

[tarekgh]

@GrabYourPitchforks would this be fixed by your coming changes?

by the way, would be a good idea to expose some overload avoid the int[] allocation.

[GrabYourPitchforks]

Yes, this would be addressed by #328. See in particular UAX#29, Sec. 3.1.1, Rule GB9a, which handles this case.

I agree that it would be good to expose a new allocation-free API for this. The PR I have open introduces such an API, but it's internal-only for now because we haven't gone through an API review process. Perhaps https://github.com/dotnet/corefx/issues/35370 would be a good candidate for new public APIs?

(For reference, the correct answer should be 1.)

[tarekgh]

Thanks @GrabYourPitchforks I assigned this issue to you so you can close it when you merge your changes.

[leppie]

(For reference, the correct answer should be 1.)

Why is the answer 1? "लो" is not a single code point as StringInfo is meant to iterate over. I used the static method just to show the inconsistency.

If it indeed was a single codepoint, I would expect it to differ from the individual chars, but it only returns the first char ((int)2354) and is thus not idempotent as one would expect the conversion between UTF16 <-> UTF32.

[GrabYourPitchforks]

StringInfo isn't meant to iterate over code points; it's meant to iterate over graphemes. In the 2-char string you provided in your sample, the second char is U+094B DEVANAGARI VOWEL SIGN O, which combines as a diacritic with the preceding consonant. So the two chars together are intended to be a single grapheme. See also the "Grapheme_Cluster_Break" property of https://unicode.org/cldr/utility/character.jsp?a=094b&B1=Show for more information.

[leppie]

Thanks. Not sure where I read it as code points, but clearly I read wrong. :D

Still an inconsistency.

[GrabYourPitchforks]

Fixed by #328.