toLocale{Upper,Lower}Case does not take into Greek tailoring in CLDR

[jungshik]

Unicode 11 3.13 delegates locale-specific tailoring of case mappings to the CLDR (except for Turkic languages). However, Ecma 402 spec does not take into account that only mentioning UnicodeData.txt and SpecialCasings.txt.

As a result, Greek uppercasing (context-dependent removal of diacritic marks) as desired by Greek speakers is not supported.

ICU-based implementations such as v8 does support it. Spun off from https://crbug.com/v8/8353

@ftang @littledan @gsathya

[jungshik]

Note also that ICU went a great length to implement this behavior to meet Greek speaker's requirement for CSS text-transform: uppercase. I don't see why EcmaScript toLocaleUppercase should be different.

[littledan]

I think we are just referencing the wrong thing, and we should include whatever language would be appropriate to enable this better behavior. PRs welcome to improve the wording--probably we should just reference the relevant part of the Unicode spec rather than specify the data files.

[anba]

I only found the following two bugs mentioning toLocale[Lower,Upper]Case in the bugs.ecmascript.org archive:

1. https://tc39.github.io/archives/bugzilla/206/
2. https://tc39.github.io/archives/bugzilla/573/

Both issues only refer to the Unicode database for case conversions.

Relevant meeting notes for 573 -> https://mail.mozilla.org/pipermail/es-discuss/2012-October/025837.html

The meeting notes refer to a strawman from Norbert, but the archived page doesn't provide any insights. From https://web.archive.org/web/*/http://wiki.ecmascript.org/doku.php?id=strawman:case_conversion

Locale Argument in Case Conversion

This proposal has been accepted for the second edition of the ECMAScript Internationalization API Specification at the November 2012 TC 39 meeting, and has been integrated into section 13.1 of the first draft of that specification.

(Nov 2012 meetings notes -> https://github.com/tc39/tc39-notes/blob/master/es6/2012-11/nov-27.md#string-case-conversion)

FWIW Java's to[Upper,Lower]Case methods also only support the case conversions from the Unicode database (tr,az,lt) and not the ones in CLDR (tr,az,lt, and el). I can't tell if that was a deliberate decision in Java, though.

[littledan]

Regardless of the history, I think we should use the more linguistically correct algorithm here.

[anba]

I don't want to oppose switching to CLDR for case conversions, sorry if my comment gave that impression. For me it's just important to document and validate that any changes to existing parts of the spec don't lead to unintentional violation of prior consensus.

This doesn't seem to be the case here, because past notes don't even mention CLDR. Probably that's because rules for Greek casing were added in CLDR 23 (released March 2013), so after the TC39 meetings mentioned above. 😄

Related CLDR issues

• http://unicode.org/cldr/trac/ticket/2645
• http://unicode.org/cldr/trac/ticket/3449
• http://unicode.org/cldr/trac/ticket/4779

The second CLDR ticket (and https://tc39.github.io/archives/bugzilla/573/) mention different upper case rules for fr_CA when compared to fr_FR for accented characters. I don't see this implemented in CLDR. Is this still an open issue and may appear in future CLDR versions?

[sffc]

What is the status of this issue?

[littledan]

@sffc IIRC we've discussed this at a couple ECMA-402 meetings, and we all agree that we should permit implementations to tailor the data of all algorithms. (This should also be relevant to currency digits.) The next step would be to write specification text to formalize this.

[sffc]

I see the open PR on this issue (#291). Are you planning to revise the PR, or are you looking for someone else to take over this issue?