Normative: Add calendar and numberingSystem options

[littledan]

This patch allows calendar and numberingSystem to be specified in
the options bag of the DateTimeFormat and NumberFormat constructors.
One use case for these options would be, when working with locales
which have two numbering systems and calendars in use--the UA default
may be the non-Western one, but in some contexts, it may be appropriate
to use the Western one. Currently, without the patch, it would be
necessary to parse the BCP 47 language tag in the application, but Intl
provides no library to do so. For example, this all occurs in this bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=1370086

Related bug: #105 . This patch leaves out "collation" because of a lack
of clear use cases.

[littledan]

Test262 tests: tc39/test262#1225

V8 implementation: https://chromium-review.googlesource.com/c/v8/v8/+/664762

[caridy]

Seems to be very straight forward to add these new options.

[littledan]

Any other opinions here? Can this be merged?

[littledan]

In particular, do you think we should go to TC39 with this change, or is it minor enough to merge it without that presentation?

[caridy]

I think we can do this without any presentation, but let's loop in few more folks to get their feedback:
@zbraniecki @jungshik

[zbraniecki]

I agree with @caridy

[caridy]

@littledan do you want to merge this?

[anba]

Typo: numeringSystem (and in the line below and in the equivalent lines for Intl.NumberFormat.)

[caridy]

@littledan can you update the PR to fix these typos?

[littledan]

Fixed the typo, apologies for the delay.

[littledan]

This PR reached consensus at the November 2017 TC39 meeting, see notes.

[anba]

Nit: Instead of

1. Let calendar be ? Get(options, "calendar").
2. If calendar is not undefined, set calendar to ? ToString(calendar).
3. Set opt.[[ca]] to calendar.

The GetOption operation should be used for consistency with access to other options.

1. Let calendar be ? GetOption(options, "calendar", "string", undefined, undefined).
2. Set opt.[[ca]] to calendar.

(And we should change Get(options, "timeZone") in InitializeDateTimeFormat to use GetOption, too.)

But probably more important than how to access and convert the new options: Currently all options-bag options seem to be validated in some way and invalid option values are rejected with an error, whereas invalid options from Unicode extensions are simply ignored (e.g. Intl.Collator("de", {caseFirst: "invalid"}) throws a RangeError, whereas Intl.Collator("de-u-kf-invalid") is silently accepted). This change introduces the first options-bag options which don't get any validation at all, which seems a bit unfortunate. And furthermore we should also formally define the allowed values for the "numberingSystem" and "calendar" options (BCP 47 vs. UTS 35 names).

For example let's consider the "calendar" option:
https://tc39.github.io/ecma402/#sec-properties-of-intl-datetimeformat-instances describes [[Calendar]] as:

[[Calendar]] is a String value with the "type" given in Unicode Technical Standard 35 for the calendar used for formatting.

But which "type" is meant here? The value of the "name" attribute of the <type> element for BCP 47 (https://unicode.org/repos/cldr/tags/latest/common/bcp47/calendar.xml) or the value of "type" attribute of the <calendar> element (https://unicode.org/repos/cldr/tags/latest/common/main/root.xml)? And is it then Intl.DateTimeFormat("de", {calendar: "gregory"}) or Intl.DateTimeFormat("de", {calendar: "gregorian"}) or maybe even both? Also what about canonicalizing the input, for example if the input is "GrEgOrIaN", does it get accepted and if it is accepted, does it get canonicalized to "gregorian"? And if we only accept known calendars per UTS 35, we probably also need to explicitly disallow the name "generic", don't we?

For example Intl.DateTimeFormat("de-u-ca-gregory").resolvedOptions().calendar prints "gregory" for me with V8 and SpiderMonkey, but "gregorian" with JSC (on Ubuntu16.04, IIRC JSC uses the system ICU which is ICU55 for Ubuntu16.04; V8 and SpiderMonkey both use ICU60). Right now I'd say only JSC is correct here and "gregorian" is the expected output. But WDYT?

[srl295]

sounds good to me

[littledan]

Unfortunately I don't think ICU has an API for validating these sorts of things. I was thinking of the validation logic a bit like, things which were based on an open-ended name would be checked through resolvedOptions.

Do you think it will be important for users to get a RangeError here? Does anyone remember the original intention with the current design for error checking? cc @NorbertLindenberg

We reviewed this patch at the ECMA 402 meeting. At that time, I'd only skimmed @anba's comment and didn't realize the validation issue. I guess this patch isn't ready to merge until we make a decision on that question.

[zbraniecki]

As discussed on the ECMA402 call - we should only validate the options to be well-formed, just like we do for extension key values.

[littledan]

OK, the newly uploaded patch is based on validating the grammar, but not contents, of the calendar and numberFormat options. It does not address @anba's other canonicalization question, however. How does this seem?

[caridy]

hmm, at first glance, "match type" feels weird to me considering that you will have to open the link, and try to get the glimpse of what type is. This seems to be simple enough that maybe we can specify it here.

[littledan]

I've rebased and updated the PR to follow @anba's suggestions above. The lowercase conversion is a really necessary bug fix IMO. Probably we need to add some additional test262 tests to verify.

[bkardell]

Ok, I am having some trouble understanding... Looking at the spec, the test and using the implementations I have a question. Here's a matrix of various inputs and qualities..

constructor args	parts has year	parts has yearName	parts has relatedYear
"en"	yes	no	no
"en", {year: "numeric"}	yes	no	no
"en-u-ca-chinese"	no	no	yes
"en-u-ca-chinese", {year: "numeric"}	no	yes	yes

as a developer I find it hard to understand the logic behind these results.. why wouldn't you have a .year in some cases? It seems very strange to pass year numeric in the last case and have a relatedYear, but not a .year? I'm sure there's probably very good reasons - I'm just wondering if someone can help explain it to me so that I can document it?

[FrankYFTang]

Ok, I am having some trouble understanding... Looking at the spec, the test and using the implementations I have a question. Here's a matrix of various inputs and qualities..

constructor args parts has year parts has yearName parts has relatedYear
"en" yes no no
"en", {year: "numeric"} yes no no
"en-u-ca-chinese" no no yes
"en-u-ca-chinese", {year: "numeric"} no yes yes
as a developer I find it hard to understand the logic behind these results.. why wouldn't you have a .year in some cases? It seems very strange to pass year numeric in the last case and have a relatedYear, but not a .year? I'm sure there's probably very good reasons - I'm just wondering if someone can help explain it to me so that I can document it?

The problem for the last part is the caller asked for a "numeric" for year, but in en-u-ca-chinese calendar, the year is not a numeric value (not 1 , 2, 3, 4, in any numbering system), but a string name
"jia-zi",
"yi-chou",
"bing-yin",
"ding-mao",
"wu-chen",
"ji-si",
"geng-wu",
"xin-wei",
"ren-shen",
"gui-you",
"jia-xu",
"yi-hai",
"bing-zi",
"ding-chou",
"wu-yin",
"ji-mao",
....
a list of 60 names of year.

[bkardell]

@FrankYFTang thanks... sorry, I still don't entirely understand. If there is a better venue to ask you these kinds of questions, let me know.. In the meantime, here are some more clarifying quesitons - sorry if they seem basic or something.. Does this mean that:

1. "en-u-ca-chinese" will never have an entry called year - no matter what you put in year option?

2. "en-u-ca-chinese" will always have an entry called yearName``, which is the year, but a string-oriented name and therefore doesn't just belong in year`?

3. "en-u-ca-chinese" will always have a property called relatedYear which will contain.... what is normally in the year except is never affected by what you put in the opt for year?

[FrankYFTang]

@FrankYFTang thanks... sorry, I still don't entirely understand. If there is a better venue to ask you these kinds of questions, let me know.. In the meantime, here are some more clarifying quesitons - sorry if they seem basic or something.. Does this mean that:

1. "en-u-ca-chinese" will never have an entry called year - no matter what you put in year option?

As the current spec, YES

2. "en-u-ca-chinese" will always have an entry called yearName, which is the year, but a string-oriented name and therefore doesn't just belong in ``year`?

YES, because the form is neither "numeric" nor "2-digit"

3. "en-u-ca-chinese" will always have a property called relatedYear which will contain.... what is normally in the year except is never affected by what you put in the opt for year?

First, it may not ALWAYS output 'relatedYear'. It MAY output it, depending on the values in the options.
Second, a better way to say it is , if the formatter output 'relatedYear', the 'relatedYear' will contain what is the value in the year OF THE Gregorian calendar, not the Chinese calendar. That is why it is called "related"- a year value NOT in that request calendar, but in a different calendar system - the Gregorian calendar. The reason in the modern day, even when user want to see the date output in Chinese calendar (or some other non Gregorian calendar), they ALSO want to see the year of Gregorian calendar next to it. That is why it is also output in the POSITION where the year of that calendar is output.

[bkardell]

Thanks @FrankYFTang that's very helpful. I still find it confusing as someone not yet deeply into this problem, but at least I kind of understand how it winds up here now!

[littledan]

@bkardell and I had talked about this before filing the issue, and the case that I found surprising was where we actually had a relatedYear but no yearName, which you can see in Brian's data above, e.g.,

var df = new Intl.DateTimeFormat("en-u-ca-chinese")
var parts = df.formatToParts(new Date())
console.log(JSON.stringify(parts))

[{"type":"month","value":"12"},{"type":"literal","value":"/"},{"type":"day","value":"21"},{"type":"literal","value":"/"},{"type":"relatedYear","value":"2019"}]

This works the same in V8 and SpiderMonkey.

@FrankYFTang wrote,

"en-u-ca-chinese" will never have an entry called year - no matter what you put in year option?

As the current spec, YES

This is the part where I didn't quite understand the current spec. I guess this is because in ToLocalTime, the calendar calculation is independent of the format being output? So, even though we're just using a month/day/year format and there's no yearName, we still use relatedYear. Is that right? The motivation here is that 2019 is not the year in the Chinese calendar? If that's the case, then I'm a bit confused about why this display format exists in the locale database...

[FrankYFTang]

@bkardell and I had talked about this before filing the issue, and the case that I found surprising was where we actually had a relatedYear but no yearName, which you can see in Brian's data above, e.g.,

var df = new Intl.DateTimeFormat("en-u-ca-chinese")
var parts = df.formatToParts(new Date())
console.log(JSON.stringify(parts))

[{"type":"month","value":"12"},{"type":"literal","value":"/"},{"type":"day","value":"21"},{"type":"literal","value":"/"},{"type":"relatedYear","value":"2019"}]

This works the same in V8 and SpiderMonkey.

@FrankYFTang wrote,

"en-u-ca-chinese" will never have an entry called year - no matter what you put in year option?

As the current spec, YES

This is the part where I didn't quite understand the current spec. I guess this is because in ToLocalTime, the calendar calculation is independent of the format being output? So, even though we're just using a month/day/year format and there's no yearName, we still use relatedYear. Is that right? The motivation here is that 2019 is not the year in the Chinese calendar? If that's the case, then I'm a bit confused about why this display format exists in the locale database...

The skeleton it asked inside ICU is "yMd" for this case, and the pattern is
yMd{"M/d/r"}
notice, the 'r' which is the related year
see
https://cs.chromium.org/chromium/src/third_party/icu/source/data/locales/en.txt?q=locales/en.txt&sq=package:chromium&dr&l=269

Feel free to file a CLDR ticket ask CLDR linguist to clarify the rationale.
In Taiwan, for example, that is now how they print on the news paper. For example
https://chenminjen07.files.wordpress.com/2015/04/b329a-e59c96e789871.jpg
The top line (above the big bold title of the headline, left next to vertical red banner of the 3 characters title of the newspaper) is date in both 'roc' calendar and 'chinese' calendar

[littledan]

Thanks for explaining, @FrankYFTang

[leobalter]

My understanding is that we have tests for this. cc @caiolima, is this correct?

[anba]

This PR landed after Intl.Locale, but didn't fix the issue outlined in tc39/proposal-intl-locale#96. Maybe revert the PR for now?