Internationalization fails on IE 10 on Windows 8 (Simplified Chinese)

[zhanzhenzhen]

Persona.org can detect my browser language and show me the Chinese version on Windows 7, but cannot on Windows 8. IE 10 on Windows 8 always shows the English version of persona.org. Maybe the reason is the "Accept-Language" header field:
zh-Hans-CN,zh-Hans;q=0.8,en-US;q=0.5,en;q=0.3
There is a strange item "zh-Hans-CN". I searched the Web and found it's a newly registered language tag.

[zhanzhenzhen]

Hi, I found it currently can only recognize "zh-CN". So maybe both "zh-Hans-CN" and "zh-Hans" cannot be recognized? These 3 are almost the same. They all mean Simplified Chinese.

[zhanzhenzhen]

Some additional information that may be helpful:
IE 10 Release Preview for Windows 7 does not have this issue. Its "Accept-Language" is "zh-CN".
But IE 10 on Windows 8 is different. It uses the OS built-in "language preferences" settings to generate this field.

[callahad]

Thanks for the detailed report! We're on it.

[shane-tomlinson]

@zhanzhenzhen - I was digging a bit more into this and found the problem. We have zh_CN and zh_TW, but not zh_Hans_CN. Is aliasing zh_Hans_CN to zh_CN valid? I am wondering if a valid approach for an extended locale such as this is to:

1. Check exact match. (zh_Hans_CN)
2. If no exact match found, check for two _'s. If yes, go to 3, if not, go to 4
3. remove the center portion of the locale and check country dialect (zh_CN)
4. Split on _, check language (zh)

[shane-tomlinson]

@mathjazz - Could you weigh in here with your expertise?

[callahad]

RFC 4646, section 4.3.2 says:

applications or protocols that truncate tags MUST do so by progressively removing subtags along with their preceding "-" from the right side of the language tag

So it should be zh-Hans-CN → zh-Hans → zh.

We should not exise the middle (zh-Hans-CN → zh-CN) because zh-Hant also exists, for Traditional Chinese.

[callahad]

The W3 has a great article on BCP47: "Understanding the New Language Tags."

Pragmatically, Verbatim has two translations for this project, one for zh-CN and one for zh-TW.

On our end, let's alias:

• zh-Hans-CN → zh-CN
• zh-Hant-TW → zh-TW

I wish we could go the other way (I think zh-CN → zh-Hans-CN is more "correct"), but that's a different issue.

[zhanzhenzhen]

@shane-tomlinson - 3 tags is not the most. There can also be 4 tags. See RFC 4646 Appendix B.
I'm not familiar with this project, but I think there are 2 things we need to add or modify:

1. We should be able to bind more than 1 language tags to 1 translation. So in the code, "zh-Hans" and "zh-CN" should be bound to the simplified chinese translation, and "zh-Hant" and "zh-TW" should be bound to the traditional chinese translation. This is the only method because "zh-Hans-CN" is not derived from "zh-CN". (Maybe you also want to set "zh" to either simplfied or traditional.)
2. We should do as callahad said, removing subtags along with their preceding "-" from the right side of the language tag. But I think this kind of functions may already exist in Mozilla's code library?

[zhanzhenzhen]

@callahad - I think alias "zh-Hans" is better than "zh-Hans-CN".

[zhanzhenzhen]

By the way, don't forget that there are also "zh-HK" (Hong Kong) and "zh-SG" (Singapore). "zh-HK" is traditional and "zh-SG" is simplified.

[callahad]

@zhanzhenzhen Great! Thank you. I'd love to alias to the higher zh-Han[ts] level if possible. Just to check,

1. Are you sure that serving zh-Hans-CN to someone that requests zh-Hans or zh-Hans-* is good?
2. Are you sure that serving zh-Hant-TW to someone that requests zh-Hant or zh-Hant-* is good?

(Or, put another way, are the translations we have for zh-CN and zh-TW sufficiently region-agnostic to stand in as zh-Hans and zh-Hant on their own?)

We can definitely make that happen, but I want to double check before making a mistake with locales. :)

[callahad]

Looks like zh-MO should also be aliased to traditional chinese. At least, that's how Dojo and Microsoft do it...

[zhanzhenzhen]

Maybe you wonder why there are "zh-CN" and "zh-Hans-CN". As far as I know, "zh" without the script sub-tag is an old form. Newer systems may use the newly registered "zh-Hans" and "zh-Hant" that looks more accurate.