[RegExp] case-insensitive matching misses characters

[goyakin]

When we generate the character casing equivalence sets for RegExp using UnicodeData.txt, we start from lowercase letters and then add the characters that they map to. However, this misses non-letter characters. For example, '\u0345', which is in the "Mn" (Mark, Nonspacing) category, should match '\u0399' when the case-insensitive flag is passed (even without the "unicode" flag), but we fail to do so.

When this is fixed, the following test should pass:

var didMatch = /\u0345/i.test('\u0399');
WScript.Echo(didMatch);

[bterlson]

We are also not properly handling canonicalization with i and u flags. /\W/iu.test("S") and /\W/iu.test("K") should return true.

[mathiasbynens]

Since that last comment tends to confuse people, here’s an explanation of why /\W/iu.test('S') and /\W/iu.test('K') should return true: https://mathiasbynens.be/notes/es6-unicode-regex#impact-i

[goyakin]

Possible spec change regarding \W: tc39/ecma262#512

[dilijev]

@mathiasbynens wrote in #1136:

Chakra returns false for each of these tests, while V8 and JavaScriptCore return true for all of them:

/\u01F1/i.test('\u01F3')
/\u0345/i.test('\u03B9')
/\u037F/i.test('\u03F3')
/\u0528/i.test('\u0529')
/\u052A/i.test('\u052B')
/\u052C/i.test('\u052D')
/\u052E/i.test('\u052F')
/\uA698/i.test('\uA699')
/\uA69A/i.test('\uA69B')
/\uA796/i.test('\uA797')
/\uA798/i.test('\uA799')
/\uA79A/i.test('\uA79B')
/\uA79C/i.test('\uA79D')
/\uA79E/i.test('\uA79F')
/\uA7AB/i.test('\u025C')
/\uA7AC/i.test('\u0261')
/\uA7AD/i.test('\u026C')
/\uA7B0/i.test('\u029E')
/\uA7B1/i.test('\u0287')

https://mathiasbynens.be/demo/regex-i

(See also: similar SpiderMonkey bug.)

[mathiasbynens]

/\W/iu.test("S") and /\W/iu.test("K") should return true.

This is no longer true now that tc39/ecma262#525 has landed.

The other bugs mentioned in this thread remain.