Some characters are mistaken for invalid identifiers

[wcshds]

version: v2024.3.1

Example

𣎴="hello world"
print(𣎴)

[debonte]

@erictraut, the 𣎴 character is supported as an indentifier by Python (running this code prints out "hello world"). The problem is that the surrogate character 55372 is missing from pyright-internal\src\parser\unicode.ts. I added the following to unicodeLuSurrogate and it fixed this issue, but a) I'm not sure that's the correct surrogate table for it to be in and b) I'm assuming there are more characters that are missing.

    55372: [57268],

unicode.ts says it was built from the npm unicode package, which says that it provides data from https://unicode.org/Public/UNIDATA/UnicodeData.txt. That file doesn't contain this surrogate.

Seems like we need to update our tables using a new source of Unicode surrogate info?

[erictraut]

I'm not sure what to say here. PEP 3131 is pretty clear about which character ranges are allowed for identifier starts and continuations. In particular, it says:

ID_Start is defined as all characters having one of the general categories uppercase letters (Lu), lowercase letters (Ll), titlecase letters (Lt), modifier letters (Lm), other letters (Lo), letter numbers (Nl), the underscore, and characters carrying the Other_ID_Start property. XID_Start then closes this set under normalization, by removing all characters whose NFKC normalization is not of the form ID_Start ID_Continue* anymore.

The character 0xD84C is not in any of these general categories. It's part of the "Cs" category, which doesn't appear in the list above.

This is reinforced by the information in this document. Search for "Characters that can start an identifier". Note that this document comes directly from the unicode organization.

I'm guessing that this is explained by one of three things:

1. This is a bug in the CPython implementation where it's not following the spec.
2. The spec has changed since PEP 3131 was written, and the CPython implementation has been amended accordingly.
3. I'm missing something.

More investigation is likely needed to determine which of the above is the explanation.

If we want to use the CPython code as a reference, here is where you can look.

[wcshds]

@erictraut @debonte I think 0xD84C is used in UTF16. I'm guessing Pylance's Javascript code didn't handle the surrogate pair well and treated 𣎴 (0x233B4) as if it is seperate characters

According to https://unicode.org/Public/UNIDATA/UnicodeData.txt, CJK ideographs can indeed be used as a valid identifier.

[debonte]

I'm working on a fix for this. CPython's character info is currently based on Unicode 15.1. I'm not sure what version ours is based on, but we're out of date.

[debonte]

This will be fixed in next week's Pylance release. It requires a Pyright pull.

[StellaHuang95]

This issue has been fixed in prerelease version 2024.3.102, which we've just released. You can find the changelog here: CHANGELOG.md