Allow non-ASCII identifiers #2457
Typos, renames and a minor reference change
unicode_idents -> non_ascii_idents Remove mention of exact spec revision Describe more how to implement confusable detection and remove mention of scope fix typo
- Loading branch information
commit 6b2a94a58ef93bb80b7f9b859a2de6a39ec86431
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| - Feature Name: non_ascii_idents | ||
| - Start Date: 2018-06-03 | ||
| - RFC PR: (leave this empty) | ||
| - Rust Issue: (leave this empty) | ||
|
|
@@ -36,17 +36,17 @@ Examples of invalid identifiers are: | |
|
|
||
| * Keywords: `impl`, `fn`, `_` (underscore), ... | ||
| * Identifiers starting with numbers or containing "non letters": `42_the_answer`, `third√of7`, `◆◆◆`, ... | ||
| * Many Emojis: 🙂, 🦀, 💩, ... | ||
|
|
||
| Similar Unicode identifiers are normalized: `a1` and `a₁` (a<subscript 1>) refer to the same variable. This also applies to accented characters which can be represented in different ways. | ||
|
|
||
| To disallow any Unicode identifiers in a project (for example to ease collaboration or for security reasons) limiting the accepted identifiers to ASCII add this lint to the `lib.rs` or `main.rs` file of your project: | ||
|
|
||
| ```rust | ||
| #![forbid(non_ascii_idents)] | ||
| ``` | ||
|
|
||
| Some Unicode character look confusingly similar to each other or even identical like the Latin **A** and the Cyrillic **А**. The compiler may warn you about easy to confuse names in the same scope. If needed (but not recommended) this warning can be silenced with a `#[allow(confusable_non_ascii_idents)]` annotation on the enclosing function or module. | ||
|
|
||
| ## Usage notes | ||
|
|
||
|
|
@@ -59,7 +59,9 @@ Private projects can use any script and language the developer(s) desire. It is | |
| # Reference-level explanation | ||
| [reference-level-explanation]: #reference-level-explanation | ||
|
|
||
| Identifiers in Rust are based on the [Unicode® Standard Annex #31 Unicode Identifier and Pattern Syntax][UAX31]. | ||
|
|
||
| Note: The supported Unicode version should be stated in the documentation. | ||
|
|
||
| The lexer defines identifiers as: | ||
|
|
||
|
|
@@ -75,19 +77,21 @@ The lexer defines identifiers as: | |
|
|
||
| Two identifiers X, Y are considered to be equal if their [NFKC forms][TR15] are equal: NFKC(X) = NFKC(Y). | ||
|
|
||
| A `non_ascii_idents` lint is added to the compiler. This lint is `allow` by default. The lint checks if any identifier in the current context contains a codepoint with a value equal to or greater than 0x80 (outside ASCII range). Not only locally defined identifiers are checked but also those imported from other crates and modules into the current context. | ||
|
|
||
| ## Confusable detection | ||
|
There was a problem hiding this comment. Currently the parser will go down a bad route when encountering an identifier that isn't a keyword. We probably want to forbid unicode idents that could be confused with any of the rust keywords. There was a problem hiding this comment. See fifth paragraph, last sentence:
|
||
|
|
||
| Rust compilers should detect confusingly similar Unicode identifiers and warn the user about it. | ||
|
|
||
| Note: This is *not* a mandatory for all Rust compilers as it requires considerable implementation effort and is not related to the core function of the compiler. It rather is a tool to detect accidental misspellings and intentional homograph attacks. | ||
|
|
||
| A new `confusable_non_ascii_idents` lint is added to the compiler. The default setting is `warn`. | ||
|
|
||
| Note: The confusable detection is set to `warn` instead of `deny` to enable forward compatibility. The list of confusable characters will be extended in the future and programs that were once valid would fail to compile. | ||
|
|
||
| The confusable detection algorithm is based on [Unicode® Technical Standard #39 Unicode Security Mechanisms Section 4 Confusable Detection][TR39Confusable]. For every distinct identifier X execute the function `skeleton(X)`. If there exist two distinct identifiers X and Y in the same crate where `skeleton(X) = skeleton(Y)` report it. | ||
|
|
||
| Note: A fast way to implement this is to compute `skeleton` for each identifier once and place the result in a hashmap as a key. If one tries to insert a key that already exists check if the two identifiers differ from each other. If so report the two confusable identifiers. | ||
|
|
||
| # Drawbacks | ||
| [drawbacks]: #drawbacks | ||
|
|
@@ -121,7 +125,7 @@ It has been suggested that Unicode identifiers should be opt-in instead of opt-o | |
|
|
||
| The current design was chosen because the algorithm and list of similar characters are already provided by the Unicode Consortium. A different algorithm and list of characters could be created. I am not aware of any other programming language implementing confusable detection. The confusable detection was primarily included because homoglyph attacks are a huge concern for some member of the community. | ||
|
|
||
| Instead of offering confusable detection the lint `forbid(non_ascii_idents)` is sufficient to protect project written in English from homoglyph attacks. Projects using different languages are probably either written by students, by a small group or inside a regional company. These projects are not threatened as much as large open source projects by homoglyph attacks but still benefit from the easier debugging of typos. | ||
|
|
||
| # Prior art | ||
| [prior-art]: #prior-art | ||
|
|
@@ -143,13 +147,13 @@ The [Go language][Go] allows identifiers in the form **Letter (Letter | Number)\ | |
| * Are Unicode characters allowed in `no_mangle` and `extern fn`s? | ||
| * How do Unicode names interact with the file system? | ||
| * Are crates with Unicode names allowed and can they be published to crates.io? | ||
| * Are `non_ascii_idents` and `confusable_non_ascii_idents` good names? | ||
|
There was a problem hiding this comment. As @eggrobin has pointed out, there are already pairs of ascii characters that are being considered confusable to each other thus the There was a problem hiding this comment. Thanks. Changed it. |
||
| * Should [ZWNJ and ZWJ be allowed in identifiers][TR31Layout]? | ||
| * Should *rustc* accept files in a different encoding than *UTF-8*? | ||
|
|
||
| [PEP 3131]: https://www.python.org/dev/peps/pep-3131/ | ||
| [UAX31]: http://www.unicode.org/reports/tr31/ | ||
| [TR15]: https://www.unicode.org/reports/tr15/ | ||
| [TR31Alternative]: http://unicode.org/reports/tr31/#Alternative_Identifier_Syntax | ||
| [TR31Layout]: https://www.unicode.org/reports/tr31/#Layout_and_Format_Control_Characters | ||
| [TR39Confusable]: https://www.unicode.org/reports/tr39/#Confusable_Detection | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Bringing my position inline so that anyone can reply to this directly.
I believe that by default we should:
latin,math,greek,hira/hiragana,ascii/reduced/limited/lets-party-like-its-1960,emoji, etc.). If a list of allowable scripts are defined in a crate, encountering chars outside of them is a hard error. On this error, confusables that are outside of the currently allowed scripts but that have a similar representation in the current script (𝜆inmath->λingreek), it should be pointed out with suggestions to either use the allowed script's char or to add the new script to the allowed list.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you please state the reasoning behind point three? If the supporting arguments were already discussed also include the opinions of other people on the matter.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Being liberal on the default is a contentious point with people bringing multiple positions. Myself I fall under the lets be relatively liberal camp, with maybe a concession to a warn by default lint when falling outside of ascii suggesting specifying allowed scripts. By doing it this way we minimize friction from non-ascii using users, while not heavily burdening ascii-only users (this is debated).
I believe we should allow people to specify specific allowed scripts is a good solution to the problem of cross script confusables sneaking in, without having people forced to decide that it is not worth it to use math symbols for formulas because they're are worried about cyrillic confusables being sneaked into the codebase. This would also allow me to use emojis for identifiers if I so choose, as that script would be disabled by default, but I could add it to the allowed list.
Allowing per scope customization would be interesting to allow people a lot of linting control when deciding that they want to allow a given script internally, but not on the public api. I would not count function argument names as part of the public api for this consideration, as we don't need to write them down, just be able to read them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would you say yourself that this is a fair and complete review of the previous arguments?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe it to be so, yes. I haven't seen my last point around per scope settings raised before, though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was discussed under the "malicious contributor" label and was deemed "not much trouble".
How does allowing more codepoints than UAX#31 Default Identifiers the evolution and readability of Rust? (See custom operators and smart quotes) For which people is it important to allow additional characters?
Why is the proposed confusable lint insufficient?
Why aren't ad hoc solutions like regular expressions sufficient?
Why are typos involving Unicode characters different from those only involving ASCII ones?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I feel that giving partial support for Unicode idents would be a disservice if we don't allow a future proof escape hatch.
Other than novelty uses (emoji), I don't know enough about other languages that might need it (my mother tongue is limited to latin).
Proactive warning and fine grained control for individual projects. A binary gate feels too blunt to mdgiven the scope.
I believe that the compiler and an editor should be enough to develop in Rust, and it should make an effort to provide help to newcomers for non obvious errors (and here I fall in the same camp of "avoid non ASCII in general" if possible, I just don't want to mandate it).
Tooling support. We already suggest based on levenshtein distance for ASCII typos, we should to the same for cobfusable chars.
Having said all of this, I'm enthusiastically in support of adding the proposed support and will do my best to make sure that the presented worries have a reasonable answer in the shape of compiler assisted support.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great 👍
Maybe update your position above if it only for novelty?
Care to give an example from your personal experience where such a feature in a programming language would have made your work easier? (Keep in mind that many languages have Unicode support for years now among them Python, Java and C++)
We already do (tested on nightly):
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My position is born of my own ignorance, I defer to people with more experience with non-western-european languages. For me (again, other than the novelty use of emoji and math symbols), my use case can be completely covered by latin (ñ, ü, é) and my counter case would be identifying confusables (which as you point out it would be handled) and bad diagnostics when changed code (smart quotes) is pasted.
Entire generation of Spanish speaking students have been "trained" to transliterate to ASCII, where
año(year) gets changed toanio(meaningless) orano(anus). I think that the focus for this document should be placed on the use of other scripts that limit developers more heavily than this.As pointed out, having a single toggle would make it hard to only allow full unicode support at different levels, like only for literals, idents, in comments, or maybe on idents for internal interfaces, while disallowing them in other places (like public interfaces).