Improve diagnostics for non XID characters in identifiers

[Manishearth]

fn main() {
    let 🦀 = "Manish";
}

errors with

error: unknown start of token: \u{1f980}
 --> src/main.rs:2:9
  |
2 |     let 🦀 = "Manish";

For non-ASCII characters, we should perhaps error with something better. I don't know what the error text should be (cc @estebank), because it's not just emoji, and there's no easy way to define "XID characters" without just linking to the spec, which seems bad. Maybe we can link to the reference?

Tagging as easy since the implementation isn't tricky, but we will probably need to figure out a good error message.

[estebank]

We probably should rework the parser to accept these as identifiers, and reject them in a later pass.

[Manishearth]

@estebank It's not that simple, bear in mind that this includes other punctuation characters and whitespace and such. A lot of these things are not identifier-like. We may need to figure out a heuristic for identifier-like, or we can just show a better error for all cases.

[jeanlucthumm]

@rustbot claim

[estebank]

It's not that simple, bear in mind that this includes other punctuation characters and whitespace and such. A lot of these things are not identifier-like.

That's fair. I left out that the recovery should be sane and only be done for things that could be an identifier, but completely agree.

We may need to figure out a heuristic for identifier-like, or we can just show a better error for all cases.

That is something that applies to all errors 🌲

[Manishearth]

Note that it is possible to use unicode general categories to tell "could be identifier" apart from "probably not" but it's still iffy.

[jeanlucthumm]

How does the E-mentor tag work in this case? I'm not sure who my point of contact is.

[Manishearth]

@jeanlucthumm just ask questions here, I'll help