Don't make statement nonterminals match pattern nonterminals

[compiler-errors]

Right now, the heuristic we use to check if a token may begin a pattern nonterminal falls back to may_be_ident:

rust/compiler/rustc_parse/src/parser/nonterminal.rs

Lines 21 to 37 in ef71f10

	fn may_be_ident(nt: &token::Nonterminal) -> bool {
	match nt {
	NtStmt(_)
	| NtPat(_)
	| NtExpr(_)
	| NtTy(_)
	| NtIdent(..)
	| NtLiteral(_) // `true`, `false`
	| NtMeta(_)
	| NtPath(_) => true,
	NtItem(_)
	| NtBlock(_)
	| NtVis(_)
	| NtLifetime(_) => false,
	}
	}

This has the unfortunate side effect that a stmt nonterminal eagerly matches against a pat nonterminal, leading to a parse error:

macro_rules! m {
    ($pat:pat) => {};
    ($stmt:stmt) => {};
}

macro_rules! m2 {
    ($stmt:stmt) => {
        m! { $stmt }
    };
}

m2! { let x = 1 }

This PR fixes it by more accurately reflecting the set of nonterminals that may begin a pattern nonterminal.

As a side-effect, I modified Token::can_begin_pattern to work correctly and used that in Parser::nonterminal_may_begin_with.

[rustbot]

r? @nnethercote

(rustbot has picked a reviewer for you, use r? to override)

[nnethercote]

Suggested change

	/// the stable set of tokens that are allowed to match an pat nonterminal.
	/// the stable set of tokens that are allowed to match a pat nonterminal.

[nnethercote]

AFAICT, the net effect here:

• All identifiers now match, i.e. more than before
• DotDotEq no longer matches
• BinOp(Or) matches if allow_leading_or is true
• NtExpr, NtMeta, NtTy now match
• NtBlock no longer matches

Is that right?

[compiler-errors]

Yes, though it should be noted that can_begin_pattern was previously only used for diagnostics and was previously just a rip off of can_begin_expr from above.

A better way to look at the changes here is to compare it to the previous match that was located in nonterminal_may_begin_with. This PR copies that logic over, adds an uninterpolate call for consistency with can_begin_expr (rather than manually handling the NtIdent arm), and adjusting the match arm for the Interpolated token.

To sum it up, I believe the only token kind that now no longer matches pat nonterminals is the statement nonterminal.

[nnethercote]

And the net effect here is that NtStmt no longer matches. Which I think is the change that actually fixes the original problem, right?

[compiler-errors]

Yes

[nnethercote]

How did you find this problem? Just curious.

This appears to change more than is necessary to fix the cited bug. Is that intended?

[compiler-errors]

How did you find this problem? Just curious.

I noticed it while authoring #120218, and thinking a bit about why there was a Token::can_begin_pattern (that, for the record, I believe I added for diagnostics purposes originally) that was not being used by Parser::nonterminal_may_begin_with.

This appears to change more than is necessary to fix the cited bug. Is that intended?

Yes. I don't see why we need to maintain an incorrect Token::can_begin_pattern just for diagnostics rather than just fixing it to actually reflect the set of tokens that can match pat nonterminals and using it everywhere.

[nnethercote]

Thanks for the explanations. r=me with the "a"/"an" nit fixed.

[compiler-errors]

Actually, given that this is a change to a contract (the macro matchers), I'll pass this by t-lang as well.

@rustbot label: +I-lang-nominated

[bors]

☔ The latest upstream changes (presumably #122830) made this pull request unmergeable. Please resolve the merge conflicts.

[scottmcm]

nit: the bool here is very non-obvious in callers. Dunno if it'd be worth having an enum or something to make it clearer, or to have the bool-taking thing be non-pub with different names for the two cases, or something.

[scottmcm]

From CE: Right now the tokens that a macro matcher may begin with is a stable guarantee. We are relaxing the assumption that pattern matchers may begin with statement metavariables ($var whose type is stmt), because when we actually try to parse such a pattern, we are always guaranteed to fail. This only allows more code to compile, and would only break future code if we specifically wanted to begin patterns with statement metavariable.

I agree that it's weird to allow a :stmt in a pattern, so am happy to say we won't. Let's see what others think, since this conversation was in a sparsely-attended triage meeting:

@rfcbot fcp merge

---

The other thing we explored was what it would take to make this actually work, since you can actually put an :expr into a pattern. But CE argued that we don't actually like that that works, it's just something we're stuck with because people used it before :literal was available, which seems fair.

[scottmcm]

From CE: Right now the tokens that a macro matcher may begin with is a stable guarantee. We are relaxing the assumption that pattern matchers may begin with statement metavariables ($var whose type is stmt), because when we actually try to parse such a pattern, we are always guaranteed to fail. This only allows more code to compile, and would only break future code if we specifically wanted to begin patterns with statement metavariable.

I agree that it's weird to allow a :stmt in a pattern, so am happy to say we won't. Let's see what others think, since this conversation was in a sparsely-attended triage meeting:

@rfcbot fcp merge

---

The other thing we explored was what it would take to make this actually work, since you can actually put an :expr into a pattern. But CE argued that we don't actually like that that works, it's just something we're stuck with because people used it before :literal was available, which seems fair.

[rfcbot]

Team member @scottmcm has proposed to merge this. The next step is review by the rest of the tagged team members:

• @joshtriplett
• @nikomatsakis
• @pnkfelix
• @scottmcm
• @tmandry

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

cc @rust-lang/lang-advisors: FCP proposed for lang, please feel free to register concerns.
See this document for info about what commands tagged team members can give me.