Hygiene opt-out (escaping) for declarative macros 2.0

[alexreg]

This feature introduces the ability to "opt-out" of the usual macro hygiene rules within definitions of declarative macros (macros 2.0), for designated occurrences of identifiers. In other words, the feature will enable one to annotate occurrences of identifiers with macro call-site hygiene rather than the default definition-site hygiene.

Rendered

CC @jseyfried @petrochenkov @nrc

[Centril]

Nicely written :)

[Centril]

Could be good to mention here that "declarative macros" does not refer to macro_rules! (it is apparent if you click the link, but in the interest of not having to do so...)

[alexreg]

Fair point. I originally had this, but somehow removed it.

[Centril]

justified by much prior research and experience

A link would be good for curious readers :)

[alexreg]

I think the "hygienic macros" links offers good justification, no?

[Centril]

Truthfully I expected more papers and citations given "much prior research", but I suppose it's enough :)

[alexreg]

Hah, okay, I'll add one or two!

[Centril]

Could be more clear: "Place" => "location in the source code"

[alexreg]

Fair point.

[Centril]

So what is the relation of this RFC to #2151?

[alexreg]

None. I might add a sentence to make that clear.

[Centril]

The quote! macro uses #... Have you considered conflicts if and when quote is redefined as a 2.0 macro?

[kennytm]

🚲 I wonder if backslash ("escaping") can be valid

pub mod \foo {
    const \BAR: u32 = 123;
}

[alexreg]

@Centril No, I'm not sure. I wonder why it doesn't use $? Grr. Maybe someone can clarify for me whether it would conflict.

[alexreg]

Added an "unresolved question" about this, incidentally.

[Centril]

Typo here? Should say "In [example D][guide-example-d]"?

[alexreg]

yep!

[Centril]

cc @dtolnay on possible conflicts, due to using #, between quote! and decl macros...

[tikue]

I think the RFC doesn't specify the semantics of nested macro calls:

Escaping of hygiene for identifiers within macros allows one to define identifiers with syntax contexts (hygiene) corresponding to the location in the source code from which the macro is invoked (the call-site) rather than the location it is defined (definition-site).

Given:

x!();

macro x() {
    y!();
}

macro y() {
    struct #Foo;
}

In the invocation of x!(), what is the call site of y!()?

[petrochenkov]

and the syntax context for every identifier in the token tree is changed to the syntax context of the call site

What is this part about?
When a macro is expanded, an identifier gets an opaque mark added by default (Span::def_site() in proc macro API) or transparent mark if opt-out is in place (Span::call_site() in proc macro API), that's all what happens.

[alexreg]

Ah, I was slightly confused about how your transparent mark worked. I'll clarify that.

[alexreg]

Let me know if it's better now.

[petrochenkov]

@tikue
Call site of y!() is inside of x so the struct Foo cannot be accessed from outside of x.
So if you have a transparent macro you can always "contain" it with another macro and prevent further name leakage.

I agree that this presents a problem if you have several layers of macro helpers, this problem needs some other solution in addition to call-site hygiene.

[alexreg]

In the invocation of x!(), what is the call site of y!()?

The definition site of x, by definition. :-)

[alexreg]

I agree that this presents a problem if you have several layers of macro helpers, this problem needs some other solution in addition to call-site hygiene.

Possibly a proc macro that can change the syntax context to that of a given identifier?

Speaking of this, this whole feature could be implemented as a proc macro with eager expansion, couldn't it?

[petrochenkov]

Syntax context of an identifier is a sequence of marks RootMark -> Mark1 -> Mark2.
Both "def-site" and "call-site" variants change it, the former to RootMark -> Mark1 -> Mark2 -> OpaqueMark, the latter to RootMark -> Mark1 -> Mark2 -> TransparentMark.
(All this is an implementation detail anyway.)

[mark-i-m]

What exactly are Marks? What does the sequence of marks in this example mean?

[petrochenkov]

What exactly are Marks?

Right now a mark is a combination of expansion ID and transparency :)

What does the sequence of marks in this example mean?

A syntactic context fully identifying what macros produced an identifier (or other token).

I'll write some docs after doing a number of refactorings in the compiler.

[alexreg]

Yeah. And an expansion ID is a particular expansion (instance of an expansion) of a macro, as I understand. Furthermore, I believe a RootMark is constructed from a span or set of spans, though I'm not 100% clear on this. Perhaps @petrochenkov can clarify.

[alexreg]

Right, but I think I explain it correctly now at least. :-)

…

On 14 Jul 2018, at 18:26, Vadim Petrochenkov ***@***.***> wrote: @petrochenkov commented on this pull request. In text/0000-macro-hygiene-optout.md <#2498 (comment)>: > +While the motivation of this feature stems from defining or "exporting" new identifiers from macros to their call-site, where it is appropriate for the macro itself to choose/compute the name, it is clear from the above semantics that this feature allows for other potential uses cases. Most notably, one can use or "import" an identifier from their call-site. This, however, is *not* recommended, since this purpose is already fulfilled well by macro parameters. On the other hand, it is not explicitly disallowed, for two reasons: + +- Defining an identifier with call-site hygiene within that macro and then using it is a perfectly reasonable scenario. +- Macro expansion is performed at the syntactical (token stream) level, before parsing, so definitions and uses cannot be easily distinguished. + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +The macro parser routine first parses the macro definition into a token stream (as before), but now also tags tokens and meta-variables with an enum value representing the kind of hygiene (definition-site or call-site). This is only enabled for new-style `macro!` macros (i.e. *decl_macro* or macros 2.0); for `macro_rules!` macros, the call-site sigil `#` is not handled specially, and gives rise to an error. The sigil is always treated as a separate token outside of macros, on the LHS of macro rules, and when not followed by an identifier on the RHS. + +It should be noted that the sigil `#` has nothing to do with the syntax for [raw identifiers][rfc-raw-identifiers], and can be disambiguated without problems. In fact, they can be used together without any issues, e.g. `#r#foo`. + +When the macro is invoked (expanded), each token tree is transcribed according to the following rules, depending on its hygiene tag. + +- *definition-site*: a normal mark is applied for the current expansion, which leaves the syntax context alone +- *call-site*: a transparent mark is applied for the current expansion, which changes the syntax context for every identifier in the token tree to that of the call site. Syntax context of an identifier is a sequence of marks RootMark -> Mark1 -> Mark2. Both "def-site" and "call-site" variants change it, the former to RootMark -> Mark1 -> Mark2 -> OpaqueMark, the latter to RootMark -> Mark1 -> Mark2 -> TransparentMark. (All this is an implementation detail anyway.) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2498 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEF3P0rcDMWTfIOtHt9OfBxv7Q-AVZDks5uGinfgaJpZM4VPRYG>.

[mark-i-m]

@alexreg Thanks for the RFC! I'm glad to see macros 2.0 starting to get RFC-ified :)

I had a couple of questions:

• Is there any leaning towards making this an eRFC or do we have a reasonably good notion of what we want?
• Regardless of whether this RFC is accepted, would it be possible to get some of this content in the rustc-guide? Specifically the explanations of different syntax contexts, marks, etc. were instructive to me.

[alexreg]

Is there any leaning towards making this an eRFC or do we have a reasonably good notion of what we want?

I think this could work as an eRFC or RFC. I think we know what we want, and the feature is very well-motivated, but I'm starting to think this feature may be best implemented as a proc macro now. (And can thus be extensible to the case I mentioned above.)

Regardless of whether this RFC is accepted, would it be possible to get some of this content in the rustc-guide? Specifically the explanations of different syntax contexts, marks, etc. were instructive to me.

Yep, I think so. Let's wait until it's merged and I have a bit more experience with things, but I'd be glad to do that.

[alexreg]

@petrochenkov Can this feature be implemented using a proc macro yet? I presume not. What more would it require though?

[petrochenkov]

@alexreg
Probably yes, given enough effort?
Procedural macros can produce call-site spans on stable, can produce def-site spans on nightly, and can parse arbitrary text including Rust code with identifiers marked with #.

[alexreg]

Probably yes, given enough effort?
Procedural macros can produce call-site spans on stable, can produce def-site spans on nightly, and can parse arbitrary text including Rust code with identifiers marked with #.

I think it's worth considering a) whether we should implement this RFC as a proc macro, b) the alternative of lift!(ident) and lift!(ident, ident2) (where in the second case it changed the syntax context of ident to that of ident2). Thoughts on both of these?

[nrc]

Currently all lifetime parameters are unhygienic, not sure if we will fix that for macros 2.0 or not.

[alexreg]

Yeah. Hopefully we will!

[petrochenkov]

Lifetimes are already hygienic in macro macros and with Span::def_site() in proc macros.

[nrc]

The obvious semantics to me is that the resulting identifier takes the name from the metavariable and the hygiene context from the call site.

[alexreg]

Yes, I really meant the former in't obviously useful, why the latter isn't obviously useful either nor does it have obvious semantics.

[nrc]

I'd expect some discussion of how this works in other languages here. In particular, Scheme has a rich system for doing this sort of thing.

[alexreg]

Hmm. I'd like to avoid learning Scheme properly for this... maybe I can dig up a decent explanation somewhere?

[nrc]

Can this feature be implemented using a proc macro yet?

I had always imagined this feature being implemented as a proc macro, rather than having dedicated syntax. I imagined that we would provide some functions to proc macros for doing things like apply the hygiene context from a to name b to produce an identifier, as well as the manual manipulation of spans. I think we would also need something along the lines of a function or marker for saying 'don't apply the hygiene marking procedure to this identifier when we expand the proc macro'. Then a crate could provide a library of macros for use by devs to manipulate hygiene in common ways. (I think this may also require eager expansion? I forget the details exactly, but you need some way of saying expand this macro in the context of the use site, not the decl site.).

I think there is a problem with this approach which is what exactly is the use site? If you have nested macros, do you mean the root of the expansion or one level of expansion? If you pass a macro to a macro before expanding it, which use site do you use. What about interactions with eager expansion?

One solution is to always take the hygiene context from an identifier, so the user has to pass some identifier in. This can be done in conjunction with a 'concat idents' kind of macro too. This does make cases like your first example a bit weird. The alternative is that you pick a default and offer ways to access the other variations as needed.

So, the solution I would propose is that you change the RFC to the functions other support necessary to support ergonomic hygiene manipulation in proc macros, then create some library macros which use these to provide hygiene manipulation to decl macros and RFC any language changes required to make that work. I think this approach will lead to a more flexible and orthogonal system.

[alexreg]

@nrc

So, the solution I would propose is that you change the RFC to the functions other support necessary to support ergonomic hygiene manipulation in proc macros, then create some library macros which use these to provide hygiene manipulation to decl macros and RFC any language changes required to make that work. I think this approach will lead to a more flexible and orthogonal system.

Yes, that could work well. Having written this, I immediately started thinking an expanded proc macros system was the better way to go... I was just encouraged down this path initially. Do you think my lift! proc macro per above is something that could be integrated into the language? It seems like just about anyone writing hygienic macros would want to use it, really.

Also, maybe you could clarify if eager expansion is actually needed for this purpose? I was convinced it was needed for concat_idents in the past, but even there I forget the exact reason why.

If you're up for a chat on Discourse about what extensions we need to make to the proc macro system, that might be nice.

[tmccombs]

I think there should be some mention of computed unhygienic symbols (for example concatenating an input parameter with a constant string to create an output symbol) either as a possible extension or in alternatives.

[alexreg]

@tmccombs I see that as somewhat orthogonal to the concern of this RFC, although hygiene control/opt-out can certainly help with using it.

[tmccombs]

@alexreg I don't think it is completely orthogonol. If there was a mechanism to generate computed unhygienic symbols, that mechanism could probably be used to achieve the same goals as this RFC.

[alexreg]

@tmccombs There's no such thing as "unhygienic" symbols in the parse tree. The way we've discussed implementing the concat! macro for example is simply ignoring the hygiene info for the arguments, doing a string-like concatenation, and assigning the resulting ident token the hygiene of the call site.

[alexreg]

@nrc

I had always imagined this feature being implemented as a proc macro, rather than having dedicated syntax. I imagined that we would provide some functions to proc macros for doing things like apply the hygiene context from a to name b to produce an identifier, as well as the manual manipulation of spans. I think we would also need something along the lines of a function or marker for saying 'don't apply the hygiene marking procedure to this identifier when we expand the proc macro'. Then a crate could provide a library of macros for use by devs to manipulate hygiene in common ways. (I think this may also require eager expansion? I forget the details exactly, but you need some way of saying expand this macro in the context of the use site, not the decl site.).

Do you have any pointers to info about the current support for hygiene retrieval and manipulation in proc macros? I can then rework this RFC accordingly.

[mark-i-m]

What's the status of this?

[alexreg]

@mark-i-m The plan is to take some of the macros @nrc described in his blog post a while ago and integrate these into core, alongside macros like concat!. We then need macros in ident-position to make this actually useful in most cases, or at least some sort of equivalent (maybe restricting macros in ident-position to within macros). I think this debate should probably be reopened. Do you have any thoughts on it?

I'd also be curious to get @Centril's view on this, as the language design guru. ;-)

[mark-i-m]

Do you have any thoughts on it?

Thanks for asking, but I'm not really knowledgeable about hygiene at all. I was mainly curious what the progress was on macros 2.0. While I would like to see it continue, it seems from the Rust2019 posts that there are other things that are priorities ATM (e.g. GATs, specialization).

Macros in the ident position does seem like an excellent feature for exploring how Rust will proceed in this area, though. Is there more documentation that can be found about these issues somewhere?

[alexreg]

Macros in the ident position does seem like an excellent feature for exploring how Rust will proceed in this area, though. Is there more documentation that can be found about these issues somewhere?

Not a lot. There was an attempt at an RFC (a while back, by @Manishearth if I remember), and some discussion on a GitHub issue, plus a series of posts by @nrc (including that one), but nothing nothing more formal.

The nice thing about enhancements to the macro system is that they can largely proceed independently of the type system or trait solving, since there aren't too many interactions. The main thing macros needs to be concerned with are other syntactical developments, like what WG-Grammar are doing, perhaps.

[Manishearth]

(yeah, I posted an RFC , but this was back when RFCs were much smaller and also I was pretty new to it, so it's not a substantial RFC and it's not too relevant now)

[Manishearth]

If you decide to go the "special proc macro" route you'll definitely need eager expansion. One consistent way to add it to the syntax is $let:

$let $myident = lift!(hello);
let $myident =5;

IIRC there have been other proposals

[alexreg]

@Manishearth Yeah, no worries, I realised it doesn't have the usual level of detail of recent RFCs, but worth referencing in any case.

I've talked about this before with a few people, and I don't think there's any inherent reason we need eager expansion for this... unless I misremember something. Is there?

[Manishearth]

The reason is that supporting macros directly in ident positions is a bit of an annoying syntax minefield.

$let doesn't actually have to be eager expansion -- come to think of it in this case it shouldn't probably, but being able to bind macros to macro variables helps do this substitution well.

There are probably other solutions.

[alexreg]

@Manishearth Yeah, this is what I meant by "inherent"... the syntax gets ugly, and intrudes on normal Rust code more than usual, but there's no technical obstacle, as far as I'm aware. :-)

I recall discussing macro bindings as well, but I think we discounted this on the basis that syntactical substitutions don't really work like bindings, and to even create a coherent system, it would require a significant expansion in complexity.

I believe one argument was to allow macros in ident-position (normal, lazy expansion), but only allow it within other macros. This is nice in a way, but I think the main argument against is was consistency (if ident-position is allowed inside macros, why not every position? -- though I forget what others are currently disallowed). This is where eager expansion (most recent attempt here) comes back again, and starts to look like the best solution, if you combine it with allowing invocations in every position/context.

CC @pierzchalski

[pierzchalski]

Oh neat, identifier-position macros might be a use-case for the same 'eager expansion' macro idea from #2320 (I'm working on cleaning that up after discussions with @alexreg).

Assuming you've got your fancy hygiene-scope-adjusting, identifier-token-producing macro mk_ident!(), then you can avoid all the identifier-position parsing issues by writing something like:

eager! {
  x = mk_ident!();
  let #x = whatever;
}

Here, the line x = mk_ident!(); means "expand mk_ident!(), then bind the resulting tokens to x for the purposes of the next line", and the line let #x = whatever means "interpolate x into #x (I'm stealing the interpolation syntax from quote::quote!) then return that interpolated result".

[alexreg]

Yes, exactly like that. :-) (There will also be other token-producing macros, probably along the lines of https://www.ncameron.org/blog/untitledconcat_idents-and-macros-in-ident-position/, but I need to write up a short RFC for that.)

I just realised there are some ambiguities in the above syntax for eager!, however. Maybe something like the following would work better.

eager! {
    $x = mk_ident!(),
    let #x = whatever;
}

Or, if these forms of macros are still supported in 2018 (I forget):

eager!(x = mk_ident!(), ...) {
    let #x = whatever;
}

[nikomatsakis]

@rfcbot fcp postpone

Hello everyone; we discussed this RFC in our backlog bonanza. The consensus was that we that we should postpone it, as we don't think we have the bandwidth to see to it right now. We do think that macros need some more work, though, and that this RFC in particular is looking at real problems (even if we're not sure whether it's the right solution or not).

We would like to encourage folks to discuss "macros 2.0" when the time comes for us to discuss our upcoming roadmap (one of the procedural changes we have in mind is to make it clearer when we'd be open to bigger proposals).

[rfcbot]

Team member @nikomatsakis has proposed to postpone this. The next step is review by the rest of the tagged team members:

• @cramertj
• @joshtriplett
• @nikomatsakis
• @pnkfelix
• @scottmcm

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[rfcbot]

The final comment period, with a disposition to postpone, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

The RFC is now postponed.