Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

parse: merge fn syntax + cleanup item parsing #68728

Merged
merged 11 commits into from
Feb 14, 2020

Conversation

Centril
Copy link
Contributor

@Centril Centril commented Feb 1, 2020

Here we continue the work in #67131 in particular to merge the grammars of fn items in various positions.

A list of language level changes (as sanctioned by the language team in #65041 (comment) and #67131):

  • self parameters are now syntactically allowed as the first parameter irrespective of item context (and in function pointers). Instead, semantic validation (ast_validation) is used.

  • Syntactically, fn items in extern { ... } blocks can now have bodies (fn foo() { ... } as opposed to fn foo();). As above, we use semantic restrictions instead.

  • Syntactically, fn items in free contexts (directly in a file or a module) can now be without bodies (fn foo(); as opposed to fn foo() { ... }. As above, we use semantic restrictions instead, including for non-ident parameter patterns.

  • const extern fn feature gating is now done post-expansion such that we do not have conditional compatibilities of function qualifiers in parsing.

  • The FnFrontMatter grammar becomes:

    Extern = "extern" StringLit ;
    FnQual = "const"? "async"? "unsafe"? Extern? ;
    FnFrontMatter = FnQual "fn" ;

    That is, all item contexts now syntactically allow const async unsafe extern "C" fn and use semantic restrictions to rule out combinations previously prevented syntactically. The semantic restrictions include in particular:

    • fns in extern { ... } can have no qualifiers.
    • const and async cannot be combined.
  • To fuse the list-of-items parsing in the 4 contexts that items are allowed, we now must permit inner attributes (#![attr]) inside trait Foo { ... } definitions. That is, we now allow e.g. trait Foo { #![attr] }. This was probably an oversight due to not using a uniform parsing mechanism, which we now do have (fn parse_item_list). The semantic support (including e.g. for linting) falls out directly from the attributes infrastructure. To ensure this, we include a test for lints.

Put together, these grammar changes allow us to substantially reduce the complexity of item parsing and its grammar. There are however some other non-language improvements that allow the compression to take place.

A list of compiler-internal changes (in particular noting the parser-external data-structure changes):

  • We use enum AllowPlus/RecoverQPath/AllowCVariadic { Yes, No } in parser/ty.rs instead of passing around 3 different bools. I felt this was necessary as it was becoming mentally taxing to track which-is-which.

  • fn visit_trait_item and fn visit_impl_item are merged into fn visit_assoc_item which now is passed an AssocCtxt to check which one it is.

  • We change FnKind to:

    pub enum FnKind<'a> {
        Fn(FnCtxt, Ident, &'a FnSig, &'a Visibility, Option<&'a Block>),
        Closure(&'a FnDecl, &'a Expr),
    }

    with:

    pub enum FnCtxt {
        Free,
        Foreign,
        Assoc(AssocCtxt),
    }

    This is then taken advantage of in tweaking the various semantic restrictions as well as in pretty printing.

  • In ItemKind::Fn, we change P<Block> to Option<P<Block>>.

  • In ForeignItemKind::Fn, we change P<FnDecl> to FnSig and P<Block> to Option<P<Block>>.

  • We change ast::{Unsafety, Spanned<Constness>}> into enum ast::{Unsafe, Const} { Yes(Span), No } respectively. This change in formulation allow us to exclude Span in the case of No, which facilitates parsing. Moreover, we also add a Span to IsAsync which is renamed to Async. The new Spans in Unsafety and Async are then taken advantage of for better diagnostics. A reason this change was made is to have a more uniform and clear naming scheme.

    The HIR keeps the structures in AST (with those definitions moved into HIR) for now to avoid regressing perf.

  • Various cleanups, bug fixes, and diagnostics improvements are made along the way. It is probably best to understand those via the diffs.

I would recommend reviewing this commit-by-commit with whitespace changes hidden.

r? @estebank @petrochenkov

@Centril Centril added T-lang Relevant to the language team, which will review and decide on the PR/issue. relnotes Marks issues that should be documented in the release notes of the next release. labels Feb 1, 2020
@Centril Centril added this to the 1.43 milestone Feb 1, 2020
@petrochenkov petrochenkov self-assigned this Feb 1, 2020
@rust-highfive

This comment has been minimized.

src/librustc_ast_passes/ast_validation.rs Outdated Show resolved Hide resolved
src/librustc_parse/parser/item.rs Outdated Show resolved Hide resolved
src/librustc_ast_passes/ast_validation.rs Outdated Show resolved Hide resolved
src/librustc_ast_passes/ast_validation.rs Outdated Show resolved Hide resolved
src/libsyntax/ast.rs Outdated Show resolved Hide resolved
src/test/ui/async-await/no-const-async.rs Outdated Show resolved Hide resolved
src/test/ui/parser/doc-before-extern-rbrace.stderr Outdated Show resolved Hide resolved
src/test/ui/parser/duplicate-visibility.stderr Outdated Show resolved Hide resolved
src/librustc_parse/parser/item.rs Outdated Show resolved Hide resolved
@petrochenkov
Copy link
Contributor

Other comments:

Please avoid "fns" in error messages, e.g. "fn in extern blocks" or "only allowed in associated fns", use "functions" instead, or at least "fn items".


Functions taking multiple bools could use bitflags instead of enums, i.e. Foo::Yes, Bar::Yes -> Foo | Bar.


The FnFrontMatter grammar becomes:

I don't see default in this list.
Looks like it's not supported in non-trait positions, so the parsing is not fully unified yet?


This change in formulation allow us to exclude Span in the case of No, which facilitates parsing.

The No span was supposed to be used as an insertion point for missing qualifiers, but if it's not used, then I'm ok with removing it.


Looks like we are very close to using a single routine like parse_mod_items for all kinds of item containers (modules, traits, impls, extern blocks, but not expression blocks yet, that'll perhaps come later).
Then we'll be able to just filter away all items that are not fn/const/type/macro-call and report much much better errors for situations like

trait Tr {
    static S: u8 = 0;
}

which currently look pretty bad in tests.

@petrochenkov petrochenkov added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Feb 1, 2020
@petrochenkov
Copy link
Contributor

Meta: this PR also was too big to be review-able carefully.
I understand that it may require some double work, but some changes from this PR are mechanical or don't affect the logic, perhaps they could be split into a separate PR from changes affecting logic.
But maybe it's too late because I already reviewed it.

@bors

This comment has been minimized.

@Centril
Copy link
Contributor Author

Centril commented Feb 2, 2020

I made sure that all commits passed UI tests so it shouldn't be hard to take chunks and move them into separate PRs (with comments addressed). I'll try to do that in way that seems sensible.

@Centril
Copy link
Contributor Author

Centril commented Feb 2, 2020

Functions taking multiple bools could use bitflags instead of enums, i.e. Foo::Yes, Bar::Yes -> Foo | Bar.

I would prefer avoiding this as these enums are mostly unrelated flags. Merging them would result in making it harder to track what state is passed in and whatnot. Part of the reason for this refactoring was so that possibly getting rid of some of the flags would be facilitated and bitflags would make it harder than bools.

I don't see default in this list.
Looks like it's not supported in non-trait positions, so the parsing is not fully unified yet?

The front matter is only the "effect qualifiers" (intrinsic to functions themselves, and unrelated to other item forms) + the fn keyword. I'm not going beyond fns in merging right now. defaultness is a property of associated items so I would need to unify types and consts too. I think we need some design discussion re. whether to syntactically allow default mod x { ... } or whether defaultness should be moved into each variant that cares about it.

The No span was supposed to be used as an insertion point for missing qualifiers, but if it's not used, then I'm ok with removing it.

Didn't seem like they were used, but we can also recover insertion points from the start / ends as the order is fixed.

Centril added a commit to Centril/rust that referenced this pull request Feb 2, 2020
parser: syntactically allow `self` in all `fn` contexts

Part of rust-lang#68728.

`self` parameters are now *syntactically* allowed as the first parameter irrespective of item context (and in function pointers). Instead, semantic validation (`ast_validation`) is used.

r? @petrochenkov
Centril added a commit to Centril/rust that referenced this pull request Feb 2, 2020
parser: syntactically allow `self` in all `fn` contexts

Part of rust-lang#68728.

`self` parameters are now *syntactically* allowed as the first parameter irrespective of item context (and in function pointers). Instead, semantic validation (`ast_validation`) is used.

r? @petrochenkov
@bors

This comment has been minimized.

@Centril Centril force-pushed the towards-fn-merge branch 2 times, most recently from eeee74c to f8b5708 Compare February 2, 2020 23:39
Centril added a commit to Centril/rust that referenced this pull request Feb 5, 2020
…henkov

Towards unified `fn` grammar

Part of rust-lang#68728.

- Syntactically, `fn` items in `extern { ... }` blocks can now have bodies (`fn foo() { ... }` as opposed to `fn foo();`). As above, we use semantic restrictions instead.

- Syntactically, `fn` items in free contexts (directly in a file or a module) can now be without bodies (`fn foo();` as opposed to `fn foo() { ... }`. As above, we use semantic restrictions instead, including for non-ident parameter patterns.

- We move towards unifying the `fn` front matter; this is fully realized in rust-lang#68728.

r? @petrochenkov
Centril added a commit to Centril/rust that referenced this pull request Feb 5, 2020
…henkov

Towards unified `fn` grammar

Part of rust-lang#68728.

- Syntactically, `fn` items in `extern { ... }` blocks can now have bodies (`fn foo() { ... }` as opposed to `fn foo();`). As above, we use semantic restrictions instead.

- Syntactically, `fn` items in free contexts (directly in a file or a module) can now be without bodies (`fn foo();` as opposed to `fn foo() { ... }`. As above, we use semantic restrictions instead, including for non-ident parameter patterns.

- We move towards unifying the `fn` front matter; this is fully realized in rust-lang#68728.

r? @petrochenkov
Dylan-DPC-zz pushed a commit to Dylan-DPC-zz/rust that referenced this pull request Feb 6, 2020
…henkov

Towards unified `fn` grammar

Part of rust-lang#68728.

- Syntactically, `fn` items in `extern { ... }` blocks can now have bodies (`fn foo() { ... }` as opposed to `fn foo();`). As above, we use semantic restrictions instead.

- Syntactically, `fn` items in free contexts (directly in a file or a module) can now be without bodies (`fn foo();` as opposed to `fn foo() { ... }`. As above, we use semantic restrictions instead, including for non-ident parameter patterns.

- We move towards unifying the `fn` front matter; this is fully realized in rust-lang#68728.

r? @petrochenkov
@bors

This comment has been minimized.

@Centril Centril force-pushed the towards-fn-merge branch 3 times, most recently from 08b4e90 to 6879ce9 Compare February 13, 2020 15:27
@Centril Centril added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Feb 13, 2020
@Centril
Copy link
Contributor Author

Centril commented Feb 13, 2020

All remaining review comments have been addressed.
I'd recommend reviewing the remainder with white-space changes ignored.

@petrochenkov
Copy link
Contributor

@bors r+

@bors
Copy link
Contributor

bors commented Feb 13, 2020

📌 Commit ad72c3a has been approved by petrochenkov

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 13, 2020
Dylan-DPC-zz pushed a commit to Dylan-DPC-zz/rust that referenced this pull request Feb 13, 2020
…enkov

parse: merge `fn` syntax + cleanup item parsing

Here we continue the work in rust-lang#67131 in particular to merge the grammars of `fn` items in various positions.

A list of *language level* changes (as sanctioned by the language team in rust-lang#65041 (comment) and rust-lang#67131):

- `self` parameters are now *syntactically* allowed as the first parameter irrespective of item context (and in function pointers). Instead, semantic validation (`ast_validation`) is used.

- Syntactically, `fn` items in `extern { ... }` blocks can now have bodies (`fn foo() { ... }` as opposed to `fn foo();`). As above, we use semantic restrictions instead.

- Syntactically, `fn` items in free contexts (directly in a file or a module) can now be without bodies (`fn foo();` as opposed to `fn foo() { ... }`. As above, we use semantic restrictions instead, including for non-ident parameter patterns.

- `const extern fn` feature gating is now done post-expansion such that we do not have conditional compatibilities of function qualifiers *in parsing*.

- The `FnFrontMatter` grammar becomes:
   ```rust
   Extern = "extern" StringLit ;
   FnQual = "const"? "async"? "unsafe"? Extern? ;
   FnFrontMatter = FnQual "fn" ;
   ```

   That is, all item contexts now *syntactically* allow `const async unsafe extern "C" fn` and use semantic restrictions to rule out combinations previously prevented syntactically. The semantic restrictions include in particular:

   - `fn`s in `extern { ... }` can have no qualifiers.
   - `const` and `async` cannot be combined.

- To fuse the list-of-items parsing in the 4 contexts that items are allowed, we now must permit inner attributes (`#![attr]`) inside `trait Foo { ... }` definitions. That is, we now allow e.g. `trait Foo { #![attr] }`. This was probably an oversight due to not using a uniform parsing mechanism, which we now do have (`fn parse_item_list`). The semantic support (including e.g. for linting) falls out directly from the attributes infrastructure. To ensure this, we include a test for lints.

Put together, these grammar changes allow us to substantially reduce the complexity of item parsing and its grammar. There are however some other non-language improvements that allow the compression to take place.

A list of *compiler-internal* changes (in particular noting the parser-external data-structure changes):

- We use `enum AllowPlus/RecoverQPath/AllowCVariadic { Yes, No }` in `parser/ty.rs` instead of passing around 3 different `bool`s. I felt this was necessary as it was becoming mentally taxing to track which-is-which.

- `fn visit_trait_item` and `fn visit_impl_item` are merged into `fn visit_assoc_item` which now is passed an `AssocCtxt` to check which one it is.

- We change `FnKind` to:

  ```rust
  pub enum FnKind<'a> {
      Fn(FnCtxt, Ident, &'a FnSig, &'a Visibility, Option<&'a Block>),
      Closure(&'a FnDecl, &'a Expr),
  }
  ```

  with:

  ```rust
  pub enum FnCtxt {
      Free,
      Foreign,
      Assoc(AssocCtxt),
  }
  ```

  This is then taken advantage of in tweaking the various semantic restrictions as well as in pretty printing.

- In `ItemKind::Fn`, we change `P<Block>` to `Option<P<Block>>`.

- In `ForeignItemKind::Fn`, we change `P<FnDecl>` to `FnSig` and `P<Block>` to `Option<P<Block>>`.

- We change `ast::{Unsafety, Spanned<Constness>}>` into `enum ast::{Unsafe, Const} { Yes(Span), No }` respectively. This change in formulation allow us to exclude `Span` in the case of `No`, which facilitates parsing. Moreover, we also add a `Span` to `IsAsync` which is renamed to `Async`. The new `Span`s in `Unsafety` and `Async` are then taken advantage of for better diagnostics. A reason this change was made is to have a more uniform and clear naming scheme.

  The HIR keeps the structures in AST (with those definitions moved into HIR) for now to avoid regressing perf.

- Various cleanups, bug fixes, and diagnostics improvements are made along the way. It is probably best to understand those via the diffs.

I would recommend reviewing this commit-by-commit with whitespace changes hidden.

r? @estebank @petrochenkov
bors added a commit that referenced this pull request Feb 13, 2020
Rollup of 9 pull requests

Successful merges:

 - #68728 (parse: merge `fn` syntax + cleanup item parsing)
 - #68938 (fix lifetime shadowing check in GATs)
 - #69057 (expand: misc cleanups and simplifications)
 - #69108 (Use HirId in TraitCandidate.)
 - #69125 (Add comment to SGX entry code)
 - #69126 (miri: fix exact_div)
 - #69127 (Enable use after scope detection in the new LLVM pass manager)
 - #69135 (Spelling error "represening" to "representing")
 - #69141 (Don't error on network failures)

Failed merges:

r? @ghost
@bors bors merged commit ad72c3a into rust-lang:master Feb 14, 2020
@Centril Centril deleted the towards-fn-merge branch February 14, 2020 10:16
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request May 16, 2020
Pkgsrc changes:
 * Bump rust bootstrap version to 1.42.0, except for Darwin/i686 where the
   bootstrap is not (yet?) available.

Upstream changes:

Version 1.43.0 (2020-04-23)
==========================

Language
--------
- [Fixed using binary operations with `&{number}` (e.g. `&1.0`) not having
  the type inferred correctly.][68129]
- [Attributes such as `#[cfg()]` can now be used on `if` expressions.][69201]

**Syntax only changes**
- [Allow `type Foo: Ord` syntactically.][69361]
- [Fuse associated and extern items up to defaultness.][69194]
- [Syntactically allow `self` in all `fn` contexts.][68764]
- [Merge `fn` syntax + cleanup item parsing.][68728]
- [`item` macro fragments can be interpolated into `trait`s, `impl`s,
  and `extern` blocks.][69366]
  For example, you may now write:
  ```rust
  macro_rules! mac_trait {
      ($i:item) => {
          trait T { $i }
      }
  }
  mac_trait! {
      fn foo() {}
  }
  ```
These are still rejected *semantically*, so you will likely receive an error but
these changes can be seen and parsed by macros and
conditional compilation.


Compiler
--------
- [You can now pass multiple lint flags to rustc to override the previous
  flags.][67885] For example; `rustc -D unused -A unused-variables` denies
  everything in the `unused` lint group except `unused-variables` which
  is explicitly allowed. However, passing `rustc -A unused-variables -D unused` denies
  everything in the `unused` lint group **including** `unused-variables` since
  the allow flag is specified before the deny flag (and therefore overridden).
- [rustc will now prefer your system MinGW libraries over its bundled libraries
  if they are available on `windows-gnu`.][67429]
- [rustc now buffers errors/warnings printed in JSON.][69227]

Libraries
---------
- [`Arc<[T; N]>`, `Box<[T; N]>`, and `Rc<[T; N]>`, now implement
  `TryFrom<Arc<[T]>>`,`TryFrom<Box<[T]>>`, and `TryFrom<Rc<[T]>>`
  respectively.][69538] **Note** These conversions are only available when `N`
  is `0..=32`.
- [You can now use associated constants on floats and integers directly, rather
  than having to import the module.][68952] e.g. You can now write `u32::MAX` or
  `f32::NAN` with no imports.
- [`u8::is_ascii` is now `const`.][68984]
- [`String` now implements `AsMut<str>`.][68742]
- [Added the `primitive` module to `std` and `core`.][67637] This module
  reexports Rust's primitive types. This is mainly useful in macros
  where you want avoid these types being shadowed.
- [Relaxed some of the trait bounds on `HashMap` and `HashSet`.][67642]
- [`string::FromUtf8Error` now implements `Clone + Eq`.][68738]

Stabilized APIs
---------------
- [`Once::is_completed`]
- [`f32::LOG10_2`]
- [`f32::LOG2_10`]
- [`f64::LOG10_2`]
- [`f64::LOG2_10`]
- [`iter::once_with`]

Cargo
-----
- [You can now set config `[profile]`s in your `.cargo/config`, or through
  your environment.][cargo/7823]
- [Cargo will now set `CARGO_BIN_EXE_<name>` pointing to a binary's
  executable path when running integration tests or benchmarks.][cargo/7697]
  `<name>` is the name of your binary as-is e.g. If you wanted the executable
  path for a binary named `my-program`you would use
  `env!("CARGO_BIN_EXE_my-program")`.

Misc
----
- [Certain checks in the `const_err` lint were deemed unrelated to const
  evaluation][69185], and have been moved to the `unconditional_panic` and
  `arithmetic_overflow` lints.

Compatibility Notes
-------------------

- [Having trailing syntax in the `assert!` macro is now a hard error.][69548]
  This has been a warning since 1.36.0.
- [Fixed `Self` not having the correctly inferred type.][69340] This incorrectly
  led to some instances being accepted, and now correctly emits a hard error.

[69340]: rust-lang/rust#69340

Internal Only
-------------
These changes provide no direct user facing benefits, but represent significant
improvements to the internals and overall performance of `rustc` and
related tools.

- [All components are now built with `opt-level=3` instead of `2`.][67878]
- [Improved how rustc generates drop code.][67332]
- [Improved performance from `#[inline]`-ing certain hot functions.][69256]
- [traits: preallocate 2 Vecs of known initial size][69022]
- [Avoid exponential behaviour when relating types][68772]
- [Skip `Drop` terminators for enum variants without drop glue][68943]
- [Improve performance of coherence checks][68966]
- [Deduplicate types in the generator witness][68672]
- [Invert control in struct_lint_level.][68725]

[67332]: rust-lang/rust#67332
[67429]: rust-lang/rust#67429
[67637]: rust-lang/rust#67637
[67642]: rust-lang/rust#67642
[67878]: rust-lang/rust#67878
[67885]: rust-lang/rust#67885
[68129]: rust-lang/rust#68129
[68672]: rust-lang/rust#68672
[68725]: rust-lang/rust#68725
[68728]: rust-lang/rust#68728
[68738]: rust-lang/rust#68738
[68742]: rust-lang/rust#68742
[68764]: rust-lang/rust#68764
[68772]: rust-lang/rust#68772
[68943]: rust-lang/rust#68943
[68952]: rust-lang/rust#68952
[68966]: rust-lang/rust#68966
[68984]: rust-lang/rust#68984
[69022]: rust-lang/rust#69022
[69185]: rust-lang/rust#69185
[69194]: rust-lang/rust#69194
[69201]: rust-lang/rust#69201
[69227]: rust-lang/rust#69227
[69548]: rust-lang/rust#69548
[69256]: rust-lang/rust#69256
[69361]: rust-lang/rust#69361
[69366]: rust-lang/rust#69366
[69538]: rust-lang/rust#69538
[cargo/7823]: rust-lang/cargo#7823
[cargo/7697]: rust-lang/cargo#7697
[`Once::is_completed`]: https://doc.rust-lang.org/std/sync/struct.Once.html#method.is_completed
[`f32::LOG10_2`]: https://doc.rust-lang.org/std/f32/consts/constant.LOG10_2.html
[`f32::LOG2_10`]: https://doc.rust-lang.org/std/f32/consts/constant.LOG2_10.html
[`f64::LOG10_2`]: https://doc.rust-lang.org/std/f64/consts/constant.LOG10_2.html
[`f64::LOG2_10`]: https://doc.rust-lang.org/std/f64/consts/constant.LOG2_10.html
[`iter::once_with`]: https://doc.rust-lang.org/std/iter/fn.once_with.html
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
relnotes Marks issues that should be documented in the release notes of the next release. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-lang Relevant to the language team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants