Stabilize Frontmatter

[epage]

Stabilization report

FCP

Summary

This report proposes the stabilization of #[feature(frontmatter)] in preparation for Cargo Script to be stabilized.

A frontmatter is a special kind of attribute intended to be read and modified by external tools, like Cargo. This puts a particular constraint on this attribute in that full awareness of the Rust grammar would be prohibitive.

For example:

#!/usr/bin/env cargo
---
[dependencies]
clap = "4"
---

fn main() {
}

The goal of Cargo script is to improve:

• Learning by reducing the bar for experimenting and prototyping
• Collaboration, including bug reports, chats, blogs, and books
• Interoperability between tools that need similar solutions (playgrounds,
• One off utilities where it makes sense (e.g. integrating with Rust-specific libraries)

Closes #136889

Tracking:

• Tracking Issue for frontmatter #136889
• Tracking Issue for cargo-script RFC 3424 cargo#12207

Reference PRs:

• docs(ref): Specify frontmatter reference#1974

cc @rust-lang/lang @rust-lang/lang-advisors, @rust-lang/compiler

What is stabilized

This is stabilizing frontmatter, an optional section in a source file that can only exist before any Rust code or comments, for use by external tools.

#!/usr/bin/env cargo

---
[dependencies]
clap = "4"
---

fn main() {
}

After the opening frontmatter fence, an infostring is allowed that can identify the contents.
Calling tools are responsible for interpreting the infostring.

What isn't stabilized

Only a subset of the allowed infostring syntax is being stabilized.

• RFC: "an optional term (one or more characters excluding whitespace and commas)"
• rustc: (XID_Start | _) ( XID_Continue | -|. )*
• The RFC allowed - in the start of an infostring which is ambiguous with frontmatter open which accepts 3 or more -
• We simplified to a minimal, closed set as a precaution with a focus on what would make most file names work
• See Implement RFC 3503: frontmatters #140035

Future possibilities include:

• Infostring attributes: the supported syntax for infostrings was restricted to leave the door open
• Tracking this as an attribute in the AST
• Multiple frontmatters

Design

Reference

• docs(ref): Specify frontmatter reference#1974

RFC history

• RFC 3503 adds a new lexical section to the Reference to describe the frontmatter, including the grammar.

Answers to unresolved questions

What questions were left unresolved by the RFC? How have they been answered? Link to any relevant lang decisions.

The RFC has no unresolved questions

Post-RFC changes

What other user-visible changes have occurred since the RFC was accepted? Describe both changes that the lang team accepted (and link to those decisions) as well as changes that are being presented to the team for the first time in this stabilization report.

• infostring: allowed characters (discussed under "What isn't stabilized")
• whitespace:
  • RFC: "whitespace"
  • rustc: "horizontal whitespace" per the Unicode standard
  • The intention when written was horizontal whitespace and I overlooked how broad rustc defines it. See also #145971
• acceptable places for frontmatter
  • RFC: "This applies anywhere shebang stripping is performed. For example, if include! strips shebangs, then it will also frontmatter."
  • rustc: support removed from --cfg (Disallow frontmatter in --cfg and --check-cfg arguments #146137) and proc-macros (Strip frontmatter in fewer places #146340)
  • rustc: support removed from include! in expression/statement-content (still supported in item context)
    • See Don't strip shebang in expr-ctxt include!(…) #146377 for having shebangs match
    • discussion at Stabilize Frontmatter #148051 (comment)
  • It was generally unexpected that shebangs were supported in --cfg
  • include! can be used to inject code into the middle of a file which becomes ambiguous between a frontmatter start and a very negated number (----x)
• cap code fence -s to 255 like # in raw string literals
  • see Stabilize Frontmatter #148051 (comment)

Key points

What decisions have been most difficult and what behaviors to be stabilized have proved most contentious? Summarize the major arguments on all sides and link to earlier documents and discussions.

• whitespace between opening and infostring
  • The RFC specified it
  • Initially, Implement RFC 3503: frontmatters #140035 was going to remove it until the discussion at Implement RFC 3503: frontmatters #140035 (comment)
  • Reasons against: "What benefit would allowing whitespace before the infostring give?"
  • Reasons for, at that time: RFC said it and we're matching markdown
  • Additional reasons for: style team has said that they want to specify the use of the space (see #145617).
• rust-analyzer wanting an external indicator for cargo-scripts
  • This would allow them to differentiate between unlinked files and cargo scripts without ambiguity and without having to read the content. The latter means they either pre-load and take a performance hit or load on-demand
  • Resolving this runs counter to some of the design goals within the RFC
  • From the outside, it is not clear why this is as big of a problem as they feel it is
  • As this is re-litigating an RFC decision, the responsibility for making the case for why this is important was put on them and there has been no progress on this in the last 2 months
  • See also #t-compiler/rust-analyzer > Frontmatter vs `import_rules!` for scripts @ 💬
• whether indented --- needs to be escaped
  • The RFC only said that --- at the start of a line needs escaping
  • #140035 also had indented --- needing escaping
  • #145754 switched the implementation to match the RFC
• interactions with -Zunpretty
  • questions raised at Add unstable frontmatter support #137193 (comment)
  • went forward with passing them through like shebang at fix: Include frontmatter in -Zunpretty output  #143708
• whether external tools should silently fail, delegating error reporting to rustc
  • Cargo has gone with high quality error reporting to report problems quickly, including non-compiling commands, rather than deferring errors to compilation. In particular, the contents of the frontmatter can make a big difference in how compilation happens.

Nightly extensions

Are there extensions to this feature that remain unstable? How do we know that we are not accidentally committing to those?

No nightly extensions exist

Doors closed

What doors does this stabilization close for later changes to the language? E.g., does this stabilization make any other RFCs, lang experiments, or known in-flight proposals more difficult or impossible to do later?

There are no known efforts that this affects.

Feedback

Call for testing

Has a "call for testing" been done? If so, what feedback was received?

Call for testing with feedback at:

• the call for testing
• reddit
• a mastodon thread
• a mastodon thread

Previous call for testing: rust-lang/rfcs#3424 (comment) with some feedback at rust-lang/cargo#12207 (comment)

Overall, feedback is enthusiastic. Any quibbles are with the Cargo side.

Nightly use

Do any known nightly users use this feature? Counting instances of #![feature(FEATURE_NAME)] on GitHub with grep might be informative.

Samples from grepping for -Zscript

• https://github.com/rustls/rustls/blob/main/admin/threads-seq.rs
• https://github.com/FractalFir/rustc_codegen_clr/blob/main/bin/fuzz.rs
• https://github.com/ariel-os/ariel-os/blob/main/scripts/vscode.rs
• https://github.com/ariel-os/ariel-os/blob/main/doc/gen_book.rs
• https://github.com/csmoe/iced-webview/blob/main/examples/mac_bundler.rs
• https://github.com/csmoe/tracing-perfetto/blob/main/update.rs
• https://github.com/Artisan-Lab/tag-std/blob/main/safety-tool/gen_rust_toolchain_toml.rs
• https://github.com/risingwavelabs/risingwave/blob/main/e2e_test/source_inline/pubsub/prepare-data.rs
• https://github.com/ReamLabs/ream/blob/master/book/cli/help.rs
• https://github.com/ChristopherBiscardi/advent-of-code/blob/main/2024/rust/scripts/get-aoc-input.rs

Implementation

Major parts

Summarize the major parts of the implementation and provide links into the code and to relevant PRs.

See, e.g., this breakdown of the major parts of async closures:

• https://rustc-dev-guide.rust-lang.org/coroutine-closures.html

Parts:

• lexer
• parser

PRs:

• Implement RFC 3503: frontmatters #140035
• Fix parsing of frontmatters with inner hyphens #142032
• fix: Include frontmatter in -Zunpretty output  #143708
• ignore frontmatters in TokenStream::new #145568
• fix(lexer): Allow '-' in the frontmatter infostring continue set #145751
• fix(lexer): Don't require frontmatters to be escaped with indented fences #145754
• test(rustfmt): Verify frontmatter is preserved #145766
• fix(lexer): Only allow horizontal whitespace in frontmatter  #146106
• Disallow frontmatter in --cfg and --check-cfg arguments #146137
• Strip frontmatter in fewer places #146340
• Fix a crash/mislex when more than one frontmatter closing possibility is considered #146899
• test(frontmatter): Cover spaces between infostring parts #148042
• test(frontmatter): Rename tests to make coverage more obvious #148073
• fix(parse): Limit frontmatter fences to 255 dashes  #149358

Coverage

Summarize the test coverage of this feature.

Consider what the "edges" of this feature are. We're particularly interested in seeing tests that assure us about exactly what nearby things we're not stabilizing. Tests should of course comprehensively demonstrate that the feature works. Think too about demonstrating the diagnostics seen when common mistakes are made and the feature is used incorrectly.

Within each test, include a comment at the top describing the purpose of the test and what set of invariants it intends to demonstrate. This is a great help to our review.

Describe any known or intentional gaps in test coverage.

Contextualize and link to test folders and individual tests.

Tests (directory):

• Location
  • May or may not be preceded by a shebang
  • Can't have comments (or other non-whitespace content) before opening
  • Whitespace is allowed before frontmatter
• General
  • CR/LR is supported
  • Invalid to have multiple frontmatters
• Infostring
  • Can't have a , in the infostring
  • Can't have a leading . or -
  • Can have a . or - elsewhere
  • Can be preceded by horizontal whitespace
• Fences
  • Invalid to have non-whitespace content after close
  • Invalid to have whitespace between newline and fence
  • Horizontal whitespace is allowed after fences
  • Fence must be closed
    • On missing close, recover after use
  • Open close lengths must match
• Content
  • Cover whitespace corner cases in content
  • Multi-byte characters don't cause crashes
  • Allow characters not accepted by the lexer
• Escaping
  • Non-leading --- doesn't need escaping
  • More - escape lines starting with fewer -
• Other uses
  • include! doesn't treat --- as frontmatter
  • proc-macro doesn't treat --- as frontmatter

Outstanding bugs

What outstanding bugs involve this feature? List them. Should any block the stabilization? Discuss why or why not.

No known bugs

Outstanding FIXMEs

What FIXMEs are still in the code for that feature and why is it OK to leave them there?

An idea for future editions

rust/compiler/rustc_parse/src/lib.rs

Lines 176 to 180 in 779e19d

	// FIXME(frontmatter): Consider stripping frontmatter in a future edition. We can't strip them
	// in the current edition since that would be breaking.
	// See also <https://github.com/rust-lang/rust/issues/145520>.
	// Alternatively, stop stripping shebangs here, too, if T-lang and crater approve.
	source_file_to_stream(psess, source_file, override_span, StripTokens::Shebang)

Diagnostic improvements can be iterated on over time, particularly with use feedback

rust/tests/ui/frontmatter/frontmatter-after-tokens.rs

Lines 3 to 6 in 779e19d

	---
	//~^ ERROR: expected item, found `-`
	// FIXME(frontmatter): make this diagnostic better
	---

rust/tests/ui/frontmatter/multifrontmatter.rs

Lines 1 to 7 in 779e19d

	---
	---
	---
	//~^ ERROR: expected item, found `-`
	// FIXME(frontmatter): make this diagnostic better
	---

Both can be handled as need is shown.

Tool changes

What changes must be made to our other tools to support this feature. Has this work been done? Link to any relevant PRs and issues.

• rustfmt (non-blocking per https://rust-lang.zulipchat.com/#narrow/channel/357797-t-rustfmt/topic/Parsing.20of.20frontmatters/near/535570252)
  • docs(style): Specify the frontmatter style #145617 (style guide)
  • test(rustfmt): Verify frontmatter is preserved #145766 (frontmatter preserved)
  • Style frontmatter according to the style guide rustfmt#6709 (formatting)
• rust-analyzer
  • feat(parser): Don't error on frontmatter rust-analyzer#20854 (syntax support)
  • Add shebang support for "cargo script" rust-analyzer#15318 (attaching scripts)
• rustdoc (both JSON and HTML)
  • Open items present on doctests
• cargo (dependent on this)
  • Tracking Issue for cargo-script RFC 3424 cargo#12207
• clippy
  • just works
• rustup
  • N/A
• docs.rs
  • N/A
• playground (non-blocking)
  • Add cargo-script support rust-playground#1049

Breaking changes

If this stabilization represents a known breaking change, link to the crater report, the analysis of the crater report, and to all PRs we've made to ecosystem projects affected by this breakage. Discuss any limitations of what we're able to know about or to fix.

No known breaking change

Type system, opsem

Compile-time checks

What compilation-time checks are done that are needed to prevent undefined behavior?

Link to tests demonstrating that these checks are being done.

N/A, this is only relevant to tools inspecting the source

Type system rules

What type system rules are enforced for this feature and what is the purpose of each?

N/A, this is only relevant to tools inspecting the source

Sound by default?

Does the feature's implementation need specific checks to prevent UB, or is it sound by default and need specific opt-in to perform the dangerous/unsafe operations? If it is not sound by default, what is the rationale?

No, this is only relevant to tools inspecting the source

Breaks the AM?

Can users use this feature to introduce undefined behavior, or use this feature to break the abstraction of Rust and expose the underlying assembly-level implementation? Describe this if so.

No, this is only relevant to tools inspecting the source

Common interactions

Temporaries

Does this feature introduce new expressions that can produce temporaries? What are the scopes of those temporaries?

No, this is only relevant to tools inspecting the source

Drop order

Does this feature raise questions about the order in which we should drop values? Talk about the decisions made here and how they're consistent with our earlier decisions.

No, this is only relevant to tools inspecting the source

Pre-expansion / post-expansion

Does this feature raise questions about what should be accepted pre-expansion (e.g. in code covered by #[cfg(false)]) versus what should be accepted post-expansion? What decisions were made about this?

No, this is only relevant to tools inspecting the source

Edition hygiene

If this feature is gated on an edition, how do we decide, in the context of the edition hygiene of tokens, whether to accept or reject code. E.g., what token do we use to decide?

N/A, this is independent of editions

SemVer implications

Does this feature create any new ways in which library authors must take care to prevent breaking downstreams when making minor-version releases? Describe these. Are these new hazards "major" or "minor" according to RFC 1105?

N/A, downstream users cannot access this feature

Exposing other features

Are there any other unstable features whose behavior may be exposed by this feature in any way? What features present the highest risk of that?

Not aware of any

History

List issues and PRs that are important for understanding how we got here.

• eRFC for Cargo Scripts
• RFC for Cargo Script
  • RFC for Frontmatter

PRs

• Implement RFC 3503: frontmatters #140035
• Fix parsing of frontmatters with inner hyphens #142032
• fix: Include frontmatter in -Zunpretty output  #143708
• ignore frontmatters in TokenStream::new #145568
• fix(lexer): Allow '-' in the frontmatter infostring continue set #145751
• fix(lexer): Don't require frontmatters to be escaped with indented fences #145754
• test(rustfmt): Verify frontmatter is preserved #145766
• fix(lexer): Only allow horizontal whitespace in frontmatter  #146106
• Disallow frontmatter in --cfg and --check-cfg arguments #146137
• Strip frontmatter in fewer places #146340
• Fix a crash/mislex when more than one frontmatter closing possibility is considered #146899
• test(frontmatter): Cover spaces between infostring parts #148042
• test(frontmatter): Rename tests to make coverage more obvious #148073

Acknowledgments

• @fee1-dead
• @fmease
• @matthewjasper
• @Teapot4195
• @Urgau

Open items

List any known items that have not yet been completed and that should be before this is stabilized.

• Should frontmatter in doctests always error? See also https://rust-lang.zulipchat.com/#narrow/channel/266220-t-rustdoc/topic/rustdoc.20and.20shebang.2Ffrontmatter/near/500470213
• Quality of errors when frontmatter are in doctests, see also Provide a more helpful diagnostic on "shebang lookalikes" #137249

[rustbot]

Some changes occurred in src/doc/style-guide

cc @rust-lang/style

[rustbot]

r? @davidtwco

rustbot has assigned @davidtwco.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[fmease]

(Commenting on an arbitrary line to make this discussion threaded)

From the PR description (emphasis and ellipses mine):

Post-RFC changes

[…]

• acceptable places for frontmatter
  • […]
  • rustc: support removed from […] include! […]
  • […]
  • include! can be used to inject code into the middle of a file which becomes ambiguous between a frontmatter start and a very negated number (----x)

In #146340 I actually struck a (temporary) compromise: We do still strip/lex/recognize frontmatter (and shebang) in item-context include!s. However, we no longer strip frontmatter in expression/statement-context include!s like in let _ = include!(…); (to prevent the manifold negation backcompat issue you've rightly mentioned).

Now, I don't know what's best here. I'm eager to know your and T-lang's stance on the matter. Note that we do strip shebang in expression/statement-context include!s which I consider to be quite odd (but still understandable on a lexical level). I have an open PR #146377 which would stop stripping shebang in that position, too. I still need to rebase+polish, lang-nominate and possibly crater it. The outcome of that T-lang discussion likely affects the decision mentioned in the first paragraph (†).

(†): If that PR was accepted it would mean that we would strip shebang+frontmatter in item-ctxt includes and wouldn't strip either(!) shebang or frontmatter in expr/stmt-ctxt includes which would make the behavior of shebang+frontmatter consistent thereby fulfilling the original goal of "This applies anywhere shebang stripping is performed." in this specific case (obviously the goal wasn't achieved in other cases, like in the proc_macro API case).

[epage]

Thanks! I've updated the stabilization report as well as linked to your comment as a place for discussing it.

[fmease]

I've now made that PR ready & lang-nominated it, sorry for the delay. See the nomination text here: #146377 (comment) (it's maybe a bit rambly since I threw it together rather quickly, I hope it's comprehensible enough).

[epage]

Thanks!

[davidtwco]

Implementation of this LGTM, r=me once t-lang has approved stabilisation

View changes since this review

[bors]

☔ The latest upstream changes (presumably #148090) made this pull request unmergeable. Please resolve the merge conflicts.

[epage]

@rustbot label +I-lang-nominated

This has been quiet for a while. There is an open issue on the rustdoc side but I figure that can be worked out in parallel to T-lang discussions in case T-lang uncovers anything. Even if this goes straight to FCP, someone can raise a blocking concern on behalf of rustdoc.

[traviscross]

Having now read through the stabilization report and the linked PRs, issues, and threads, this seems to me to be ready for stabilization modulo the open rustdoc item mentioned by @epage, getting the Reference PR into a final state, and documenting the behavior of include! with respect to frontmatter (for which I'll file a concern). Consequently, I propose...

@rfcbot fcp merge lang

Thanks to @epage for the thorough stabilization report and for his many efforts in pushing along this feature.

[rust-rfcbot]

Team member @traviscross has proposed to merge this. The next step is review by the rest of the tagged team members:

• @joshtriplett
• @nikomatsakis
• @scottmcm
• @tmandry
• @traviscross

Concerns:

• document-frontmatter-removal (Stabilize Frontmatter #148051 (comment))
• do-we-care-about-context-free-grammar resolved by Stabilize Frontmatter #148051 (comment)
• do-we-want-to-allow-carriage-returns (Stabilize Frontmatter #148051 (comment))
• jack-should-we-strip-frontmatter-in-item-context (Stabilize Frontmatter #148051 (comment))

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

cc @rust-lang/lang-advisors: FCP proposed for lang, please feel free to register concerns.
See this document for info about what commands tagged team members can give me.

[traviscross]

@rfcbot concern do-we-want-to-allow-carriage-returns

We talked about this in the lang call today. We generally agreed that, in the interest of consistency and conservatism, it'd make sense to not allow CRs that survive CRLF normalization within frontmatter, just as we reject these within raw string literals. If someone has a use-case for these, we could always allow it later (with a lint), and if we did, perhaps it'd make sense to similarly allow it (with a lint) for raw string literals.

[epage]

We talked about this in the lang call today. We generally agreed that, in the interest of consistency and conservatism, it'd make sense to not allow CRs that survive CRLF normalization within frontmatter, just as we reject these within raw string literals. If someone has a use-case for these, we could always allow it later (with a lint), and if we did, perhaps it'd make sense to similarly allow it (with a lint) for raw string literals.

I think likening frontmatter to raw string literals can be taken too far. While on a practical level I am not blocked on stray carriage returns, on a conceptual level, this feels off. This is a section we are intentionally saying "this is owned by an external tool" and so to apply Rust specific rules to the contained content feels off to me.

[tmandry]

I share your conceptual model @epage. The argument that convinced me was: It can catch confusing situations for users where they have a stray CR that looks like a line break but isn't, and since we would want a lint even if we do allow it, the simplest thing is just not to allow it. This simplifies the reference now and doesn't prevent us from adding the capability later.

[epage]

While I disagree, I've gone ahead and posted #149823.

[epage]

From #149823 (comment)

Given that text-direction-codepoint-in-literal and text-direction-codepoint-in-comment are lints, I do think it makes sense for this and the similar error in raw strings to be a lint as well. I doubt that anyone is going to allow it, but I still think it makes sense to not be a hard error.

That said, I don't think switching this from a hard error to a lint should be a blocker. Let's go ahead with the simple version of this change to unblock stabilization.

From #149823 (comment)

joshtriplett this only prevents the use of CR and not the text direction code points.

However, this is making me more against doing CR in the first place if CR as an error is just a "simple version" of what we actually want.

If these should be lints, they should be lints in the tool that owns processing the frontmatter. If we have them as lints in rustc, we then have both tools running those lints as they likely have a format outside of the frontmatter that should also lint and that would be generic code across the files. Having both lint can lead to a bad user experience including:

* disagreeing on lint levels and the user having to manage that

* potentially showing duplicate warnings

If we error on CR now and then remove it because tools should do it, that is a subtle change that could easily get lost in communicating back out to those tools.

Something else to keep in mind with all of this is that at least for use cases like cargo or buck2, they'll process it before rustc sees it, so if they feel it is an issue it will be handled already.

[rustbot]

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

[rust-log-analyzer]

The job aarch64-gnu-llvm-20-1 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

---- [ui] tests/ui/frontmatter/fence-too-many-dashes.rs stdout ----
Saved the actual stderr to `/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/frontmatter/fence-too-many-dashes/fence-too-many-dashes.stderr`
diff of stderr:

1 error: too many `-` symbols: frontmatter openings may be delimited by up to 255 `-` symbols, but found 256
2 
- error: aborting due to 1 previous error
+ warning: the feature `frontmatter` has been stable since 1.94.0-nightly and no longer requires an attribute to enable
+   --> $DIR/fence-too-many-dashes.rs:7:12
+    |
---
To only update this specific test, also pass `--test-args frontmatter/fence-too-many-dashes.rs`

error: 1 errors occurred comparing output.
status: exit status: 1
command: env -u RUSTC_LOG_COLOR RUSTC_ICE="0" RUST_BACKTRACE="short" "/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/tests/ui/frontmatter/fence-too-many-dashes.rs" "-Zthreads=1" "-Zsimulate-remapped-rust-src-base=/rustc/FAKE_PREFIX" "-Ztranslate-remapped-path-to-local-path=no" "-Z" "ignore-directory-in-diagnostics-source-blocks=/cargo" "-Z" "ignore-directory-in-diagnostics-source-blocks=/checkout/vendor" "--sysroot" "/checkout/obj/build/aarch64-unknown-linux-gnu/stage2" "--target=aarch64-unknown-linux-gnu" "--check-cfg" "cfg(test,FALSE)" "--error-format" "json" "--json" "future-incompat" "-Ccodegen-units=1" "-Zui-testing" "-Zdeduplicate-diagnostics=no" "-Zwrite-long-types-to-disk=no" "-Cstrip=debuginfo" "--emit" "metadata" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/frontmatter/fence-too-many-dashes" "-A" "unused" "-W" "unused_attributes" "-A" "internal_features" "-A" "unused_parens" "-A" "unused_braces" "-Crpath" "-Cdebuginfo=0" "-Lnative=/checkout/obj/build/aarch64-unknown-linux-gnu/native/rust-test-helpers"
stdout: none
--- stderr -------------------------------
error: too many `-` symbols: frontmatter openings may be delimited by up to 255 `-` symbols, but found 256

warning: the feature `frontmatter` has been stable since 1.94.0-nightly and no longer requires an attribute to enable
##[warning]  --> /checkout/tests/ui/frontmatter/fence-too-many-dashes.rs:7:12
   |
LL | #![feature(frontmatter)]

[nikomatsakis]

@rfcbot reviewed

[workingjubilee]

re: jack-should-we-strip-frontmatter-in-item-context

We landed #146377, do we have an answer for this concern now? I see that PR landed as about expr-context stripping? Do we not have an answer for item-context yet?

[jackh726]

The concern is still open because, yes, #146377 is only expr-ctxt. Note that this sort of overlaps with the document-frontmatter-removal concern - if when that is documented, the lang team agrees that the current behavior is appropriate, then I'll withdraw my concern. But I wanted to make sure to file a distinct concern to be sure the team considers the question.

[joshtriplett]

This is still blocked, for a variety of reasons. With my champion hat on: I think we should revisit it in a lang meeting soon. @traviscross, could you bump it up, please, so we can unblock it?

Summary of what needs lang attention:

Major: Revisit and definitively determine the desired behavior for bare CRs

When we originally looked at this, we said it should disallow bare CRs (as opposed to CR-LF pairs) in frontmatter, due to the way they'd cause the code to look. @epage raised the concern, which as the champion I agree with, that disallowing CRs will create potential conflicts with whatever tool parses the frontmatter: if the tool emits warnings or errors for bare CRs, there would be duplicate warnings from the tool and from rustc. @epage proposes that the tool should be responsible for this.

I want to reiterate the general lang policy that rustc is not intended to be robust against malicious Rust code. While we did adopt lints for cases like text direction codepoints (1 2), I don't think in general we should attempt to handle arbitrary malice in misleadingly presented code.

With that in mind, I would propose that we not attempt to detect bare CRs at this time. We can, at a later time, consider whether there are specific cases we feel need lints (not hard errors) in rustc; however, at this time I don't think there are likely to be any such cases.

So, my proposal is that we close #149823 and resolve the concern do-we-want-to-allow-carriage-returns with no further changes to the current state in rustc.

Minor: Frontmatter stripping in item context

There's a concern regarding frontmatter (and shebang) stripping for include! in item context. Based on previous discussion, I think we already have a lang team consensus on the following, but we should confirm explicitly: We want to remove frontmatter and shebang stripping for include! in item context, and request a PR fixing include!, a crater run (even though we expect it to show nothing), and the obvious reference PR (deleting the line - Shebang removal when invoked in an item context (as opposed to expression or statement contexts).).

Solving this would address the concerns document-frontmatter-removal and jack-should-we-strip-frontmatter-in-item-context.

[traviscross]

Bumped. It was awaiting a comment describing the ask (such as yours there) to make it more actionable.

[joshtriplett]

Bumped. It was awaiting a comment describing the ask (such as yours there) to make it more actionable.

If that is what you were waiting on, I think that needs clearer signposting so folks know what would unblock something.

[traviscross]

...disallowing CRs will create potential conflicts with whatever tool parses the frontmatter: if the tool emits warnings or errors for bare CRs, there would be duplicate warnings from the tool and from rustc.

I don't really follow. If rustc doesn't allow bare CRs and the tool bothers to check for this, the tool can simply abort before invoking rustc, right? Perhaps you could explain further why this is a problem.

I want to reiterate the general lang policy that rustc is not intended to be robust against malicious Rust code. While we did adopt lints for cases like text direction codepoints (1 2), I don't think in general we should attempt to handle arbitrary malice in misleadingly presented code.

The (approximate) grammar for raw string literals is:

RAW_STRING_LITERAL -> `r` RAW_STRING_CONTENT SUFFIX?

RAW_STRING_CONTENT ->
      `"` ( ~CR )*? `"`
    | `#` RAW_STRING_CONTENT `#`

If we develop a consensus that this should change to CHAR*? rather than ( ~CR )*?, then I'd withdraw the concern.

Without that, though, my worry remains that 1) whatever reasons we went with ( ~CR )*? for this likely apply to frontmatter as well and that 2) it would seem unnecessarily inconsistent for the grammar to diverge in this way between our two otherwise-similar fenced constructs. Just as we could say that rustc doesn't much itself care about the contents of a frontmatter section, we could say that it doesn't much care about the contents of a raw string literal either.

For my part, I think of what we're doing here as defining the *.rs file format. In some circumstances, maybe there's a tool that's processing the file first. But in other circumstances there won't be. Tool or no tool, we still have to fully define how a *.rs file is to be lexed and parsed.

Absent a consensus that we're going to move to CHAR*? for raw string literals, the conservative thing to do, in my view, is to save the space here on frontmatter by using the ( ~CR )*? grammar, and that's what I'd (re)propose to do so as to move forward frontmatter. We could always relax this later.

[traviscross]

Based on previous discussion, I think we already have a lang team consensus on the following, but we should confirm explicitly: We want to remove frontmatter and shebang stripping for include! in item context...

I do not believe we had a team consensus on this. Both @tmandry and I had raised questions about why we'd want to do this given that it could only cause fewer programs to compile. Other languages commonly handle shebangs gracefully on their similar constructs, probably (i'd imagine) for this reason.

(I.e., I don't think we finished the discussion. I don't know where we'd all fall on this after finishing it.)

[joshtriplett]

I don't really follow. If rustc doesn't allow bare CRs and the tool bothers to check for this, the tool can simply abort before invoking rustc, right? Perhaps you could explain further why this is a problem.

That's for the case of hard-error; consider the case where it's a warning or lint, instead. Tool warns, tool invokes rustc, rustc warns/denies.

---

Re raw strings:

If we develop a consensus that this should change to CHAR*? rather than ( ~CR )*?, then I'd withdraw the concern.

I would, broadly, argue that this should be a deny-by-default lint rather than a hard error; I think it's inconsistent for CR to be a hard error and RTL override characters to be a deny-by-default lint.

That said, to be explicit: I would at this time check a box for any of the three paths that allows us to make a definitive statement and forward progress towards stabilization: 1) defer for now, 2) hard error for now, or 3) deny-by-default lint. My preference would be 3 over 1 and 1 over 2. But what I don't want to have happen is that we fail to reach a consensus and thus fail to stabilize and ship this in a timely fashion.

---

Based on previous discussion, I think we already have a lang team consensus on the following, but we should confirm explicitly: We want to remove frontmatter and shebang stripping for include! in item context...

I do not believe we had a team consensus on this. Both @tmandry and I had raised questions about why we'd want to do this given that it could only cause fewer programs to compile. Other languages commonly handle shebangs gracefully on their similar constructs, probably (i'd imagine) for this reason.

(To be clear, I'm only saying that I don't believe we finished this discussion, not that any of us have a firm position on the question.)

I had thought we had, but apparently not; we should finish that discussion, then.

I agree that it will exclusively cause fewer programs to compile. I would hypothesize that there are essentially zero such programs currently, given that shebangs on Rust are rare (in part because we haven't shipped cargo-script yet) and frontmatter is not yet stable. It also simplifies the language and the complexity of include!, to be equivalent to a hypothetical parse_as_rust!(include_bytes!(...)). It's also trivial to work around if you actually want the stripping, without putting the burden on include!.

That said, as with the above: I would check a box for any of "don't strip" (preferred), "strip both in item context for now and decide later", "always strip both in item context", or even "only strip shebangs and not frontmatter" (least preferred), as long as we go ahead and make a definitive decision. What I'd like to avoid is blocking frontmatter further on a decision about changing shebang stripping, which AFAICT we've done here out of regard for a combination of consistency and seemingly not wanting to add and subsequently remove frontmatter stripping.

[traviscross]

Bumped. It was awaiting a comment describing the ask (such as yours there) to make it more actionable.

If that is what you were waiting on, I think that needs clearer signposting so folks know what would unblock something.

It wasn't clear this was blocked on us having a discussion.

We had earlier discussed it and developed a consensus for how to resolve the CR concern. Nobody, as far as I can tell, reported back here that it had been either resolved (in the way we had asked) or raised here a request for us to reconsider our consensus (until your comment).

Regarding @jackh726's concern, I didn't see any movement here on that or a particular ask for us on it either (again until your comment).

Our nomination queue is rather deep, and that strains not just lang but also ops. It's my view that it's as much the role of champions as anyone to notice when an item that should be discussed with lang does not have any clear ask of that on the relevant PR or issue.

[tmandry]

Without that, though, my worry remains that 1) whatever reasons we went with ( ~CR )*? for this likely apply to frontmatter as well and that 2) it would seem unnecessarily inconsistent for the grammar to diverge in this way between our two otherwise-similar fenced constructs. Just as we could say that rustc doesn't much itself care about the contents of a frontmatter section, we could say that it doesn't much care about the contents of a raw string literal either.

I don't think the concerns apply equally well to frontmatter. CR is an invisible character and including one in a string meaningfully changes the behavior of a program. Whereas for frontmatter, it's usually "just code" which is not inherently sensitive to more whitespace. Philosophically I agree that the concern is within the domain of the tool parsing the frontmatter.

As I wrote above, I'm ok with starting out with a more restrictive version since that buys us something (a simpler reference) and we can always relax it later. But I still don't consider it "correct".

[traviscross]

I don't think the concerns apply equally well to frontmatter. CR is an invisible character and including one in a string meaningfully changes the behavior of a program. Whereas for frontmatter, it's usually "just code" which is not inherently sensitive to more whitespace. Philosophically I agree that the concern is within the domain of the tool parsing the frontmatter.

This is perhaps a philosophical point, but I still think these are roughly the same. A CR in a string doesn't necessarily change the behavior of the program -- that's up to the program -- rustc has no idea whether it will or not.¹ Similarly, rustc has no idea whether or not a bare CR in the frontmatter will have any effect on a tool. It's again up to something other than rustc.

But in making my main point -- consistency of the grammar in our file format -- it's not actually even rustc that I'm thinking about. I'm thinking about r-a, rustfmt, tree-sitter, the formal grammar tool @ehuss is writing, etc. Tools other than rustc need to be able to fully lex and parse a Rust file -- sometimes with the expectation that they'll be able to reserialize the file roughly as they found it -- and to these tools, frontmatter and raw string literals look very much alike.

Footnotes

1. The raw string literal could equally well be "just code" too, e.g. a SQL block. ↩

[tmandry]

To be more precise, the CR changes the value of a raw string literal. While rustc doesn't know if this affects program behavior, it's very likely that it does, and it's within rustc's scope to check the correctness of programs.

Finding issues in frontmatter is not within rustc's scope. Your point about simplifying tools is taken, but for many tools this might be one of the places where it's better to accept a superset of Rust.

---

With all that said, I agree with Josh that we should accept CR with a deny-by-default lint instead of building it into the grammar. If we can all agree on that being the eventual goal, perhaps we should shift the discussion to talking about the best way of getting there.

[traviscross]

With all that said, I agree with Josh that we should accept CR with a deny-by-default lint instead of building it into the grammar. If we can all agree on that being the eventual goal, perhaps we should shift the discussion to talking about the best way of getting there.

It has occurred to me that maybe we should go the other way. Maybe there's not any good reason to allow CRs that survive CRLF normalization in a Rust file, period.

E.g., after CRLF normalization, the byte string CR CR LF becomes CR LF (!). Arguably, at that point, what we have produced is file corruption. It's difficult to produce examples of Rust programs that can observe this weirdness today because we exclude CRs in so many places in our grammar. If we were to go in the direction of allowing these CRs, this weirdness -- that's a peculiar artifact of how we do CRLF normalization -- would be readily observable.