Fix double handling in `collect_tokens` #129346

nnethercote · 2024-08-21T06:41:42Z

Double handling of AST nodes can occur in collect_tokens. This is when an inner call to collect_tokens produces an AST node, and then an outer call to collect_tokens produces the same AST node. This can happen in a few places, e.g. expression statements where the statement delegates HasTokens and HasAttrs to the expression. It will also happen more after #124141.

This PR fixes some double handling cases that cause problems, including #129166.

r? @petrochenkov

nnethercote · 2024-08-21T06:43:20Z

The last three commits are tricky and hard to review. Even after multiple weeks of working with this code, I still find it hard to understand and reason about, and I wish it could be made simpler. But the good news is that these commits are enough to the get final failing test (tests/ui/proc-macros/macro-rules-derive-cfg.rs) working in #124141.

nnethercote · 2024-08-21T07:46:09Z

I think this should get a crater run, just because I don't fully trust that the test suite is thoroughly testing cfg_attr stuff.

And why not do a perf run, just to check there's no regressions.

@bors try @rust-timer queue

bors · 2024-08-21T07:47:21Z

⌛ Trying commit 7d65809 with merge 8b22b49...

…ollect_tokens, r=<try> Fix double handling in `collect_tokens` Double handling of AST nodes can occur in `collect_tokens`. This is when an inner call to `collect_tokens` produces an AST node, and then an outer call to `collect_tokens` produces the same AST node. This can happen in a few places, e.g. expression statements where the statement delegates `HasTokens` and `HasAttrs` to the expression. It will also happen more after rust-lang#124141. This PR fixes some double handling cases that cause problems, including rust-lang#129166. r? `@petrochenkov`

bors · 2024-08-21T09:41:52Z

☀️ Try build successful - checks-actions
Build commit: 8b22b49 (8b22b49ea046aed5d405c5bda40972c039aa70b5)

rust-timer · 2024-08-21T11:46:32Z

Finished benchmarking commit (8b22b49): comparison URL.

Overall result: ❌ regressions - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.3%	[0.2%, 0.7%]	43
Regressions ❌ (secondary)	0.4%	[0.1%, 0.6%]	16
Improvements ✅ (primary)	-0.3%	[-0.3%, -0.3%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.3%	[-0.3%, 0.7%]	44

Max RSS (memory usage)

Results (secondary -2.7%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.7%	[-3.4%, -2.0%]	2
All ❌✅ (primary)	-	-	0

Cycles

Results (secondary -2.3%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.3%	[-2.3%, -2.3%]	1
All ❌✅ (primary)	-	-	0

Binary size

Results (secondary 0.0%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.0%	[0.0%, 0.0%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Bootstrap: 748.876s -> 749.555s (0.09%)
Artifact size: 338.91 MiB -> 338.99 MiB (0.02%)

tests/ui/conditional-compilation/invalid-node-range-issue-129166.rs

petrochenkov · 2024-08-21T15:36:44Z

Which change is a source of the perf regression, 7d65809 probably?
All the others look like optimizations.

petrochenkov · 2024-08-21T15:38:05Z

I'll start crater once #129346 (comment) is addressed and CI is green.
@rustbot author

nnethercote · 2024-08-22T07:04:55Z

I have adding a commit that avoids cloning ret.attrs() except when necessary. This gets back part of the perf regression. The rest is due to Parser::clone being more expensive because of the new FxHashSet in CaptureState. I tried changing it to a GrowableBitSet which seemed like it should have helped but didn't.

Let's see what the new status is:
@bors try @rust-time queue

…ollect_tokens, r=<try> Fix double handling in `collect_tokens` Double handling of AST nodes can occur in `collect_tokens`. This is when an inner call to `collect_tokens` produces an AST node, and then an outer call to `collect_tokens` produces the same AST node. This can happen in a few places, e.g. expression statements where the statement delegates `HasTokens` and `HasAttrs` to the expression. It will also happen more after rust-lang#124141. This PR fixes some double handling cases that cause problems, including rust-lang#129166. r? `@petrochenkov`

bors · 2024-08-22T07:06:07Z

⌛ Trying commit cab3178 with merge a70947b...

bors · 2024-08-22T09:05:06Z

☀️ Try build successful - checks-actions
Build commit: a70947b (a70947b8062a88e6f76266f2540de6ea6c2bfba5)

lqd · 2024-08-22T09:22:07Z

@rust-timer build a70947b

rust-timer · 2024-08-22T10:52:15Z

Finished benchmarking commit (a70947b): comparison URL.

Overall result: ❌ regressions - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.3%	[0.2%, 0.7%]	25
Regressions ❌ (secondary)	0.4%	[0.2%, 0.6%]	17
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.3%	[0.2%, 0.7%]	25

Max RSS (memory usage)

Results (primary -0.8%, secondary 2.4%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.9%	[0.5%, 1.3%]	2
Regressions ❌ (secondary)	2.4%	[2.1%, 2.7%]	2
Improvements ✅ (primary)	-4.3%	[-4.3%, -4.3%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-0.8%	[-4.3%, 1.3%]	3

Cycles

Results (secondary 2.3%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.3%	[2.3%, 2.3%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Binary size

Results (secondary -0.1%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.1%	[-0.1%, -0.1%]	1
All ❌✅ (primary)	-	-	0

Bootstrap: 750.761s -> 751.313s (0.07%)
Artifact size: 338.92 MiB -> 339.03 MiB (0.03%)

nnethercote · 2024-08-23T21:01:19Z

Ok, IntervalSet gives similar results to RangeMap, which is great! There is still a very small perf regression but it's less than it was, and the fix is necessary for correctness both now and even more when #124141 gets merged. So I think this is good enough for a crater run.

@bors try

bors · 2024-08-23T21:02:30Z

⌛ Trying commit d4bf28c with merge 96ad39d...

…ollect_tokens, r=<try> Fix double handling in `collect_tokens` Double handling of AST nodes can occur in `collect_tokens`. This is when an inner call to `collect_tokens` produces an AST node, and then an outer call to `collect_tokens` produces the same AST node. This can happen in a few places, e.g. expression statements where the statement delegates `HasTokens` and `HasAttrs` to the expression. It will also happen more after rust-lang#124141. This PR fixes some double handling cases that cause problems, including rust-lang#129166. r? `@petrochenkov`

bors · 2024-08-23T23:01:06Z

☀️ Try build successful - checks-actions
Build commit: 96ad39d (96ad39d809ea5a132d223acdaa01b9a8d3a26348)

nnethercote · 2024-08-23T23:35:29Z

@craterbot check

craterbot · 2024-08-23T23:35:36Z

👌 Experiment pr-129346 created and queued.
🤖 Automatically detected try build 96ad39d
🔍 You can check out the queue and this experiment's details.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

nnethercote · 2024-08-24T03:41:22Z

The reason that IntervalSet works well is that multi-line /// doc comments get split into one attribute per line. So it's common to have lots of attributes on an AST node, but those attributes all have consecutive AttrIds. Much nicer to have a single range like 50..100 instead of 50 individual entries.

I really wish that multiple /// doc comment lines instead collapsed into a single attribute. This would make it easier to give good perf inside the compiler. I suspect it's not possible to change this because of proc macros.

craterbot · 2024-09-04T01:07:51Z

🚧 Experiment pr-129346 is now running

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

craterbot · 2024-09-06T10:01:53Z

🎉 Experiment pr-129346 is completed!
📊 6 regressed and 3 fixed (504686 total)
📰 Open the full report.

⚠️ If you notice any spurious failure please add them to the blacklist!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

nnethercote · 2024-09-06T10:17:06Z

No genuine regressions in the crater run. @petrochenkov, are you happy for it to merge?

petrochenkov · 2024-09-07T10:18:25Z

@bors r+

bors · 2024-09-07T10:18:28Z

📌 Commit d4bf28c has been approved by petrochenkov

It is now in the queue for this repository.

bors · 2024-09-08T05:35:27Z

⌛ Testing commit d4bf28c with merge 6d05f12...

bors · 2024-09-08T08:03:50Z

☀️ Test successful - checks-actions
Approved by: petrochenkov
Pushing 6d05f12 to master...

rust-timer · 2024-09-08T09:17:59Z

Finished benchmarking commit (6d05f12): comparison URL.

Overall result: ❌✅ regressions and improvements - no action needed

@rustbot label: -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.3%	[0.3%, 0.3%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.4%	[-0.4%, -0.3%]	2
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

Results (primary -1.9%, secondary 1.9%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.6%	[2.6%, 2.6%]	1
Regressions ❌ (secondary)	3.0%	[2.3%, 4.2%]	7
Improvements ✅ (primary)	-3.4%	[-3.5%, -3.4%]	3
Improvements ✅ (secondary)	-0.7%	[-0.7%, -0.6%]	3
All ❌✅ (primary)	-1.9%	[-3.5%, 2.6%]	4

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 754.658s -> 756.515s (0.25%)
Artifact size: 341.04 MiB -> 341.16 MiB (0.04%)

danielhenrymantilla · 2024-10-18T10:47:43Z

FWIW, this did regress the following test: https://docs.rs/crate/cfg_eval/0.1.2/source/tests/item.rs

https://github.com/danielhenrymantilla/cfg_eval.rs/actions/workflows/future-proof.yml

I don't know why crater did not see it 😔

danielhenrymantilla · 2024-10-18T10:50:33Z

tests/ui/proc-macro/macro-rules-derive-cfg.rs

-        struct Foo {
-            val: [bool; {
-                let a = #[cfg_attr(not(FALSE), rustc_dummy(first))] $expr;
-                0
-            }]
-        }


This case stopped working?

Noratrieb · 2024-10-18T10:51:51Z

even if crater found your crate, it was only a check run

rustbot assigned petrochenkov Aug 21, 2024

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Aug 21, 2024

This comment has been minimized.

Sign in to view

nnethercote force-pushed the fix-double-handling-in-collect_tokens branch from a0f9bfd to 7d65809 Compare August 21, 2024 07:44

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Aug 21, 2024

This comment has been minimized.

Sign in to view

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Aug 21, 2024

petrochenkov reviewed Aug 21, 2024

View reviewed changes

tests/ui/conditional-compilation/invalid-node-range-issue-129166.rs Show resolved Hide resolved

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 21, 2024

nnethercote force-pushed the fix-double-handling-in-collect_tokens branch from 7d65809 to aeaa62b Compare August 21, 2024 22:54

This comment has been minimized.

Sign in to view

craterbot added S-waiting-on-crater Status: Waiting on a crater run to be completed. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Aug 23, 2024

craterbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-crater Status: Waiting on a crater run to be completed. labels Sep 6, 2024

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 7, 2024

bors added the merged-by-bors This PR was explicitly merged by bors. label Sep 8, 2024

bors merged commit 6d05f12 into rust-lang:master Sep 8, 2024
7 checks passed

rustbot added this to the 1.83.0 milestone Sep 8, 2024

bors mentioned this pull request Sep 8, 2024

interpret: make typed copies lossy wrt provenance and padding #129778

Merged

rustbot removed the perf-regression Performance regression. label Sep 8, 2024

nnethercote deleted the fix-double-handling-in-collect_tokens branch September 8, 2024 22:13

danielhenrymantilla reviewed Oct 18, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix double handling in `collect_tokens` #129346

Fix double handling in `collect_tokens` #129346

nnethercote commented Aug 21, 2024

nnethercote commented Aug 21, 2024

This comment has been minimized.

nnethercote commented Aug 21, 2024

This comment has been minimized.

bors commented Aug 21, 2024

This comment has been minimized.

bors commented Aug 21, 2024

This comment has been minimized.

rust-timer commented Aug 21, 2024

petrochenkov commented Aug 21, 2024

petrochenkov commented Aug 21, 2024

nnethercote commented Aug 22, 2024

bors commented Aug 22, 2024

bors commented Aug 22, 2024

lqd commented Aug 22, 2024

This comment has been minimized.

rust-timer commented Aug 22, 2024

nnethercote commented Aug 23, 2024

bors commented Aug 23, 2024

bors commented Aug 23, 2024

nnethercote commented Aug 23, 2024

craterbot commented Aug 23, 2024

nnethercote commented Aug 24, 2024

craterbot commented Sep 4, 2024

craterbot commented Sep 6, 2024

nnethercote commented Sep 6, 2024

petrochenkov commented Sep 7, 2024

bors commented Sep 7, 2024

bors commented Sep 8, 2024

bors commented Sep 8, 2024

rust-timer commented Sep 8, 2024

danielhenrymantilla commented Oct 18, 2024

danielhenrymantilla Oct 18, 2024

Noratrieb commented Oct 18, 2024

Fix double handling in collect_tokens #129346

Fix double handling in collect_tokens #129346

Conversation

nnethercote commented Aug 21, 2024

nnethercote commented Aug 21, 2024

This comment has been minimized.

nnethercote commented Aug 21, 2024

This comment has been minimized.

bors commented Aug 21, 2024

This comment has been minimized.

bors commented Aug 21, 2024

This comment has been minimized.

rust-timer commented Aug 21, 2024

Overall result: ❌ regressions - ACTION NEEDED

petrochenkov commented Aug 21, 2024

petrochenkov commented Aug 21, 2024

nnethercote commented Aug 22, 2024

bors commented Aug 22, 2024

bors commented Aug 22, 2024

lqd commented Aug 22, 2024

This comment has been minimized.

rust-timer commented Aug 22, 2024

Overall result: ❌ regressions - ACTION NEEDED

nnethercote commented Aug 23, 2024

bors commented Aug 23, 2024

bors commented Aug 23, 2024

nnethercote commented Aug 23, 2024

craterbot commented Aug 23, 2024

nnethercote commented Aug 24, 2024

craterbot commented Sep 4, 2024

craterbot commented Sep 6, 2024

nnethercote commented Sep 6, 2024

petrochenkov commented Sep 7, 2024

bors commented Sep 7, 2024

bors commented Sep 8, 2024

bors commented Sep 8, 2024

rust-timer commented Sep 8, 2024

Overall result: ❌✅ regressions and improvements - no action needed

danielhenrymantilla commented Oct 18, 2024

Choose a reason for hiding this comment

Noratrieb commented Oct 18, 2024

Fix double handling in `collect_tokens` #129346

Fix double handling in `collect_tokens` #129346