Fix double handling in `collect_tokens` by nnethercote · Pull Request #129346 · rust-lang/rust

nnethercote · 2024-08-21T06:41:42Z

Double handling of AST nodes can occur in collect_tokens. This is when an inner call to collect_tokens produces an AST node, and then an outer call to collect_tokens produces the same AST node. This can happen in a few places, e.g. expression statements where the statement delegates HasTokens and HasAttrs to the expression. It will also happen more after #124141.

This PR fixes some double handling cases that cause problems, including #129166.

r? @petrochenkov

nnethercote · 2024-08-21T06:43:20Z

The last three commits are tricky and hard to review. Even after multiple weeks of working with this code, I still find it hard to understand and reason about, and I wish it could be made simpler. But the good news is that these commits are enough to the get final failing test (tests/ui/proc-macros/macro-rules-derive-cfg.rs) working in #124141.

nnethercote · 2024-08-21T07:46:09Z

I think this should get a crater run, just because I don't fully trust that the test suite is thoroughly testing cfg_attr stuff.

And why not do a perf run, just to check there's no regressions.

@bors try @rust-timer queue

bors · 2024-08-21T07:47:21Z

⌛ Trying commit 7d65809 with merge 8b22b49...

…ollect_tokens, r=<try> Fix double handling in `collect_tokens` Double handling of AST nodes can occur in `collect_tokens`. This is when an inner call to `collect_tokens` produces an AST node, and then an outer call to `collect_tokens` produces the same AST node. This can happen in a few places, e.g. expression statements where the statement delegates `HasTokens` and `HasAttrs` to the expression. It will also happen more after rust-lang#124141. This PR fixes some double handling cases that cause problems, including rust-lang#129166. r? `@petrochenkov`

bors · 2024-08-21T09:41:52Z

☀️ Try build successful - checks-actions
Build commit: 8b22b49 (8b22b49ea046aed5d405c5bda40972c039aa70b5)

rust-timer · 2024-08-21T11:46:32Z

Finished benchmarking commit (8b22b49): comparison URL.

Overall result: ❌ regressions - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.3%	[0.2%, 0.7%]	43
Regressions ❌ (secondary)	0.4%	[0.1%, 0.6%]	16
Improvements ✅ (primary)	-0.3%	[-0.3%, -0.3%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.3%	[-0.3%, 0.7%]	44

Max RSS (memory usage)

Results (secondary -2.7%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.7%	[-3.4%, -2.0%]	2
All ❌✅ (primary)	-	-	0

Cycles

Results (secondary -2.3%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.3%	[-2.3%, -2.3%]	1
All ❌✅ (primary)	-	-	0

Binary size

Results (secondary 0.0%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.0%	[0.0%, 0.0%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Bootstrap: 748.876s -> 749.555s (0.09%)
Artifact size: 338.91 MiB -> 338.99 MiB (0.02%)

petrochenkov · 2024-08-21T15:36:44Z

Which change is a source of the perf regression, 7d65809 probably?
All the others look like optimizations.

petrochenkov · 2024-08-21T15:38:05Z

I'll start crater once #129346 (comment) is addressed and CI is green.
@rustbot author

nnethercote · 2024-08-22T07:04:55Z

I have adding a commit that avoids cloning ret.attrs() except when necessary. This gets back part of the perf regression. The rest is due to Parser::clone being more expensive because of the new FxHashSet in CaptureState. I tried changing it to a GrowableBitSet which seemed like it should have helped but didn't.

Let's see what the new status is:
@bors try @rust-time queue

…ollect_tokens, r=<try> Fix double handling in `collect_tokens` Double handling of AST nodes can occur in `collect_tokens`. This is when an inner call to `collect_tokens` produces an AST node, and then an outer call to `collect_tokens` produces the same AST node. This can happen in a few places, e.g. expression statements where the statement delegates `HasTokens` and `HasAttrs` to the expression. It will also happen more after rust-lang#124141. This PR fixes some double handling cases that cause problems, including rust-lang#129166. r? `@petrochenkov`

bors · 2024-08-22T07:06:07Z

⌛ Trying commit cab3178 with merge a70947b...

bors · 2024-08-22T09:05:06Z

☀️ Try build successful - checks-actions
Build commit: a70947b (a70947b8062a88e6f76266f2540de6ea6c2bfba5)

lqd · 2024-08-22T09:22:07Z

@rust-timer build a70947b

rust-timer · 2024-08-22T10:52:15Z

Finished benchmarking commit (a70947b): comparison URL.

Overall result: ❌ regressions - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.3%	[0.2%, 0.7%]	25
Regressions ❌ (secondary)	0.4%	[0.2%, 0.6%]	17
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.3%	[0.2%, 0.7%]	25

Max RSS (memory usage)

Results (primary -0.8%, secondary 2.4%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.9%	[0.5%, 1.3%]	2
Regressions ❌ (secondary)	2.4%	[2.1%, 2.7%]	2
Improvements ✅ (primary)	-4.3%	[-4.3%, -4.3%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-0.8%	[-4.3%, 1.3%]	3

Cycles

Results (secondary 2.3%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.3%	[2.3%, 2.3%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Binary size

Results (secondary -0.1%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.1%	[-0.1%, -0.1%]	1
All ❌✅ (primary)	-	-	0

Bootstrap: 750.761s -> 751.313s (0.07%)
Artifact size: 338.92 MiB -> 339.03 MiB (0.03%)

rust-timer · 2024-09-08T09:17:59Z

Finished benchmarking commit (6d05f12): comparison URL.

Overall result: ❌✅ regressions and improvements - no action needed

@rustbot label: -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.3%	[0.3%, 0.3%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.4%	[-0.4%, -0.3%]	2
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

Results (primary -1.9%, secondary 1.9%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.6%	[2.6%, 2.6%]	1
Regressions ❌ (secondary)	3.0%	[2.3%, 4.2%]	7
Improvements ✅ (primary)	-3.4%	[-3.5%, -3.4%]	3
Improvements ✅ (secondary)	-0.7%	[-0.7%, -0.6%]	3
All ❌✅ (primary)	-1.9%	[-3.5%, 2.6%]	4

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 754.658s -> 756.515s (0.25%)
Artifact size: 341.04 MiB -> 341.16 MiB (0.04%)

danielhenrymantilla · 2024-10-18T10:47:43Z

FWIW, this did regress the following test: https://docs.rs/crate/cfg_eval/0.1.2/source/tests/item.rs

https://github.com/danielhenrymantilla/cfg_eval.rs/actions/workflows/future-proof.yml

I don't know why crater did not see it 😔

danielhenrymantilla · 2024-10-18T10:50:33Z

- struct Foo {
- val: [bool; {
- let a = #[cfg_attr(not(FALSE), rustc_dummy(first))] $expr;
- 0
- }]
- }


This case stopped working?

Fixed in #132587.

Noratrieb · 2024-10-18T10:51:51Z

even if crater found your crate, it was only a check run

It caused a test regression in the `cfg_eval.rs` crate. (The bugfix in rust-lang#129346 was in a different commit; this commit was just a code simplification.)

lqd · 2024-11-04T11:46:40Z

even if crater found your crate, it was only a check run

and since beta runs are test runs, the regression is indeed present in the 1.83 unexpected test failures: https://crater-reports.s3.amazonaws.com/beta-1.83-1/beta-2024-10-19/gh/danielhenrymantilla.cfg_eval.rs/log.txt

…placement-ranges, r=petrochenkov Revert "Avoid nested replacement ranges" from rust-lang#129346. It caused a test regression in the `cfg_eval.rs` crate. (The bugfix in rust-lang#129346 was in a different commit; this commit was just a code simplification.) r? `@petrochenkov`

…kingjubilee Rollup of 10 pull requests Successful merges: - rust-lang#129884 (mark some target features as 'forbidden' so they cannot be (un)set with -Ctarget-feature) - rust-lang#132153 (Stabilise `const_char_encode_utf16`.) - rust-lang#132473 ([core/fmt] Replace checked slice indexing by unchecked to support panic-free code) - rust-lang#132571 (add const_eval_select macro to reduce redundancy) - rust-lang#132587 (Revert "Avoid nested replacement ranges" from rust-lang#129346.) - rust-lang#132596 ([rustdoc] Fix `--show-coverage` when JSON output format is used) - rust-lang#132598 (Clippy: Move some attribute lints to be early pass (post expansion)) - rust-lang#132601 (Update books) - rust-lang#132606 (Improve example of `impl Pattern for &[char]`) - rust-lang#132609 (docs: fix grammar in doc comment at unix/process.rs) r? `@ghost` `@rustbot` modify labels: rollup

…placement-ranges, r=petrochenkov Revert "Avoid nested replacement ranges" from rust-lang#129346. It caused a test regression in the `cfg_eval.rs` crate. (The bugfix in rust-lang#129346 was in a different commit; this commit was just a code simplification.) r? `@petrochenkov`

…kingjubilee Rollup of 11 pull requests Successful merges: - rust-lang#131153 (Improve duplicate derive Copy/Clone diagnostics) - rust-lang#132025 (fix suggestion for diagnostic error E0027) - rust-lang#132303 (More tests for non-exhaustive C-like enums in FFI) - rust-lang#132492 (remove support for extern-block const intrinsics) - rust-lang#132587 (Revert "Avoid nested replacement ranges" from rust-lang#129346.) - rust-lang#132596 ([rustdoc] Fix `--show-coverage` when JSON output format is used) - rust-lang#132598 (Clippy: Move some attribute lints to be early pass (post expansion)) - rust-lang#132601 (Update books) - rust-lang#132606 (Improve example of `impl Pattern for &[char]`) - rust-lang#132608 (document `type_implements_trait`) - rust-lang#132609 (docs: fix grammar in doc comment at unix/process.rs) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of rust-lang#132587 - nnethercote:revert-avoid-nested-replacement-ranges, r=petrochenkov Revert "Avoid nested replacement ranges" from rust-lang#129346. It caused a test regression in the `cfg_eval.rs` crate. (The bugfix in rust-lang#129346 was in a different commit; this commit was just a code simplification.) r? `@petrochenkov`

It caused a test regression in the `cfg_eval.rs` crate. (The bugfix in rust-lang#129346 was in a different commit; this commit was just a code simplification.) (cherry picked from commit 981dc02)

[beta] backports - rustdoc: skip stability inheritance for some item kinds rust-lang#132481 - Avoid use imports in thread_local_inner! in static rust-lang#132101 - Also treat `impl` definition parent as transparent regarding modules rust-lang#132453 - Revert "Avoid nested replacement ranges" from rust-lang#129346. rust-lang#132587 r? cuviper

…s-test, r=nnethercote Add regression test for nested replacement ranges in `collect_tokens` rust-lang#129346 removed support for nested replacement ranges in `collect_tokens`, and simplified the nesting out of `tests/ui/proc-macro/macro-rules-derive-cfg.rs` accordingly. That removal turned out to break real-world code, so it was reverted in rust-lang#132587 — but the revert only restored the compiler code, not the original nested shape of the test, leaving the nested case uncovered ever since. This reinstates the nested case as a separate test: the pre-rust-lang/rust#129346 shape of `macro-rules-derive-cfg.rs` (a `cfg_attr` expansion nested inside a `let` statement, nested inside an anonymous constant), combined with the extra inner attributes that rust-lang#129346 added to the simplified version. If nested replacement range support regresses again, this test fails. The expected stdout was verified against current nightly; the sibling test's checked-in stdout reproduces byte-for-byte under the same setup. Closesrust-lang#132727

Rollup merge of #159529 - UditDewan:nested-replacement-ranges-test, r=nnethercote Add regression test for nested replacement ranges in `collect_tokens` #129346 removed support for nested replacement ranges in `collect_tokens`, and simplified the nesting out of `tests/ui/proc-macro/macro-rules-derive-cfg.rs` accordingly. That removal turned out to break real-world code, so it was reverted in #132587 — but the revert only restored the compiler code, not the original nested shape of the test, leaving the nested case uncovered ever since. This reinstates the nested case as a separate test: the pre-rust-lang/rust#129346 shape of `macro-rules-derive-cfg.rs` (a `cfg_attr` expansion nested inside a `let` statement, nested inside an anonymous constant), combined with the extra inner attributes that #129346 added to the simplified version. If nested replacement range support regresses again, this test fails. The expected stdout was verified against current nightly; the sibling test's checked-in stdout reproduces byte-for-byte under the same setup. Closes#132727

…=nnethercote Add regression test for nested replacement ranges in `collect_tokens` rust-lang/rust#129346 removed support for nested replacement ranges in `collect_tokens`, and simplified the nesting out of `tests/ui/proc-macro/macro-rules-derive-cfg.rs` accordingly. That removal turned out to break real-world code, so it was reverted in rust-lang/rust#132587 — but the revert only restored the compiler code, not the original nested shape of the test, leaving the nested case uncovered ever since. This reinstates the nested case as a separate test: the pre-rust-lang/rust#129346 shape of `macro-rules-derive-cfg.rs` (a `cfg_attr` expansion nested inside a `let` statement, nested inside an anonymous constant), combined with the extra inner attributes that rust-lang/rust#129346 added to the simplified version. If nested replacement range support regresses again, this test fails. The expected stdout was verified against current nightly; the sibling test's checked-in stdout reproduces byte-for-byte under the same setup. Closesrust-lang/rust#132727

rustbot assigned petrochenkov Aug 21, 2024

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Aug 21, 2024

This comment has been minimized.
Sign in to view

nnethercote force-pushed the fix-double-handling-in-collect_tokens branch from a0f9bfd to 7d65809Compare August 21, 2024 07:44

This comment has been minimized.
Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Aug 21, 2024

This comment has been minimized.
Sign in to view

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Aug 21, 2024

petrochenkov reviewed Aug 21, 2024
View reviewed changes

Comment threadtests/ui/conditional-compilation/invalid-node-range-issue-129166.rs

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 21, 2024

nnethercote force-pushed the fix-double-handling-in-collect_tokens branch from 7d65809 to aeaa62bCompare August 21, 2024 22:54

This comment has been minimized.
Sign in to view

rustbot added this to the 1.83.0 milestone Sep 8, 2024

bors mentioned this pull request Sep 8, 2024
interpret: make typed copies lossy wrt provenance and padding #129778
Merged

rustbot removed the perf-regression Performance regression. label Sep 8, 2024

nnethercote deleted the fix-double-handling-in-collect_tokens branch September 8, 2024 22:13

danielhenrymantilla reviewed Oct 18, 2024
View reviewed changes

nnethercote mentioned this pull request Nov 4, 2024
Revert "Avoid nested replacement ranges" from #129346. #132587
Merged

workingjubilee mentioned this pull request Nov 4, 2024
Rollup of 10 pull requests #132616
Closed

workingjubilee mentioned this pull request Nov 5, 2024
Rollup of 11 pull requests #132626
Merged

jieyouxu mentioned this pull request Nov 7, 2024
Add a regression test for #132587 #132727
Closed

cuviper mentioned this pull request Nov 7, 2024
[beta] backports #132733
Merged

UditDewan mentioned this pull request Jul 18, 2026
Add regression test for nested replacement ranges in collect_tokens#159529
Merged

cyrgani mentioned this pull request Aug 18, 2026
ICE: attempt to subtract with overflow#127440
Closed

Uh oh!

Uh oh!

Conversation

nnethercote commented Aug 21, 2024

Uh oh!

nnethercote commented Aug 21, 2024

Uh oh!

This comment has been minimized.

nnethercote commented Aug 21, 2024

Uh oh!

This comment has been minimized.

bors commented Aug 21, 2024

Uh oh!

This comment has been minimized.

bors commented Aug 21, 2024

Uh oh!

This comment has been minimized.

rust-timer commented Aug 21, 2024

Overall result: ❌ regressions - ACTION NEEDED

Uh oh!

Uh oh!

petrochenkov commented Aug 21, 2024

Uh oh!

petrochenkov commented Aug 21, 2024

Uh oh!

nnethercote commented Aug 22, 2024

Uh oh!

bors commented Aug 22, 2024

Uh oh!

bors commented Aug 22, 2024

Uh oh!

lqd commented Aug 22, 2024

Uh oh!

This comment has been minimized.

rust-timer commented Aug 22, 2024

Overall result: ❌ regressions - ACTION NEEDED

Uh oh!

rust-timer commented Sep 8, 2024

Overall result: ❌✅ regressions and improvements - no action needed

Uh oh!

danielhenrymantilla commented Oct 18, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Noratrieb commented Oct 18, 2024

Uh oh!

lqd commented Nov 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants