Construct `tokens` for attrs made by `mk_attr_word` and other variants by mu001999 · Pull Request #159277 · rust-lang/rust

mu001999 · 2026-07-14T09:02:58Z

tokens won't be constructed for generated built-in attrs before, so that after #158854, #[rustc_test_entrypoint_marker] without tokens would be added when expanding #[test]. Then later expansions would cause the ICE when they wanna get the attr's tokens.

This PR makes tokens constructed for such attrs, although this specific ICE only need editing mk_attr_word, but I think it would be good to also edit mk_attr_nested_word and mk_attr_name_value_str.

jdonszelmann · 2026-07-14T09:25:48Z

r? me

jdonszelmann

It's a big fix, but I think it actually addresses multiple longstanding issues. EIIs do the same, where they generate attributes, and I'm sure it was secretly vulnerable to the same ICE. Same for bench and global_allocator prolly too. I guess it's maybe even for the better we found out about this hehe, even though I guess I'm to blame for causing the regression.

Thanks @mu001999 !

r=me if you're ready @mu001999 , maybe undraft it, but I think this is a fine solution :3

View changes since this review

jdonszelmann · 2026-07-14T09:31:02Z

+ #[test_macros::identity_attr]
+ fn test() {}
+}
+


I love that you test it with macro_attr, it's a much more concise test like this

rust-bors · 2026-07-14T09:33:27Z

📋 Only open, non-draft PRs can be approved.

mu001999 · 2026-07-14T11:03:18Z

@bors r=jdonszelmann rollup

rust-bors · 2026-07-14T11:03:21Z

📌 Commit 29668b5 has been approved by jdonszelmann

It is now in the queue for this repository.

…nszelmann Construct `tokens` for attrs made by `mk_attr_word` and other variants Fixesrust-lang#159261 `tokens` won't be constructed for generated built-in attrs before, so that after rust-lang#158854, `#[rustc_test_entrypoint_marker]` without `tokens` would be added when expanding `#[test]`. Then later expansions would cause the ICE when they wanna get the attr's `tokens`. This PR makes `tokens` constructed for such attrs, although this specific ICE only need editing `mk_attr_word`, but I think it would be good to also edit `mk_attr_nested_word` and `mk_attr_name_value_str`.

…uwer Rollup of 22 pull requests Successful merges: - #156047 (Fix trait method resolution on an adjusted never type) - #157824 (Comptime inherent impls) - #158235 (Store `DefId` instead of `EiiDecl` in `EiiImplResolution::Known`) - #158723 (Support EII on Windows MSVC) - #158993 (rerun in original typing mode if we meet any opaques in post analysis) - #159160 (Make `HasTokens` a sub-trait of `HasAttrs`.) - #159183 (Introduce InstanceKind::LlvmIntrinsic) - #159251 (Bump rustc-perf submodule) - #155013 (Suggest the `[const] Destruct` bound for type parameters in const functions when missing) - #159155 (unstable book: Document `diagnostic_on_unknown` feature) - #159235 (Add regression test for #95719) - #159243 (inline Once wait and wait_force) - #159255 (Replace shortened type with `_` instead of `...` as placeholder) - #159259 (Add regression test for #144033) - #159265 (bootstrap: skip intrinsic-test when rustfmt is unavailable) - #159269 (disable range-len-try-from.rs on s390x) - #159272 (slice: make swap delegate to swap_unchecked) - #159274 (bootstrap: Rename `std_crates_for_run_make` to `std_crates_for_make_run`) - #159275 (remove obsolete comment) - #159277 (Construct `tokens` for attrs made by `mk_attr_word` and other variants) - #159283 (Remove obsolete verbose flag from deref/ref suggestions) - #159290 (rustc-dev-guide subtree update)

Fixes#7786. Rust `nightly-2026-07-14` introduced rust-lang/rust#159261, which makes `linux-build` deterministically ICE while expanding a `#[test]` followed by an attribute macro. Lance hits this in the `lance-encoding` test binary before any tests execute, and the same failure is present on `main`. Pin the job's toolchain installation and `cargo llvm-cov` invocation to the last known-good `nightly-2026-07-13`. This is a temporary CI mitigation; restore the floating nightly after a dated release containing rust-lang/rust#159277 passes the reproduction and the full job.

…uwer Rollup of 22 pull requests Successful merges: - #156047 (Fix trait method resolution on an adjusted never type) - #157824 (Comptime inherent impls) - #158235 (Store `DefId` instead of `EiiDecl` in `EiiImplResolution::Known`) - #158723 (Support EII on Windows MSVC) - #158993 (rerun in original typing mode if we meet any opaques in post analysis) - #159160 (Make `HasTokens` a sub-trait of `HasAttrs`.) - #159183 (Introduce InstanceKind::LlvmIntrinsic) - #159251 (Bump rustc-perf submodule) - #155013 (Suggest the `[const] Destruct` bound for type parameters in const functions when missing) - #159155 (unstable book: Document `diagnostic_on_unknown` feature) - #159235 (Add regression test for #95719) - #159243 (inline Once wait and wait_force) - #159255 (Replace shortened type with `_` instead of `...` as placeholder) - #159259 (Add regression test for #144033) - #159265 (bootstrap: skip intrinsic-test when rustfmt is unavailable) - #159269 (disable range-len-try-from.rs on s390x) - #159272 (slice: make swap delegate to swap_unchecked) - #159274 (bootstrap: Rename `std_crates_for_run_make` to `std_crates_for_make_run`) - #159275 (remove obsolete comment) - #159277 (Construct `tokens` for attrs made by `mk_attr_word` and other variants) - #159283 (Remove obsolete verbose flag from deref/ref suggestions) - #159290 (rustc-dev-guide subtree update)

Rollup merge of #159277 - mu001999-contrib:fix-159261, r=jdonszelmann Construct `tokens` for attrs made by `mk_attr_word` and other variants Fixes#159261 `tokens` won't be constructed for generated built-in attrs before, so that after #158854, `#[rustc_test_entrypoint_marker]` without `tokens` would be added when expanding `#[test]`. Then later expansions would cause the ICE when they wanna get the attr's `tokens`. This PR makes `tokens` constructed for such attrs, although this specific ICE only need editing `mk_attr_word`, but I think it would be good to also edit `mk_attr_nested_word` and `mk_attr_name_value_str`.

…uwer Rollup of 22 pull requests Successful merges: - rust-lang/rust#156047 (Fix trait method resolution on an adjusted never type) - rust-lang/rust#157824 (Comptime inherent impls) - rust-lang/rust#158235 (Store `DefId` instead of `EiiDecl` in `EiiImplResolution::Known`) - rust-lang/rust#158723 (Support EII on Windows MSVC) - rust-lang/rust#158993 (rerun in original typing mode if we meet any opaques in post analysis) - rust-lang/rust#159160 (Make `HasTokens` a sub-trait of `HasAttrs`.) - rust-lang/rust#159183 (Introduce InstanceKind::LlvmIntrinsic) - rust-lang/rust#159251 (Bump rustc-perf submodule) - rust-lang/rust#155013 (Suggest the `[const] Destruct` bound for type parameters in const functions when missing) - rust-lang/rust#159155 (unstable book: Document `diagnostic_on_unknown` feature) - rust-lang/rust#159235 (Add regression test for rust-lang/rust#95719) - rust-lang/rust#159243 (inline Once wait and wait_force) - rust-lang/rust#159255 (Replace shortened type with `_` instead of `...` as placeholder) - rust-lang/rust#159259 (Add regression test for rust-lang/rust#144033) - rust-lang/rust#159265 (bootstrap: skip intrinsic-test when rustfmt is unavailable) - rust-lang/rust#159269 (disable range-len-try-from.rs on s390x) - rust-lang/rust#159272 (slice: make swap delegate to swap_unchecked) - rust-lang/rust#159274 (bootstrap: Rename `std_crates_for_run_make` to `std_crates_for_make_run`) - rust-lang/rust#159275 (remove obsolete comment) - rust-lang/rust#159277 (Construct `tokens` for attrs made by `mk_attr_word` and other variants) - rust-lang/rust#159283 (Remove obsolete verbose flag from deref/ref suggestions) - rust-lang/rust#159290 (rustc-dev-guide subtree update)

Instead of collecting them. In the vast majority of cases we can synthesize tokens that are indistinguishable from collected tokens. This is feasible because (a) attributes have a rigid micro-grammar and (b) we record tokens for for the internal pieces in the interesting cases (e.g. in `AttrArgs` within `AttrItem::args`). This requires adding a flag `use_precise_delim_token_spans` to distinguish the common case (perfect fidelity) from the uncommon case (token spans are degraded). This is a small perf win. It also has some other benefits. - `NormalAttr` shrinks slightly because the `tokens` field is removed. This means `NormalAttr` is now a trivial wrapper around `AttrItem` (and will be removed in a subsequent commit). - `Attribute` no longer needs a (no-op) impl of `HasAttrs`. - Builtin attributes had grown synthesized token streams (e.g. #159277). These are no longer needed and `mk_attr_tokens` is removable. - `expand_cfg_attr_item` no longer has to do its token surgery. The commit modifies the attr-complex-fn.rs test to demonstrate how `#`/`!`/`[`/`]` token spans degrade in the uncommon case. (This case is so uncommon that no existing tests triggered it.)

Construct tokens for attrs made by mk_attr_word
29668b5

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Jul 14, 2026

mu001999 changed the title ~~Construct tokens for attrs made by mk_attr_word and other variants~~Construct tokens for attrs made by mk_attr_word and other variantsJul 14, 2026

rustbot assigned jdonszelmann Jul 14, 2026

jdonszelmann approved these changes Jul 14, 2026
View reviewed changes

mu001999 marked this pull request as ready for review July 14, 2026 11:02

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 14, 2026

rust-borsBot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 14, 2026

JonathanBrouwer mentioned this pull request Jul 14, 2026
Rollup of 17 pull requests #159281
Closed

This was referenced Jul 14, 2026
Rollup of 21 pull requests #159292
Closed
Rollup of 22 pull requests #159293
Merged

This was referenced Jul 14, 2026
bug: Rust linux-build ICEs on nightly-2026-07-14 lance-format/lance#7786
Closed
ci: pin Rust nightly around test attribute ICE lance-format/lance#7787
Merged

rust-borsBot merged commit 9fb001a into rust-lang:mainJul 15, 2026
13 checks passed

rustbot added this to the 1.99.0 milestone Jul 15, 2026

Gri-ffin mentioned this pull request Jul 16, 2026
Panic with attribute is missing tokensrust-lang/rust-clippy#17423
Closed

nnethercote mentioned this pull request Jul 29, 2026
Synthesize tokens for attributes #160133
Draft

Navigation Menu

Uh oh!

Uh oh!

Construct `tokens` for attrs made by `mk_attr_word` and other variants - #159277

Construct `tokens` for attrs made by `mk_attr_word` and other variants#159277
rust-bors[bot] merged 1 commit into
rust-lang:mainfrom
mu001999-contrib:fix-159261

mu001999 commented Jul 14, 2026•
edited
Loading

Uh oh!

jdonszelmann commented Jul 14, 2026

Uh oh!

jdonszelmann left a comment •
edited
Loading

Uh oh!

jdonszelmannJul 14, 2026

Uh oh!

rust-borsBot commented Jul 14, 2026

Uh oh!

mu001999 commented Jul 14, 2026

Uh oh!

rust-borsBot commented Jul 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

mu001999 commented Jul 14, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jdonszelmann commented Jul 14, 2026

Uh oh!

jdonszelmann left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jdonszelmannJul 14, 2026

Choose a reason for hiding this comment

Uh oh!

rust-borsBot commented Jul 14, 2026

Uh oh!

mu001999 commented Jul 14, 2026

Uh oh!

rust-borsBot commented Jul 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mu001999 commented Jul 14, 2026•
edited
Loading

jdonszelmann left a comment •
edited
Loading