Refactor token collection to capture trailing token immediately

[Aaron1011]

Split out from #80689 - when we start capturing more information about attribute targets, we'll need to know in advance if we're capturing a trailing token or not.

r? @ghost

[Aaron1011]

@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion.

[bors]

⌛ Trying commit c0d77f1c76c8d324c77755cd99908860d4c3eec0 with merge 125fb95a2bc8697595e30ef058c5cc82b2222284...

[bors]

☀️ Try build successful - checks-actions
Build commit: 125fb95a2bc8697595e30ef058c5cc82b2222284 (125fb95a2bc8697595e30ef058c5cc82b2222284)

[rust-timer]

Queued 125fb95a2bc8697595e30ef058c5cc82b2222284 with parent d03fe84, future comparison URL.

@rustbot label: +S-waiting-on-perf

[rust-timer]

Finished benchmarking try commit (125fb95a2bc8697595e30ef058c5cc82b2222284): comparison url.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. Please note that if the perf results are neutral, you should likely undo the rollup=never given below by specifying rollup- to bors.

Importantly, though, if the results of this run are non-neutral do not roll this PR up -- it will mask other regressions or improvements in the roll up.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf

[Aaron1011]

@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion.

[bors]

⌛ Trying commit 73a976ffd206a284f6485baf7a37545ef6cc6343 with merge df2eeabc93790b4803c40f482839a38c56301e3b...

[petrochenkov]

Something like this was my first thought when you introduced the semicolon token collection - "you just need to refactor statement parsing carefully, so the parsed statements already contain semicolons".
Statement parsing is kind of a mess though, and is plagued by ad hoc diagnostic code as well, so it was kind of understandable why adding the add_trailing_semi hack could be less invasive.

The end goal for statement parsing is to use it basically everywhere - for parsing statements not only in blocks, but also in all contexts where items can be written (so e.g. let x = 10; is parsed and gracefully rejected even if written at module level).
Centril started this work in #69366 and related PRs, but didn't finish it unfortunately.
I think we can change (or even slightly break) the behavior of the (almost unused) stmt nonterminal to achieve this goal, if it becomes necessary.

[bors]

☀️ Try build successful - checks-actions
Build commit: df2eeabc93790b4803c40f482839a38c56301e3b (df2eeabc93790b4803c40f482839a38c56301e3b)

[rust-timer]

Queued df2eeabc93790b4803c40f482839a38c56301e3b with parent 4275ef6, future comparison URL.

@rustbot label: +S-waiting-on-perf

[rust-timer]

Finished benchmarking try commit (df2eeabc93790b4803c40f482839a38c56301e3b): comparison url.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. Please note that if the perf results are neutral, you should likely undo the rollup=never given below by specifying rollup- to bors.

Importantly, though, if the results of this run are non-neutral do not roll this PR up -- it will mask other regressions or improvements in the roll up.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf

[Aaron1011]

@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion.

[bors]

⌛ Trying commit f5acaa1f663e5f0979aad348b2bf1e081c1ef520 with merge 48d68b3b8d788525a0310668dfae564f7edf7e09...

[bors]

☀️ Try build successful - checks-actions
Build commit: 48d68b3b8d788525a0310668dfae564f7edf7e09 (48d68b3b8d788525a0310668dfae564f7edf7e09)

[Aaron1011]

@petrochenkov: The force_capture changes are needed to avoid exposing the underlying issue in #81007 further. I'll split out the changes first, and then rebase this PR on top of that.

[Aaron1011]

Blocked on #81177.

[Aaron1011]

@petrochenkov: I've rebased this PR against master. The content is almost the same as before - a small adjustment to maybe_collect_tokens was needed.

[Aaron1011]

@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion.

[bors]

⌛ Trying commit ccfc292 with merge 134fdeae99435ada2f12fd43c12f277914f9a9f7...

[bors]

☀️ Try build successful - checks-actions
Build commit: 134fdeae99435ada2f12fd43c12f277914f9a9f7 (134fdeae99435ada2f12fd43c12f277914f9a9f7)

[rust-timer]

Queued 134fdeae99435ada2f12fd43c12f277914f9a9f7 with parent dc1eee2, future comparison URL.

@rustbot label: +S-waiting-on-perf

[rust-timer]

Finished benchmarking try commit (134fdeae99435ada2f12fd43c12f277914f9a9f7): comparison url.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. Please note that if the perf results are neutral, you should likely undo the rollup=never given below by specifying rollup- to bors.

Importantly, though, if the results of this run are non-neutral do not roll this PR up -- it will mask other regressions or improvements in the roll up.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf

[petrochenkov]

@bors r+

[bors]

📌 Commit ccfc292 has been approved by petrochenkov

[bors]

⌛ Testing commit ccfc292 with merge 693ed05...

[bors]

☀️ Test successful - checks-actions
Approved by: petrochenkov
Pushing 693ed05 to master...