Speed up `EverInitializedPlaces` by nnethercote · Pull Request #160033 · rust-lang/rust

nnethercote · 2026-07-28T01:17:03Z

By simplifying its domain. Details in individual commits.

nnethercote · 2026-07-28T02:45:42Z

Speed up `EverInitializedPlaces`

rust-bors · 2026-07-28T04:57:46Z

☀️ Try build successful (CI)
Build commit: f89e821 (f89e821e955ecec1d472560b4370d1efcee76631)
Base parent: d3ea035 (d3ea0356830dc019038110eba0bc8f790aaef2e4)

rust-timer · 2026-07-28T05:38:04Z

Finished benchmarking commit (f89e821): comparison URL.

Overall result: ❌✅ regressions and improvements - please read:

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	3.0%	[0.6%, 5.4%]	5
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.3%	[-0.6%, -0.1%]	12
Improvements ✅ (secondary)	-6.3%	[-17.0%, -0.2%]	16
All ❌✅ (primary)	0.7%	[-0.6%, 5.4%]	17

Max RSS (memory usage)

Results (primary -2.7%, secondary -0.4%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	3.4%	[2.0%, 7.1%]	4
Improvements ✅ (primary)	-2.7%	[-2.8%, -2.5%]	2
Improvements ✅ (secondary)	-2.5%	[-4.1%, -1.1%]	7
All ❌✅ (primary)	-2.7%	[-2.8%, -2.5%]	2

Cycles

Results (primary -2.6%, secondary -7.5%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.1%	[2.1%, 2.1%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-3.6%	[-5.3%, -2.1%]	5
Improvements ✅ (secondary)	-7.5%	[-10.4%, -2.3%]	8
All ❌✅ (primary)	-2.6%	[-5.3%, 2.1%]	6

Binary size

Results (secondary 0.1%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.1%	[0.0%, 0.1%]	4
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Bootstrap: 487.844s -> 486.33s (-0.31%)
Artifact size: 388.73 MiB -> 388.91 MiB (0.05%)

nnethercote · 2026-07-28T10:04:57Z

A case where looking at cycles/wall-time is better than instruction counts. For cycles the two best results are -5% for cranelift-codegen and -10% for match-stress. I think the discrepancy has something to do with the old code using memcmp more.

@rustbot label: +perf-regression-triaged

cjgillot · 2026-07-31T02:10:30Z

I'm interested in this PR even if #160193 reduces the expected perf benefit. What makes you keep it as draft?

nnethercote · 2026-07-31T02:55:47Z

I'm waiting for #160193 to land so we can re-run perf. It's quite possible that the speed-up disappears completely. And I'm also a bit unhappy with the extra complexity in the second commit.

mehdiakiki · 2026-08-01T16:00:58Z

Hi @nnethercote !
I was investigating #159944 and since you said that you were waiting to see whether the benefit remains after #160193 I ran a targeted local A/B test on current main with both commits from this PR applied already. so I measured the following:

800 awaits: current main 4.96s this PR 1.28s
1600 awaits: current main 25.08s this PR 3.41s

and the separate temporary counters showed ever_init visits falling from 9,942,007 to 39,201 at 800 awaits and from 39 million approx to 78 thousands at 1600 awaits. also the maximum dropped to two visits per block.

so at least for the #159944 reproducer the performance appears to remain substantial after #160193.

nnethercote · 2026-08-01T22:59:00Z

@mehdiakiki, @zozo123: these comments are not helpful. Please stop.

nnethercote · 2026-08-02T09:48:17Z

#160193 merged, let's re-measure perf.

@bors try @rust-timer queue

Speed up `EverInitializedPlaces`

cjgillot · 2026-08-02T16:37:53Z

+ // Must be a call terminator. Genned only by `apply_call_return_effect`, i.e. along
+ // the call's return edge.
+ let TerminatorKind::Call { target: call_target, .. } =
+ init_block_data.terminator().kind
+ else {
+ bug!("`NonPanicPathOnly` should only be seen on a `Call` terminator");
+ };
+ queue.extend(call_target);


apply_call_return_effect is also called on Yield and InlineAsm.
Suggested change
// Must be a call terminator. Genned only by `apply_call_return_effect`, i.e. along
// the call's return edge.
letTerminatorKind::Call{target: call_target, .. } =
init_block_data.terminator().kind
else {
bug!("`NonPanicPathOnly` should only be seen on a `Call` terminator");
};
queue.extend(call_target);
// Genned only by `apply_call_return_effect`, i.e. along an assigning return edge.
letTerminatorEdges::AssignOnReturn{ return_, .. } =
init_block_data.terminator().edges()
else {
bug!("`NonPanicPathOnly` should only be seen on a returning terminators");
};
queue.extend(return_);
View changes since the review

Currently it tracks a bit for every `Init`, but it only uses the tracked data for locals. This means it is tracking data for projections that is unused. This commit shrinks the domain to only track data for `Local`s, going from `MixedBitSet<InitIndex>` to `DenseBitSet<Local>`. This does regress the error messages in one test: liveness-assign-imm-local-notes.rs. The next commit will fix that. The commit also removes some `debug` statements which probably haven't seen use in a long time. They can be re-added easily if anyone needs them in the future.

This commit fixes the error message regression in the previous commit by recomputing flow information at error-reporting time. `is_local_ever_initialized` is replaced in two ways. - In `check_access_permissions`: by the new `first_reaching_init` function which picks the first init that can reach the error location along a path that doesn't cross `StorageDead(local)`. - In `add_used_mut`: by a simple `contains` test.

nnethercote · 2026-08-03T02:25:14Z

I made a few changes.

Adjust EverInitializedPlaces so that apply_primary_terminator_effect and apply_call_return_effect have no overlap (one handles the !InitKind::NonPanicPathOnly case, the other the InitKind::NonPanicPathOnly case).
Use TerminatorEdges::AssignOnReturn.
Make init_reaches_location an EverInitializedPlaces method.
Simplify init_reaches_location: change the queue to work on locations, which eliminates the enum and the second loop.

rustbot · 2026-08-03T02:25:21Z

r? @wesleywiser

rustbot has assigned @wesleywiser.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

Owners of files modified in this PR: compiler
compiler expanded to 75 candidates
Random selection from 19 candidates

cjgillot · 2026-08-03T02:38:31Z

- }
+ state.gen_all(init_loc_map[call_loc].iter().copied().filter_map(|ii| {
+ let init = &move_data.inits[ii];
+ if init.kind == InitKind::NonPanicPathOnly {


Why do we check NonPanicPathOnly now?
View changes since the review

Look at Forward::apply_effects_in_block:
It calls apply_primary_terminator_effect unconditionally on the terminator.
It calls apply_call_return_effect for an AssignOnReturn terminator with at least one return edge (Call, Yield, InlineAsm).
So there is double handling of some inits, specifically Deep inits for Yield/InlineAsm.
apply_primary_terminator_effect already filters out the NonPanicPathOnly cases, so apply_call_return_effect can apply the inverse filter to avoid the double handling.
Double handling doesn't really matter, because gen is an idempotent operation. However, removing the double handling makes the if/else if/else in init_reaches_location cleanly mirror apply_primary_statement_effect / apply_call_return_effect / apply_primary_terminator_effect, as per the comments in init_reaches_location.

cjgillot · 2026-08-03T23:59:16Z

@bors r+

rust-bors · 2026-08-03T23:59:19Z

📌 Commit 3a7e145 has been approved by cjgillot

It is now in the queue for this repository.

JonathanBrouwer · 2026-08-04T12:39:55Z

@bors rollup=iffy
Including this in a special perf-sensitive rollup

…uwer Rollup of 10 perf-sensitive pull requests Successful merges: - #157281 (perf: skip irrelevant foreign impls when building the specialization graph) - #159403 (Next steps for FnDef binder changes (instantiate most FnDef binders)) - #159763 (Optimize crate resolution for large workspace) - #160033 (Speed up `EverInitializedPlaces`) - #160268 (perf: store the fulfillment engine inline in ObligationCtxt) - #160317 (perf: Cache already-checked types in the privacy visitor) - #160399 (interpret: skip deref-projection validity checks when they are not needed) - #160451 (Deduplicate target and host filesearch) - #160453 (Add fast path to `escape_string_symbol`) - #160454 (Add offload guard flags to typeck to prevent perf regressions)

@cjgillot

Rollup merge of #160033 - nnethercote:speed-up-EverInit, r=cjgillot Speed up `EverInitializedPlaces` By simplifying its domain. Details in individual commits. r? @cjgillot

JonathanBrouwer · 2026-08-05T06:40:27Z

Verifying that actual perf results after merge match expected results
@rust-timer build 65b521b

…uwer Rollup of 10 perf-sensitive pull requests Successful merges: - rust-lang/rust#157281 (perf: skip irrelevant foreign impls when building the specialization graph) - rust-lang/rust#159403 (Next steps for FnDef binder changes (instantiate most FnDef binders)) - rust-lang/rust#159763 (Optimize crate resolution for large workspace) - rust-lang/rust#160033 (Speed up `EverInitializedPlaces`) - rust-lang/rust#160268 (perf: store the fulfillment engine inline in ObligationCtxt) - rust-lang/rust#160317 (perf: Cache already-checked types in the privacy visitor) - rust-lang/rust#160399 (interpret: skip deref-projection validity checks when they are not needed) - rust-lang/rust#160451 (Deduplicate target and host filesearch) - rust-lang/rust#160453 (Add fast path to `escape_string_symbol`) - rust-lang/rust#160454 (Add offload guard flags to typeck to prevent perf regressions)

rust-timer · 2026-08-05T09:43:34Z

Finished benchmarking commit (65b521b): comparison URL.

Overall result: ✅ improvements - no action needed

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.2%	[0.2%, 0.2%]	1
Improvements ✅ (primary)	-0.4%	[-0.6%, -0.2%]	10
Improvements ✅ (secondary)	-5.9%	[-17.1%, -0.3%]	17
All ❌✅ (primary)	-0.4%	[-0.6%, -0.2%]	10

Max RSS (memory usage)

Results (primary 1.0%, secondary 2.4%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	1.8%	[0.4%, 5.6%]	8
Regressions ❌ (secondary)	2.4%	[0.7%, 6.3%]	6
Improvements ✅ (primary)	-2.2%	[-3.6%, -0.8%]	2
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	1.0%	[-3.6%, 5.6%]	10

Cycles

Results (primary -0.5%, secondary -3.3%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	0.8%	[0.7%, 1.0%]	2
Regressions ❌ (secondary)	0.7%	[0.4%, 1.0%]	5
Improvements ✅ (primary)	-0.8%	[-1.3%, -0.5%]	9
Improvements ✅ (secondary)	-4.9%	[-11.7%, -0.5%]	13
All ❌✅ (primary)	-0.5%	[-1.3%, 1.0%]	11

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 489.577s -> 489.996s (0.09%)
Artifact size: 390.24 MiB -> 390.29 MiB (0.01%)

…uwer Rollup of 10 perf-sensitive pull requests Successful merges: - rust-lang/rust#157281 (perf: skip irrelevant foreign impls when building the specialization graph) - rust-lang/rust#159403 (Next steps for FnDef binder changes (instantiate most FnDef binders)) - rust-lang/rust#159763 (Optimize crate resolution for large workspace) - rust-lang/rust#160033 (Speed up `EverInitializedPlaces`) - rust-lang/rust#160268 (perf: store the fulfillment engine inline in ObligationCtxt) - rust-lang/rust#160317 (perf: Cache already-checked types in the privacy visitor) - rust-lang/rust#160399 (interpret: skip deref-projection validity checks when they are not needed) - rust-lang/rust#160451 (Deduplicate target and host filesearch) - rust-lang/rust#160453 (Add fast path to `escape_string_symbol`) - rust-lang/rust#160454 (Add offload guard flags to typeck to prevent perf regressions)

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Jul 28, 2026

This comment has been minimized.
Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jul 28, 2026

rust-borsBot pushed a commit that referenced this pull request Jul 28, 2026
Auto merge of#160033- nnethercote:speed-up-EverInit, r=<try>
f89e821
Speed up `EverInitializedPlaces`

This comment has been minimized.
Sign in to view

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Jul 28, 2026

rustbot added the perf-regression-triaged The performance regression has been triaged. label Jul 28, 2026

nnethercote mentioned this pull request Jul 30, 2026
Improve CFG traversal #160193
Merged

This comment was marked as low quality.
Sign in to view

nnethercote force-pushed the speed-up-EverInit branch from e7339de to 4b3859fCompare August 2, 2026 09:46

This comment has been minimized.
Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Aug 2, 2026

This comment has been minimized.
Sign in to view

rust-borsBot pushed a commit that referenced this pull request Aug 2, 2026
Auto merge of#160033- nnethercote:speed-up-EverInit, r=<try>
4328ef8
Speed up `EverInitializedPlaces`

cjgillot reviewed Aug 2, 2026
View reviewed changes

Comment threadcompiler/rustc_borrowck/src/lib.rs Outdated

nnethercote added 2 commits August 3, 2026 11:41

nnethercote force-pushed the speed-up-EverInit branch from 4b3859f to 3a7e145Compare August 3, 2026 02:23

nnethercote marked this pull request as ready for review August 3, 2026 02:25

rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Aug 3, 2026

rustbot assigned wesleywiser Aug 3, 2026

rustbot removed the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Aug 3, 2026

rustbot assigned cjgillot and unassigned wesleywiser Aug 3, 2026

cjgillot reviewed Aug 3, 2026
View reviewed changes

rust-borsBot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 3, 2026

JonathanBrouwer mentioned this pull request Aug 4, 2026
Rollup of 10 perf-sensitive pull requests #160506
Merged

rust-borsBot merged commit 169500c into rust-lang:mainAug 4, 2026
13 checks passed

rustbot added this to the 1.99.0 milestone Aug 4, 2026

rust-timer added a commit that referenced this pull request Aug 4, 2026
Unrolled build for#160033
65b521b
Rollup merge of #160033 - nnethercote:speed-up-EverInit, r=cjgillot Speed up `EverInitializedPlaces` By simplifying its domain. Details in individual commits. r? @cjgillot

nnethercote deleted the speed-up-EverInit branch August 4, 2026 22:13

This comment has been minimized.
Sign in to view

-// Must be a call terminator. Genned only by `apply_call_return_effect`, i.e. along
-// the call's return edge.
-letTerminatorKind::Call{target: call_target, .. } =
- init_block_data.terminator().kind
- else {
-bug!("`NonPanicPathOnly` should only be seen on a `Call` terminator");
-};
- queue.extend(call_target);
+// Genned only by `apply_call_return_effect`, i.e. along an assigning return edge.
+letTerminatorEdges::AssignOnReturn{ return_, .. } =
+ init_block_data.terminator().edges()
+ else {
+bug!("`NonPanicPathOnly` should only be seen on a returning terminators");
+};
+ queue.extend(return_);

Uh oh!

Uh oh!

Conversation

nnethercote commented Jul 28, 2026• edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nnethercote commented Jul 28, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

rust-borsBot commented Jul 28, 2026

Uh oh!

This comment has been minimized.

rust-timer commented Jul 28, 2026

Overall result: ❌✅ regressions and improvements - please read:

Uh oh!

nnethercote commented Jul 28, 2026

Uh oh!

cjgillot commented Jul 31, 2026

Uh oh!

nnethercote commented Jul 31, 2026

Uh oh!

mehdiakiki commented Aug 1, 2026

Uh oh!

This comment was marked as low quality.

This comment was marked as low quality.

This comment was marked as low quality.

This comment was marked as low quality.

This comment was marked as low quality.

nnethercote commented Aug 1, 2026

Uh oh!

nnethercote commented Aug 2, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

cjgillotAug 2, 2026• edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nnethercote commented Aug 3, 2026

Uh oh!

rustbot commented Aug 3, 2026

Uh oh!

cjgillotAug 3, 2026• edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cjgillot commented Aug 3, 2026

Uh oh!

rust-borsBot commented Aug 3, 2026

Uh oh!

JonathanBrouwer commented Aug 4, 2026

Uh oh!

Uh oh!

JonathanBrouwer commented Aug 5, 2026

Uh oh!

This comment has been minimized.

rust-timer commented Aug 5, 2026

Overall result: ✅ improvements - no action needed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

nnethercote commented Jul 28, 2026•
edited by rustbot
Loading

cjgillotAug 2, 2026•
edited by rustbot
Loading

cjgillotAug 3, 2026•
edited by rustbot
Loading