Optimize dependency file search - #153131

Merged
rust-bors[bot] merged 4 commits into
rust-lang:mainfrom
Kobzol:filesearch-opt
Mar 2, 2026
Merged

Optimize dependency file search#153131
rust-bors[bot] merged 4 commits into
rust-lang:mainfrom
Kobzol:filesearch-opt

Conversation

@Kobzol

@KobzolKobzol commented Feb 26, 2026•
edited by rustbot
Loading

Copy link
Copy Markdown
Member

View all comments

I tried to look into the slowdown reported in rust-lang/cargo#16665.

I created a Rust hello world program, and used this Python script to create a directory containing 200k files:

frompathlibimportPathdir=Path("deps")
dir.mkdir(parents=True, exist_ok=True)
foriinrange(200000):
path=dir/f"file{i:07}.o"withopen(path, "w") asf:
f.write("\n")

Then I tried to do various small microoptimalizations and simplifications to the code that iterates the search directories. Each individual commit improved performance, with the third one having the biggest effect.

Here are the results on main vs the last commit with the stage1 compiler on Linux, using hyperfine "rustc +stage1 src/main.rs -L deps" -r 30 (there's IO involved, so it's good to let it run for a while):

Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 299.4 ms ± 2.7 ms [User: 161.9 ms, System: 144.9 ms]
Range (min … max): 294.8 ms … 307.1 ms 30 runs
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 208.1 ms ± 4.5 ms [User: 87.3 ms, System: 128.7 ms]
Range (min … max): 202.4 ms … 219.6 ms 30 runs

Would be cool if someone could try this on macOS (maybe @ehuss - not sure if you have macOS or you only commented about its behavior on the Cargo issue :) ).

I also tried to prefilter the paths (not in this PR); right now we load everything and then we filter files with given prefixes, that's wasteful. Filtering just files starting with lib would get us down to ~150ms here. (The baseline without -L is ~80ms on my PC). The rest of the 70ms is essentially allocations from iterating the directory entries and sorting. That would be very hard to change - iterating the directory entries (de)allocates a lot of intermediate paths :( We'd have to implement the iteration by hand with either arena allocation, or at least some better management of memory.

r? @nnethercote

@rustbotrustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Feb 26, 2026
@Kobzol

Kobzol commented Feb 26, 2026

Copy link
Copy Markdown
MemberAuthor

Not expecting any big wins there, but let's check:

@bors try @rust-timer queue

@rust-timer

This comment has been minimized.

Sign in to view
@rust-bors

This comment has been minimized.

Sign in to view
@rustbotrustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Feb 26, 2026
rust-borsBot pushed a commit that referenced this pull request Feb 26, 2026
@bors
Auto merge of#153131- Kobzol:filesearch-opt, r=<try>
71d4456
Optimize dependency file search
petrochenkov
petrochenkov reviewed Feb 26, 2026
View reviewed changes
Comment threadcompiler/rustc_session/src/search_paths.rs
@rust-bors

rust-borsBot commented Feb 26, 2026

Copy link
Copy Markdown
Contributor

☀️ Try build successful (CI)
Build commit: 71d4456 (71d445654a32204fbf86bfec11e657fb97adcd22, parent: 69b78537fac74de40f009b076bcbbf54b77683ad)

@rust-timer

This comment has been minimized.

Sign in to view
@ehuss

ehuss commented Feb 26, 2026

Copy link
Copy Markdown
Contributor

Unfortunately I am unable to test because of #153077, I am unable to get rustc to build.

@rust-timer

rust-timer commented Feb 26, 2026

Copy link
Copy Markdown
Collaborator

Finished benchmarking commit (71d4456): comparison URL.

Overall result: ❌✅ regressions and improvements - please read the text below

Benchmarking this pull request means it may be perf-sensitive – we'll automatically label it not fit for rolling up. You can override this, but we strongly advise not to, due to possible changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please do so in sufficient writing along with @rustbot label: +perf-regression-triaged. If not, please fix the regressions and do another perf run. If its results are neutral or positive, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

meanrangecount
Regressions ❌
(primary)		0.3%[0.2%, 0.3%]4
Regressions ❌
(secondary)		0.0%[0.0%, 0.0%]1
Improvements ✅
(primary)		--0
Improvements ✅
(secondary)		-0.4%[-0.6%, -0.1%]7
All ❌✅ (primary)0.3%[0.2%, 0.3%]4

Max RSS (memory usage)

Results (secondary 0.1%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

meanrangecount
Regressions ❌
(primary)		--0
Regressions ❌
(secondary)		6.8%[6.8%, 6.8%]1
Improvements ✅
(primary)		--0
Improvements ✅
(secondary)		-6.7%[-6.7%, -6.7%]1
All ❌✅ (primary)--0

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 480.132s -> 480.258s (0.03%)
Artifact size: 395.80 MiB -> 395.77 MiB (-0.01%)

@rustbotrustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Feb 26, 2026
@ehuss

ehuss commented Feb 26, 2026

Copy link
Copy Markdown
Contributor

OK, got some data:

With this PR:

Benchmark 1: ~/Proj/rust/rust/build/aarch64-apple-darwin/stage1/bin/rustc src/main.rs -L deps
Time (mean ± σ): 197.4 ms ± 3.9 ms [User: 143.7 ms, System: 117.9 ms]
Range (min … max): 192.4 ms … 207.7 ms 30 runs

Without:

Benchmark 1: ~/Proj/rust/rust/build/aarch64-apple-darwin/stage1/bin/rustc src/main.rs -L deps
Time (mean ± σ): 248.9 ms ± 5.1 ms [User: 189.5 ms, System: 124.2 ms]
Range (min … max): 241.1 ms … 262.6 ms 30 runs

The tests are super noisy, though (I get significantly different results between runs). But it seems like a pretty good win.

@nnethercote

nnethercote commented Feb 27, 2026

Copy link
Copy Markdown
Contributor

Nice wins.

@bors r+

@rust-bors

rust-borsBot commented Feb 27, 2026

Copy link
Copy Markdown
Contributor

📌 Commit f450289 has been approved by nnethercote

It is now in the queue for this repository.

@rust-borsrust-borsBot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 27, 2026
bjorn3
bjorn3 reviewed Feb 27, 2026
View reviewed changes
Comment threadcompiler/rustc_session/src/search_paths.rs
Err(..) => Default::default(),
};
files.sort_by(|lhs, rhs| lhs.file_name_str.cmp(&rhs.file_name_str));
files.sort_unstable_by(|lhs, rhs| lhs.file_name_str.cmp(&rhs.file_name_str));

@bjorn3bjorn3Feb 27, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there no possibility of two files with the same file_name_str?

@bjorn3bjorn3Feb 27, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although I guess that would result in either one being picked deterministically (if it is the only matching one), which one is picked not mattering (if both files having the same crate hash) or an error about multiple candidates getting emitted.

@KobzolKobzolFeb 27, 2026

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We sorted (and searched) by the UTF-8 name before anyway, so there shouldn't hopefully be any change in behavior.

@rust-bors

This comment has been minimized.

Sign in to view
rust-borsBot pushed a commit that referenced this pull request Mar 1, 2026
@bors
Auto merge of#153131- Kobzol:filesearch-opt, r=nnethercote
07af97d
Optimize dependency file search
I tried to look into the slowdown reported in rust-lang/cargo#16665.
I created a Rust hello world program, and used this Python script to create a directory containing 200k files:
```python
from pathlib import Path
dir = Path("deps")
dir.mkdir(parents=True, exist_ok=True)
for i in range(200000):
path = dir / f"file{i:07}.o"
with open(path, "w") as f:
f.write("\n")
```
Then I tried to do various small microoptimalizations and simplifications to the code that iterates the search directories. Each individual commit improved performance, with the third one having the biggest effect.
Here are the results on `main` vs the last commit with the stage1 compiler on Linux, using `hyperfine "rustc +stage1 src/main.rs -L deps" -r 30` (there's IO involved, so it's good to let it run for a while):
```bash
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 299.4 ms ± 2.7 ms [User: 161.9 ms, System: 144.9 ms]
Range (min … max): 294.8 ms … 307.1 ms 30 runs
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 208.1 ms ± 4.5 ms [User: 87.3 ms, System: 128.7 ms]
Range (min … max): 202.4 ms … 219.6 ms 30 runs
```
Would be cool if someone could try this on macOS (maybe @ehuss - not sure if you have macOS or you only commented about its behavior on the Cargo issue :) ).
I also tried to prefilter the paths (not in this PR); right now we load everything and then we filter files with given prefixes, that's wasteful. Filtering just files starting with `lib` would get us down to ~150ms here. (The baseline without `-L` is ~80ms on my PC). The rest of the 70ms is essentially allocations from iterating the directory entries and sorting. That would be very hard to change - iterating the directory entries (de)allocates a lot of intermediate paths :( We'd have to implement the iteration by hand with either arena allocation, or at least some better management of memory.
r? @nnethercote
@rust-borsrust-borsBot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Mar 1, 2026
@rust-bors

rust-borsBot commented Mar 1, 2026

Copy link
Copy Markdown
Contributor

💔 Test for 07af97d failed: CI. Failed job:

@rust-log-analyzer

rust-log-analyzer commented Mar 1, 2026

Copy link
Copy Markdown
Collaborator

The job aarch64-gnu failed! Check out the build log: (web)(plain enhanced)(plain)

Click to see the possible cause of the failure (guessed by this bot)
/dev/sda15 98M 6.4M 92M 7% /boot/efi
tmpfs 1.6G 16K 1.6G 1% /run/user/1001
================================================================================
Sufficient disk space available (120307208KB >= 52428800KB). Skipping cleanup.
##[group]Run src/ci/scripts/setup-environment.sh
src/ci/scripts/setup-environment.sh
shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
---
[1773/3894] Building ARMGenMCCodeEmitter.inc...
##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
[1774/3894] Building ARMGenMCPseudoLowering.inc...
Session terminated, killing shell...::group::Clock drift check
local time: Sun Mar 1 08:40:58 UTC 2026
network time: [1775/3894] Building ARMGenInstrInfo.inc...
Sun, 01 Mar 2026 08:40:58 GMT
##[endgroup]
[1776/3894] Building ARMGenRegisterBank.inc...

@Kobzol

Kobzol commented Mar 1, 2026

Copy link
Copy Markdown
MemberAuthor

@bors retry

Spurious worker restart.

@rust-borsrust-borsBot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 1, 2026
This was referenced Mar 1, 2026
Merged
Merged
@rust-bors

This comment has been minimized.

Sign in to view
@rust-borsrust-borsBot added merged-by-bors This PR was explicitly merged by bors. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Mar 2, 2026
@rust-bors

rust-borsBot commented Mar 2, 2026

Copy link
Copy Markdown
Contributor

☀️ Test successful - CI
Approved by: nnethercote
Duration: 3h 40m 3s
Pushing e7d90c6 to main...

@rust-bors
rust-borsBot merged commit e7d90c6 into rust-lang:mainMar 2, 2026
13 checks passed
@rustbotrustbot added this to the 1.96.0 milestone Mar 2, 2026
@github-actions

github-actionsBot commented Mar 2, 2026

Copy link
Copy Markdown
Contributor
What is this? This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 8038127 (parent) -> e7d90c6 (this PR)

Test differences

Show 2 test diffs

2 doctest diffs were found. These are ignored, as they are noisy.

Test dashboard

Run

cargo run --manifest-path src/ci/citool/Cargo.toml -- \
test-dashboard e7d90c695a39426baf5ae705de2f9570a72229de --output-dir test-dashboard

And then open test-dashboard/index.html in your browser to see an overview of all executed tests.

Job duration changes

  1. dist-aarch64-llvm-mingw: 1h 33m -> 1h 46m (+14.1%)
  2. dist-aarch64-apple: 1h 56m -> 2h 12m (+13.1%)
  3. aarch64-apple: 3h 8m -> 3h 33m (+13.0%)
  4. dist-x86_64-apple: 2h 42m -> 2h 25m (-10.7%)
  5. arm-android: 1h 37m -> 1h 47m (+10.4%)
  6. dist-ohos-x86_64: 1h 14m -> 1h 21m (+9.7%)
  7. x86_64-gnu-aux: 2h 7m -> 2h 18m (+8.5%)
  8. x86_64-gnu-distcheck: 2h 7m -> 2h 18m (+8.4%)
  9. x86_64-gnu-miri: 1h 23m -> 1h 30m (+7.7%)
  10. dist-arm-linux-musl: 1h 38m -> 1h 30m (-7.3%)
How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

@rust-timer

rust-timer commented Mar 2, 2026

Copy link
Copy Markdown
Collaborator

Finished benchmarking commit (e7d90c6): comparison URL.

Overall result: ❌✅ regressions and improvements - please read the text below

Our benchmarks found a performance regression caused by this PR.
This might be an actual regression, but it can also be just noise.

Next Steps:

  • If the regression was expected or you think it can be justified,
    please write a comment with sufficient written justification, and add
    @rustbot label: +perf-regression-triaged to it, to mark the regression as triaged.
  • If you think that you know of a way to resolve the regression, try to create
    a new PR with a fix for the regression.
  • If you do not understand the regression or you think that it is just noise,
    you can ask the @rust-lang/wg-compiler-performance working group for help (members of this group
    were already notified of this PR).

@rustbot label: +perf-regression
cc @rust-lang/wg-compiler-performance

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

meanrangecount
Regressions ❌
(primary)		0.1%[0.1%, 0.1%]1
Regressions ❌
(secondary)		--0
Improvements ✅
(primary)		--0
Improvements ✅
(secondary)		-0.4%[-0.6%, -0.2%]7
All ❌✅ (primary)0.1%[0.1%, 0.1%]1

Max RSS (memory usage)

Results (secondary -4.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

meanrangecount
Regressions ❌
(primary)		--0
Regressions ❌
(secondary)		--0
Improvements ✅
(primary)		--0
Improvements ✅
(secondary)		-4.0%[-5.4%, -2.6%]3
All ❌✅ (primary)--0

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

Results (secondary 0.1%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

meanrangecount
Regressions ❌
(primary)		--0
Regressions ❌
(secondary)		0.1%[0.1%, 0.1%]3
Improvements ✅
(primary)		--0
Improvements ✅
(secondary)		--0
All ❌✅ (primary)--0

Bootstrap: 480.971s -> 480.149s (-0.17%)
Artifact size: 396.96 MiB -> 396.96 MiB (0.00%)

@Kobzol

Kobzol commented Mar 2, 2026•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

The regression seems like noise. Tiny wins on large-workspace, matching pre-merge run.

@rustbot label: +perf-regression-triaged

@rustbotrustbot added the perf-regression-triaged The performance regression has been triaged. label Mar 2, 2026
@osiewicz

osiewicz commented Mar 14, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Timings for Mac, as requested in OP (with the same scenario as the one exercised in OP):

➜ rustc-153131 hyperfine "rustc +nightly-2026-03-01 main.rs -L deps" -r 30
Benchmark 1: rustc +nightly-2026-03-01 main.rs -L deps
Time (mean ± σ): 215.3 ms ± 1.9 ms [User: 161.2 ms, System: 122.0 ms]
Range (min … max): 212.2 ms … 220.7 ms 30 runs

vs:

➜ rustc-153131 hyperfine "rustc +nightly-2026-03-06 main.rs -L deps" -r 30
Benchmark 1: rustc +nightly-2026-03-06 main.rs -L deps
Time (mean ± σ): 177.9 ms ± 1.6 ms [User: 128.5 ms, System: 120.6 ms]
Range (min … max): 175.2 ms … 183.2 ms 30 runs

And a baseline:

➜ rustc-153131 hyperfine "rustc +nightly-2026-03-06 main.rs" -r 30
Benchmark 1: rustc +nightly-2026-03-06 main.rs
Time (mean ± σ): 68.0 ms ± 1.1 ms [User: 87.0 ms, System: 52.7 ms]
Range (min … max): 66.8 ms … 71.9 ms 30 runs

@Kobzol
Kobzol deleted the filesearch-opt branch March 14, 2026 10:10
@KobzolKobzol mentioned this pull request Mar 15, 2026
Open
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@petrochenkovpetrochenkovpetrochenkov left review comments
@bjorn3bjorn3bjorn3 left review comments
@nnethercotennethercotennethercote approved these changes

Assignees

@nnethercotennethercote

Labels

merged-by-borsThis PR was explicitly merged by bors.perf-regressionPerformance regression.perf-regression-triagedThe performance regression has been triaged.T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.

Projects

None yet

Milestone

1.96.0

Development

Successfully merging this pull request may close these issues.

9 participants

@Kobzol@rust-timer@ehuss@nnethercote@rust-log-analyzer@osiewicz@petrochenkov@bjorn3@rustbot
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Optimize dependency file search - #153131
Merged
rust-bors[bot] merged 4 commits into
rust-lang:mainfrom 
Kobzol:filesearch-opt
Mar 2, 2026
Merged
Optimize dependency file search#153131
rust-bors[bot] merged 4 commits into
rust-lang:mainfrom 
Kobzol:filesearch-opt
Conversation
@Kobzol
@KobzolKobzol
commented
Feb 26, 2026•

edited by rustbot

Loading

Copy link


Copy Markdown

Member
View all comments
I tried to look into the slowdown reported in rust-lang/cargo#16665.
I created a Rust hello world program, and used this Python script to create a directory containing 200k files:
frompathlibimportPathdir=Path("deps")
dir.mkdir(parents=True, exist_ok=True)
foriinrange(200000):
path=dir/f"file{i:07}.o"withopen(path, "w") asf:
f.write("\n")
Then I tried to do various small microoptimalizations and simplifications to the code that iterates the search directories. Each individual commit improved performance, with the third one having the biggest effect.
Here are the results on main vs the last commit with the stage1 compiler on Linux, using hyperfine "rustc +stage1 src/main.rs -L deps" -r 30 (there's IO involved, so it's good to let it run for a while):
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 299.4 ms ± 2.7 ms [User: 161.9 ms, System: 144.9 ms]
Range (min … max): 294.8 ms … 307.1 ms 30 runs
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 208.1 ms ± 4.5 ms [User: 87.3 ms, System: 128.7 ms]
Range (min … max): 202.4 ms … 219.6 ms 30 runs
Would be cool if someone could try this on macOS (maybe @ehuss - not sure if you have macOS or you only commented about its behavior on the Cargo issue :) ).
I also tried to prefilter the paths (not in this PR); right now we load everything and then we filter files with given prefixes, that's wasteful. Filtering just files starting with lib would get us down to ~150ms here. (The baseline without -L is ~80ms on my PC). The rest of the 70ms is essentially allocations from iterating the directory entries and sorting. That would be very hard to change - iterating the directory entries (de)allocates a lot of intermediate paths :( We'd have to implement the iteration by hand with either arena allocation, or at least some better management of memory.
r? @nnethercote
@rustbotrustbot
added

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
T-compiler
Relevant to the compiler team, which will review and decide on the PR/issue.
labels
Feb 26, 2026
@Kobzol
Kobzol
commented
Feb 26, 2026

Copy link


Copy Markdown

MemberAuthor
Not expecting any big wins there, but let's check:
@bors try @rust-timer queue
@rust-timer

This comment has been minimized.

Sign in to view
@rust-bors

This comment has been minimized.

Sign in to view
@rustbotrustbot
added
the 
S-waiting-on-perf
Status: Waiting on a perf run to be completed.
label
Feb 26, 2026
rust-borsBot
pushed a commit
that referenced
this pull request
Feb 26, 2026
@bors
Auto merge of#153131- Kobzol:filesearch-opt, r=<try>
71d4456
Optimize dependency file search
petrochenkov
petrochenkov
reviewed
Feb 26, 2026
View reviewed changes
Comment threadcompiler/rustc_session/src/search_paths.rs
@rust-bors
rust-borsBot
commented
Feb 26, 2026

Copy link


Copy Markdown

Contributor
☀️ Try build successful (CI)

Build commit: 71d4456 (71d445654a32204fbf86bfec11e657fb97adcd22, parent: 69b78537fac74de40f009b076bcbbf54b77683ad)
@rust-timer

This comment has been minimized.

Sign in to view
@ehuss
ehuss
commented
Feb 26, 2026

Copy link


Copy Markdown

Contributor
Unfortunately I am unable to test because of #153077, I am unable to get rustc to build.
@rust-timer
rust-timer
commented
Feb 26, 2026

Copy link


Copy Markdown

Collaborator
Finished benchmarking commit (71d4456): comparison URL.
Overall result: ❌✅ regressions and improvements - please read the text below
Benchmarking this pull request means it may be perf-sensitive – we'll automatically label it not fit for rolling up. You can override this, but we strongly advise not to, due to possible changes in compiler perf.
Next Steps: If you can justify the regressions found in this try perf run, please do so in sufficient writing along with @rustbot label: +perf-regression-triaged. If not, please fix the regressions and do another perf run. If its results are neutral or positive, the label will be automatically removed.
@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression
Instruction count
Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.
meanrangecount
Regressions ❌ 
 (primary)		0.3%[0.2%, 0.3%]4
Regressions ❌ 
 (secondary)		0.0%[0.0%, 0.0%]1
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-0.4%[-0.6%, -0.1%]7
All ❌✅ (primary)0.3%[0.2%, 0.3%]4
Max RSS (memory usage)
Results (secondary 0.1%)
A less reliable metric. May be of interest, but not used to determine the overall result above.
meanrangecount
Regressions ❌ 
 (primary)		--0
Regressions ❌ 
 (secondary)		6.8%[6.8%, 6.8%]1
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-6.7%[-6.7%, -6.7%]1
All ❌✅ (primary)--0
Cycles
This benchmark run did not return any relevant results for this metric.
Binary size
This benchmark run did not return any relevant results for this metric.
Bootstrap: 480.132s -> 480.258s (0.03%)
Artifact size: 395.80 MiB -> 395.77 MiB (-0.01%)
@rustbotrustbot
added

perf-regression
Performance regression.
and removed

S-waiting-on-perf
Status: Waiting on a perf run to be completed.
labels
Feb 26, 2026
@ehuss
ehuss
commented
Feb 26, 2026

Copy link


Copy Markdown

Contributor
OK, got some data:
With this PR:
Benchmark 1: ~/Proj/rust/rust/build/aarch64-apple-darwin/stage1/bin/rustc src/main.rs -L deps
Time (mean ± σ): 197.4 ms ± 3.9 ms [User: 143.7 ms, System: 117.9 ms]
Range (min … max): 192.4 ms … 207.7 ms 30 runs

Without:
Benchmark 1: ~/Proj/rust/rust/build/aarch64-apple-darwin/stage1/bin/rustc src/main.rs -L deps
Time (mean ± σ): 248.9 ms ± 5.1 ms [User: 189.5 ms, System: 124.2 ms]
Range (min … max): 241.1 ms … 262.6 ms 30 runs

The tests are super noisy, though (I get significantly different results between runs). But it seems like a pretty good win.
@nnethercote
nnethercote
commented
Feb 27, 2026

Copy link


Copy Markdown

Contributor
Nice wins.
@bors r+
@rust-bors
rust-borsBot
commented
Feb 27, 2026

Copy link


Copy Markdown

Contributor
📌 Commit f450289 has been approved by nnethercote
It is now in the queue for this repository.
@rust-borsrust-borsBot
added

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
and removed

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
labels
Feb 27, 2026
bjorn3
bjorn3
reviewed
Feb 27, 2026
View reviewed changes
Comment threadcompiler/rustc_session/src/search_paths.rs
 Err(..) => Default::default(),
 };
 files.sort_by(|lhs, rhs| lhs.file_name_str.cmp(&rhs.file_name_str));
 files.sort_unstable_by(|lhs, rhs| lhs.file_name_str.cmp(&rhs.file_name_str));
@bjorn3bjorn3Feb 27, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there no possibility of two files with the same file_name_str?
@bjorn3bjorn3Feb 27, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although I guess that would result in either one being picked deterministically (if it is the only matching one), which one is picked not mattering (if both files having the same crate hash) or an error about multiple candidates getting emitted.
@KobzolKobzolFeb 27, 2026

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We sorted (and searched) by the UTF-8 name before anyway, so there shouldn't hopefully be any change in behavior.
@rust-bors

This comment has been minimized.

Sign in to view
rust-borsBot
pushed a commit
that referenced
this pull request
Mar 1, 2026
@bors
Auto merge of#153131- Kobzol:filesearch-opt, r=nnethercote
07af97d
Optimize dependency file search
I tried to look into the slowdown reported in rust-lang/cargo#16665.
I created a Rust hello world program, and used this Python script to create a directory containing 200k files:
```python
from pathlib import Path
dir = Path("deps")
dir.mkdir(parents=True, exist_ok=True)
for i in range(200000):
path = dir / f"file{i:07}.o"
with open(path, "w") as f:
f.write("\n")
```
Then I tried to do various small microoptimalizations and simplifications to the code that iterates the search directories. Each individual commit improved performance, with the third one having the biggest effect.
Here are the results on `main` vs the last commit with the stage1 compiler on Linux, using `hyperfine "rustc +stage1 src/main.rs -L deps" -r 30` (there's IO involved, so it's good to let it run for a while):
```bash
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 299.4 ms ± 2.7 ms [User: 161.9 ms, System: 144.9 ms]
Range (min … max): 294.8 ms … 307.1 ms 30 runs
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 208.1 ms ± 4.5 ms [User: 87.3 ms, System: 128.7 ms]
Range (min … max): 202.4 ms … 219.6 ms 30 runs
```
Would be cool if someone could try this on macOS (maybe @ehuss - not sure if you have macOS or you only commented about its behavior on the Cargo issue :) ).
I also tried to prefilter the paths (not in this PR); right now we load everything and then we filter files with given prefixes, that's wasteful. Filtering just files starting with `lib` would get us down to ~150ms here. (The baseline without `-L` is ~80ms on my PC). The rest of the 70ms is essentially allocations from iterating the directory entries and sorting. That would be very hard to change - iterating the directory entries (de)allocates a lot of intermediate paths :( We'd have to implement the iteration by hand with either arena allocation, or at least some better management of memory.
r? @nnethercote
@rust-borsrust-borsBot
added

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
and removed

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
labels
Mar 1, 2026
@rust-bors
rust-borsBot
commented
Mar 1, 2026

Copy link


Copy Markdown

Contributor
💔 Test for 07af97d failed: CI. Failed job:
@rust-log-analyzer
rust-log-analyzer
commented
Mar 1, 2026

Copy link


Copy Markdown

Collaborator
The job aarch64-gnu failed! Check out the build log: (web)(plain enhanced)(plain)
Click to see the possible cause of the failure (guessed by this bot)
/dev/sda15 98M 6.4M 92M 7% /boot/efi
tmpfs 1.6G 16K 1.6G 1% /run/user/1001
================================================================================
Sufficient disk space available (120307208KB >= 52428800KB). Skipping cleanup.
##[group]Run src/ci/scripts/setup-environment.sh
src/ci/scripts/setup-environment.sh
shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
---
[1773/3894] Building ARMGenMCCodeEmitter.inc...
##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
[1774/3894] Building ARMGenMCPseudoLowering.inc...
Session terminated, killing shell...::group::Clock drift check
local time: Sun Mar 1 08:40:58 UTC 2026
network time: [1775/3894] Building ARMGenInstrInfo.inc...
Sun, 01 Mar 2026 08:40:58 GMT
##[endgroup]
[1776/3894] Building ARMGenRegisterBank.inc...

@Kobzol
Kobzol
commented
Mar 1, 2026

Copy link


Copy Markdown

MemberAuthor
@bors retry
Spurious worker restart.
@rust-borsrust-borsBot
added

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
and removed

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
labels
Mar 1, 2026

This was referenced Mar 1, 2026
 Merged

 Merged

@rust-bors

This comment has been minimized.

Sign in to view
@rust-borsrust-borsBot
added

merged-by-bors
This PR was explicitly merged by bors.
and removed

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
labels
Mar 2, 2026
@rust-bors
rust-borsBot
commented
Mar 2, 2026

Copy link


Copy Markdown

Contributor
☀️ Test successful - CI

Approved by: nnethercote

Duration: 3h 40m 3s

Pushing e7d90c6 to main...
@rust-bors
rust-borsBot
merged commit e7d90c6
into
rust-lang:mainMar 2, 2026

13 checks passed

@rustbotrustbot
added this to the 1.96.0 milestone
Mar 2, 2026
@github-actions
github-actionsBot
commented
Mar 2, 2026

Copy link


Copy Markdown

Contributor
What is this?
This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 8038127 (parent) -> e7d90c6 (this PR)
Test differences
Show 2 test diffs

2 doctest diffs were found. These are ignored, as they are noisy.
Test dashboard
Run
cargo run --manifest-path src/ci/citool/Cargo.toml -- \
test-dashboard e7d90c695a39426baf5ae705de2f9570a72229de --output-dir test-dashboard
And then open test-dashboard/index.html in your browser to see an overview of all executed tests.
Job duration changes
  1. dist-aarch64-llvm-mingw: 1h 33m -> 1h 46m (+14.1%)
  2. dist-aarch64-apple: 1h 56m -> 2h 12m (+13.1%)
  3. aarch64-apple: 3h 8m -> 3h 33m (+13.0%)
  4. dist-x86_64-apple: 2h 42m -> 2h 25m (-10.7%)
  5. arm-android: 1h 37m -> 1h 47m (+10.4%)
  6. dist-ohos-x86_64: 1h 14m -> 1h 21m (+9.7%)
  7. x86_64-gnu-aux: 2h 7m -> 2h 18m (+8.5%)
  8. x86_64-gnu-distcheck: 2h 7m -> 2h 18m (+8.4%)
  9. x86_64-gnu-miri: 1h 23m -> 1h 30m (+7.7%)
  10. dist-arm-linux-musl: 1h 38m -> 1h 30m (-7.3%)
How to interpret the job duration changes?
Job durations can vary a lot, based on the actual runner instance

that executed the job, system noise, invalidated caches, etc. The table above is provided

mostly for t-infra members, for simpler debugging of potential CI slow-downs.
@rust-timer
rust-timer
commented
Mar 2, 2026

Copy link


Copy Markdown

Collaborator
Finished benchmarking commit (e7d90c6): comparison URL.
Overall result: ❌✅ regressions and improvements - please read the text below
Our benchmarks found a performance regression caused by this PR.

This might be an actual regression, but it can also be just noise.
Next Steps:
  • If the regression was expected or you think it can be justified,
    
please write a comment with sufficient written justification, and add
    @rustbot label: +perf-regression-triaged to it, to mark the regression as triaged.
  • If you think that you know of a way to resolve the regression, try to create
    
a new PR with a fix for the regression.
  • If you do not understand the regression or you think that it is just noise,
    
you can ask the @rust-lang/wg-compiler-performance working group for help (members of this group
    
were already notified of this PR).
@rustbot label: +perf-regression

cc @rust-lang/wg-compiler-performance
Instruction count
Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.
meanrangecount
Regressions ❌ 
 (primary)		0.1%[0.1%, 0.1%]1
Regressions ❌ 
 (secondary)		--0
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-0.4%[-0.6%, -0.2%]7
All ❌✅ (primary)0.1%[0.1%, 0.1%]1
Max RSS (memory usage)
Results (secondary -4.0%)
A less reliable metric. May be of interest, but not used to determine the overall result above.
meanrangecount
Regressions ❌ 
 (primary)		--0
Regressions ❌ 
 (secondary)		--0
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-4.0%[-5.4%, -2.6%]3
All ❌✅ (primary)--0
Cycles
This benchmark run did not return any relevant results for this metric.
Binary size
Results (secondary 0.1%)
A less reliable metric. May be of interest, but not used to determine the overall result above.
meanrangecount
Regressions ❌ 
 (primary)		--0
Regressions ❌ 
 (secondary)		0.1%[0.1%, 0.1%]3
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		--0
All ❌✅ (primary)--0
Bootstrap: 480.971s -> 480.149s (-0.17%)
Artifact size: 396.96 MiB -> 396.96 MiB (0.00%)
@Kobzol
Kobzol
commented
Mar 2, 2026•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
The regression seems like noise. Tiny wins on large-workspace, matching pre-merge run.
@rustbot label: +perf-regression-triaged
@rustbotrustbot
added
the 
perf-regression-triaged
The performance regression has been triaged.
label
Mar 2, 2026
@osiewicz
osiewicz
commented
Mar 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Timings for Mac, as requested in OP (with the same scenario as the one exercised in OP):
➜ rustc-153131 hyperfine "rustc +nightly-2026-03-01 main.rs -L deps" -r 30
Benchmark 1: rustc +nightly-2026-03-01 main.rs -L deps
Time (mean ± σ): 215.3 ms ± 1.9 ms [User: 161.2 ms, System: 122.0 ms]
Range (min … max): 212.2 ms … 220.7 ms 30 runs

vs:
➜ rustc-153131 hyperfine "rustc +nightly-2026-03-06 main.rs -L deps" -r 30
Benchmark 1: rustc +nightly-2026-03-06 main.rs -L deps
Time (mean ± σ): 177.9 ms ± 1.6 ms [User: 128.5 ms, System: 120.6 ms]
Range (min … max): 175.2 ms … 183.2 ms 30 runs

And a baseline:
➜ rustc-153131 hyperfine "rustc +nightly-2026-03-06 main.rs" -r 30
Benchmark 1: rustc +nightly-2026-03-06 main.rs
Time (mean ± σ): 68.0 ms ± 1.1 ms [User: 87.0 ms, System: 52.7 ms]
Range (min … max): 66.8 ms … 71.9 ms 30 runs

@Kobzol
Kobzol
deleted the

filesearch-opt

branch
March 14, 2026 10:10
@KobzolKobzol
mentioned this pull request
Mar 15, 2026

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@petrochenkovpetrochenkovpetrochenkov left review comments
@bjorn3bjorn3bjorn3 left review comments
@nnethercotennethercotennethercote approved these changes

Assignees

@nnethercotennethercote

Labels

merged-by-borsThis PR was explicitly merged by bors.perf-regressionPerformance regression.perf-regression-triagedThe performance regression has been triaged.T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.

Projects


None yet


Milestone

1.96.0

Development

Successfully merging this pull request may close these issues.

9 participants

@Kobzol@rust-timer@ehuss@nnethercote@rust-log-analyzer@osiewicz@petrochenkov@bjorn3@rustbot

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Optimize dependency file search - #153131
Merged
rust-bors[bot] merged 4 commits into
rust-lang:mainfrom 
Kobzol:filesearch-opt
Mar 2, 2026
Merged
Optimize dependency file search#153131
rust-bors[bot] merged 4 commits into
rust-lang:mainfrom 
Kobzol:filesearch-opt
Conversation
@Kobzol
@KobzolKobzol
commented
Feb 26, 2026•

edited by rustbot

Loading

Copy link


Copy Markdown

Member
View all comments
I tried to look into the slowdown reported in rust-lang/cargo#16665.
I created a Rust hello world program, and used this Python script to create a directory containing 200k files:
frompathlibimportPathdir=Path("deps")
dir.mkdir(parents=True, exist_ok=True)
foriinrange(200000):
path=dir/f"file{i:07}.o"withopen(path, "w") asf:
f.write("\n")
Then I tried to do various small microoptimalizations and simplifications to the code that iterates the search directories. Each individual commit improved performance, with the third one having the biggest effect.
Here are the results on main vs the last commit with the stage1 compiler on Linux, using hyperfine "rustc +stage1 src/main.rs -L deps" -r 30 (there's IO involved, so it's good to let it run for a while):
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 299.4 ms ± 2.7 ms [User: 161.9 ms, System: 144.9 ms]
Range (min … max): 294.8 ms … 307.1 ms 30 runs
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 208.1 ms ± 4.5 ms [User: 87.3 ms, System: 128.7 ms]
Range (min … max): 202.4 ms … 219.6 ms 30 runs
Would be cool if someone could try this on macOS (maybe @ehuss - not sure if you have macOS or you only commented about its behavior on the Cargo issue :) ).
I also tried to prefilter the paths (not in this PR); right now we load everything and then we filter files with given prefixes, that's wasteful. Filtering just files starting with lib would get us down to ~150ms here. (The baseline without -L is ~80ms on my PC). The rest of the 70ms is essentially allocations from iterating the directory entries and sorting. That would be very hard to change - iterating the directory entries (de)allocates a lot of intermediate paths :( We'd have to implement the iteration by hand with either arena allocation, or at least some better management of memory.
r? @nnethercote
@rustbotrustbot
added

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
T-compiler
Relevant to the compiler team, which will review and decide on the PR/issue.
labels
Feb 26, 2026
@Kobzol
Kobzol
commented
Feb 26, 2026

Copy link


Copy Markdown

MemberAuthor
Not expecting any big wins there, but let's check:
@bors try @rust-timer queue
@rust-timer

This comment has been minimized.

Sign in to view
@rust-bors

This comment has been minimized.

Sign in to view
@rustbotrustbot
added
the 
S-waiting-on-perf
Status: Waiting on a perf run to be completed.
label
Feb 26, 2026
rust-borsBot
pushed a commit
that referenced
this pull request
Feb 26, 2026
@bors
Auto merge of#153131- Kobzol:filesearch-opt, r=<try>
71d4456
Optimize dependency file search
petrochenkov
petrochenkov
reviewed
Feb 26, 2026
View reviewed changes
Comment threadcompiler/rustc_session/src/search_paths.rs
@rust-bors
rust-borsBot
commented
Feb 26, 2026

Copy link


Copy Markdown

Contributor
☀️ Try build successful (CI)

Build commit: 71d4456 (71d445654a32204fbf86bfec11e657fb97adcd22, parent: 69b78537fac74de40f009b076bcbbf54b77683ad)
@rust-timer

This comment has been minimized.

Sign in to view
@ehuss
ehuss
commented
Feb 26, 2026

Copy link


Copy Markdown

Contributor
Unfortunately I am unable to test because of #153077, I am unable to get rustc to build.
@rust-timer
rust-timer
commented
Feb 26, 2026

Copy link


Copy Markdown

Collaborator
Finished benchmarking commit (71d4456): comparison URL.
Overall result: ❌✅ regressions and improvements - please read the text below
Benchmarking this pull request means it may be perf-sensitive – we'll automatically label it not fit for rolling up. You can override this, but we strongly advise not to, due to possible changes in compiler perf.
Next Steps: If you can justify the regressions found in this try perf run, please do so in sufficient writing along with @rustbot label: +perf-regression-triaged. If not, please fix the regressions and do another perf run. If its results are neutral or positive, the label will be automatically removed.
@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression
Instruction count
Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.
meanrangecount
Regressions ❌ 
 (primary)		0.3%[0.2%, 0.3%]4
Regressions ❌ 
 (secondary)		0.0%[0.0%, 0.0%]1
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-0.4%[-0.6%, -0.1%]7
All ❌✅ (primary)0.3%[0.2%, 0.3%]4
Max RSS (memory usage)
Results (secondary 0.1%)
A less reliable metric. May be of interest, but not used to determine the overall result above.
meanrangecount
Regressions ❌ 
 (primary)		--0
Regressions ❌ 
 (secondary)		6.8%[6.8%, 6.8%]1
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-6.7%[-6.7%, -6.7%]1
All ❌✅ (primary)--0
Cycles
This benchmark run did not return any relevant results for this metric.
Binary size
This benchmark run did not return any relevant results for this metric.
Bootstrap: 480.132s -> 480.258s (0.03%)
Artifact size: 395.80 MiB -> 395.77 MiB (-0.01%)
@rustbotrustbot
added

perf-regression
Performance regression.
and removed

S-waiting-on-perf
Status: Waiting on a perf run to be completed.
labels
Feb 26, 2026
@ehuss
ehuss
commented
Feb 26, 2026

Copy link


Copy Markdown

Contributor
OK, got some data:
With this PR:
Benchmark 1: ~/Proj/rust/rust/build/aarch64-apple-darwin/stage1/bin/rustc src/main.rs -L deps
Time (mean ± σ): 197.4 ms ± 3.9 ms [User: 143.7 ms, System: 117.9 ms]
Range (min … max): 192.4 ms … 207.7 ms 30 runs

Without:
Benchmark 1: ~/Proj/rust/rust/build/aarch64-apple-darwin/stage1/bin/rustc src/main.rs -L deps
Time (mean ± σ): 248.9 ms ± 5.1 ms [User: 189.5 ms, System: 124.2 ms]
Range (min … max): 241.1 ms … 262.6 ms 30 runs

The tests are super noisy, though (I get significantly different results between runs). But it seems like a pretty good win.
@nnethercote
nnethercote
commented
Feb 27, 2026

Copy link


Copy Markdown

Contributor
Nice wins.
@bors r+
@rust-bors
rust-borsBot
commented
Feb 27, 2026

Copy link


Copy Markdown

Contributor
📌 Commit f450289 has been approved by nnethercote
It is now in the queue for this repository.
@rust-borsrust-borsBot
added

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
and removed

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
labels
Feb 27, 2026
bjorn3
bjorn3
reviewed
Feb 27, 2026
View reviewed changes
Comment threadcompiler/rustc_session/src/search_paths.rs
 Err(..) => Default::default(),
 };
 files.sort_by(|lhs, rhs| lhs.file_name_str.cmp(&rhs.file_name_str));
 files.sort_unstable_by(|lhs, rhs| lhs.file_name_str.cmp(&rhs.file_name_str));
@bjorn3bjorn3Feb 27, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there no possibility of two files with the same file_name_str?
@bjorn3bjorn3Feb 27, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although I guess that would result in either one being picked deterministically (if it is the only matching one), which one is picked not mattering (if both files having the same crate hash) or an error about multiple candidates getting emitted.
@KobzolKobzolFeb 27, 2026

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We sorted (and searched) by the UTF-8 name before anyway, so there shouldn't hopefully be any change in behavior.
@rust-bors

This comment has been minimized.

Sign in to view
rust-borsBot
pushed a commit
that referenced
this pull request
Mar 1, 2026
@bors
Auto merge of#153131- Kobzol:filesearch-opt, r=nnethercote
07af97d
Optimize dependency file search
I tried to look into the slowdown reported in rust-lang/cargo#16665.
I created a Rust hello world program, and used this Python script to create a directory containing 200k files:
```python
from pathlib import Path
dir = Path("deps")
dir.mkdir(parents=True, exist_ok=True)
for i in range(200000):
path = dir / f"file{i:07}.o"
with open(path, "w") as f:
f.write("\n")
```
Then I tried to do various small microoptimalizations and simplifications to the code that iterates the search directories. Each individual commit improved performance, with the third one having the biggest effect.
Here are the results on `main` vs the last commit with the stage1 compiler on Linux, using `hyperfine "rustc +stage1 src/main.rs -L deps" -r 30` (there's IO involved, so it's good to let it run for a while):
```bash
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 299.4 ms ± 2.7 ms [User: 161.9 ms, System: 144.9 ms]
Range (min … max): 294.8 ms … 307.1 ms 30 runs
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 208.1 ms ± 4.5 ms [User: 87.3 ms, System: 128.7 ms]
Range (min … max): 202.4 ms … 219.6 ms 30 runs
```
Would be cool if someone could try this on macOS (maybe @ehuss - not sure if you have macOS or you only commented about its behavior on the Cargo issue :) ).
I also tried to prefilter the paths (not in this PR); right now we load everything and then we filter files with given prefixes, that's wasteful. Filtering just files starting with `lib` would get us down to ~150ms here. (The baseline without `-L` is ~80ms on my PC). The rest of the 70ms is essentially allocations from iterating the directory entries and sorting. That would be very hard to change - iterating the directory entries (de)allocates a lot of intermediate paths :( We'd have to implement the iteration by hand with either arena allocation, or at least some better management of memory.
r? @nnethercote
@rust-borsrust-borsBot
added

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
and removed

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
labels
Mar 1, 2026
@rust-bors
rust-borsBot
commented
Mar 1, 2026

Copy link


Copy Markdown

Contributor
💔 Test for 07af97d failed: CI. Failed job:
@rust-log-analyzer
rust-log-analyzer
commented
Mar 1, 2026

Copy link


Copy Markdown

Collaborator
The job aarch64-gnu failed! Check out the build log: (web)(plain enhanced)(plain)
Click to see the possible cause of the failure (guessed by this bot)
/dev/sda15 98M 6.4M 92M 7% /boot/efi
tmpfs 1.6G 16K 1.6G 1% /run/user/1001
================================================================================
Sufficient disk space available (120307208KB >= 52428800KB). Skipping cleanup.
##[group]Run src/ci/scripts/setup-environment.sh
src/ci/scripts/setup-environment.sh
shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
---
[1773/3894] Building ARMGenMCCodeEmitter.inc...
##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
[1774/3894] Building ARMGenMCPseudoLowering.inc...
Session terminated, killing shell...::group::Clock drift check
local time: Sun Mar 1 08:40:58 UTC 2026
network time: [1775/3894] Building ARMGenInstrInfo.inc...
Sun, 01 Mar 2026 08:40:58 GMT
##[endgroup]
[1776/3894] Building ARMGenRegisterBank.inc...

@Kobzol
Kobzol
commented
Mar 1, 2026

Copy link


Copy Markdown

MemberAuthor
@bors retry
Spurious worker restart.
@rust-borsrust-borsBot
added

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
and removed

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
labels
Mar 1, 2026

This was referenced Mar 1, 2026
 Merged

 Merged

@rust-bors

This comment has been minimized.

Sign in to view
@rust-borsrust-borsBot
added

merged-by-bors
This PR was explicitly merged by bors.
and removed

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
labels
Mar 2, 2026
@rust-bors
rust-borsBot
commented
Mar 2, 2026

Copy link


Copy Markdown

Contributor
☀️ Test successful - CI

Approved by: nnethercote

Duration: 3h 40m 3s

Pushing e7d90c6 to main...
@rust-bors
rust-borsBot
merged commit e7d90c6
into
rust-lang:mainMar 2, 2026

13 checks passed

@rustbotrustbot
added this to the 1.96.0 milestone
Mar 2, 2026
@github-actions
github-actionsBot
commented
Mar 2, 2026

Copy link


Copy Markdown

Contributor
What is this?
This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 8038127 (parent) -> e7d90c6 (this PR)
Test differences
Show 2 test diffs

2 doctest diffs were found. These are ignored, as they are noisy.
Test dashboard
Run
cargo run --manifest-path src/ci/citool/Cargo.toml -- \
test-dashboard e7d90c695a39426baf5ae705de2f9570a72229de --output-dir test-dashboard
And then open test-dashboard/index.html in your browser to see an overview of all executed tests.
Job duration changes
  1. dist-aarch64-llvm-mingw: 1h 33m -> 1h 46m (+14.1%)
  2. dist-aarch64-apple: 1h 56m -> 2h 12m (+13.1%)
  3. aarch64-apple: 3h 8m -> 3h 33m (+13.0%)
  4. dist-x86_64-apple: 2h 42m -> 2h 25m (-10.7%)
  5. arm-android: 1h 37m -> 1h 47m (+10.4%)
  6. dist-ohos-x86_64: 1h 14m -> 1h 21m (+9.7%)
  7. x86_64-gnu-aux: 2h 7m -> 2h 18m (+8.5%)
  8. x86_64-gnu-distcheck: 2h 7m -> 2h 18m (+8.4%)
  9. x86_64-gnu-miri: 1h 23m -> 1h 30m (+7.7%)
  10. dist-arm-linux-musl: 1h 38m -> 1h 30m (-7.3%)
How to interpret the job duration changes?
Job durations can vary a lot, based on the actual runner instance

that executed the job, system noise, invalidated caches, etc. The table above is provided

mostly for t-infra members, for simpler debugging of potential CI slow-downs.
@rust-timer
rust-timer
commented
Mar 2, 2026

Copy link


Copy Markdown

Collaborator
Finished benchmarking commit (e7d90c6): comparison URL.
Overall result: ❌✅ regressions and improvements - please read the text below
Our benchmarks found a performance regression caused by this PR.

This might be an actual regression, but it can also be just noise.
Next Steps:
  • If the regression was expected or you think it can be justified,
    
please write a comment with sufficient written justification, and add
    @rustbot label: +perf-regression-triaged to it, to mark the regression as triaged.
  • If you think that you know of a way to resolve the regression, try to create
    
a new PR with a fix for the regression.
  • If you do not understand the regression or you think that it is just noise,
    
you can ask the @rust-lang/wg-compiler-performance working group for help (members of this group
    
were already notified of this PR).
@rustbot label: +perf-regression

cc @rust-lang/wg-compiler-performance
Instruction count
Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.
meanrangecount
Regressions ❌ 
 (primary)		0.1%[0.1%, 0.1%]1
Regressions ❌ 
 (secondary)		--0
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-0.4%[-0.6%, -0.2%]7
All ❌✅ (primary)0.1%[0.1%, 0.1%]1
Max RSS (memory usage)
Results (secondary -4.0%)
A less reliable metric. May be of interest, but not used to determine the overall result above.
meanrangecount
Regressions ❌ 
 (primary)		--0
Regressions ❌ 
 (secondary)		--0
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-4.0%[-5.4%, -2.6%]3
All ❌✅ (primary)--0
Cycles
This benchmark run did not return any relevant results for this metric.
Binary size
Results (secondary 0.1%)
A less reliable metric. May be of interest, but not used to determine the overall result above.
meanrangecount
Regressions ❌ 
 (primary)		--0
Regressions ❌ 
 (secondary)		0.1%[0.1%, 0.1%]3
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		--0
All ❌✅ (primary)--0
Bootstrap: 480.971s -> 480.149s (-0.17%)
Artifact size: 396.96 MiB -> 396.96 MiB (0.00%)
@Kobzol
Kobzol
commented
Mar 2, 2026•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
The regression seems like noise. Tiny wins on large-workspace, matching pre-merge run.
@rustbot label: +perf-regression-triaged
@rustbotrustbot
added
the 
perf-regression-triaged
The performance regression has been triaged.
label
Mar 2, 2026
@osiewicz
osiewicz
commented
Mar 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Timings for Mac, as requested in OP (with the same scenario as the one exercised in OP):
➜ rustc-153131 hyperfine "rustc +nightly-2026-03-01 main.rs -L deps" -r 30
Benchmark 1: rustc +nightly-2026-03-01 main.rs -L deps
Time (mean ± σ): 215.3 ms ± 1.9 ms [User: 161.2 ms, System: 122.0 ms]
Range (min … max): 212.2 ms … 220.7 ms 30 runs

vs:
➜ rustc-153131 hyperfine "rustc +nightly-2026-03-06 main.rs -L deps" -r 30
Benchmark 1: rustc +nightly-2026-03-06 main.rs -L deps
Time (mean ± σ): 177.9 ms ± 1.6 ms [User: 128.5 ms, System: 120.6 ms]
Range (min … max): 175.2 ms … 183.2 ms 30 runs

And a baseline:
➜ rustc-153131 hyperfine "rustc +nightly-2026-03-06 main.rs" -r 30
Benchmark 1: rustc +nightly-2026-03-06 main.rs
Time (mean ± σ): 68.0 ms ± 1.1 ms [User: 87.0 ms, System: 52.7 ms]
Range (min … max): 66.8 ms … 71.9 ms 30 runs

@Kobzol
Kobzol
deleted the

filesearch-opt

branch
March 14, 2026 10:10
@KobzolKobzol
mentioned this pull request
Mar 15, 2026

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@petrochenkovpetrochenkovpetrochenkov left review comments
@bjorn3bjorn3bjorn3 left review comments
@nnethercotennethercotennethercote approved these changes

Assignees

@nnethercotennethercote

Labels

merged-by-borsThis PR was explicitly merged by bors.perf-regressionPerformance regression.perf-regression-triagedThe performance regression has been triaged.T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.

Projects


None yet


Milestone

1.96.0

Development

Successfully merging this pull request may close these issues.

9 participants

@Kobzol@rust-timer@ehuss@nnethercote@rust-log-analyzer@osiewicz@petrochenkov@bjorn3@rustbot

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Optimize dependency file search - #153131
Merged
rust-bors[bot] merged 4 commits into
rust-lang:mainfrom 
Kobzol:filesearch-opt
Mar 2, 2026
Merged
Optimize dependency file search#153131
rust-bors[bot] merged 4 commits into
rust-lang:mainfrom 
Kobzol:filesearch-opt
Conversation
@Kobzol
@KobzolKobzol
commented
Feb 26, 2026•

edited by rustbot

Loading

Copy link


Copy Markdown

Member
View all comments
I tried to look into the slowdown reported in rust-lang/cargo#16665.
I created a Rust hello world program, and used this Python script to create a directory containing 200k files:
frompathlibimportPathdir=Path("deps")
dir.mkdir(parents=True, exist_ok=True)
foriinrange(200000):
path=dir/f"file{i:07}.o"withopen(path, "w") asf:
f.write("\n")
Then I tried to do various small microoptimalizations and simplifications to the code that iterates the search directories. Each individual commit improved performance, with the third one having the biggest effect.
Here are the results on main vs the last commit with the stage1 compiler on Linux, using hyperfine "rustc +stage1 src/main.rs -L deps" -r 30 (there's IO involved, so it's good to let it run for a while):
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 299.4 ms ± 2.7 ms [User: 161.9 ms, System: 144.9 ms]
Range (min … max): 294.8 ms … 307.1 ms 30 runs
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 208.1 ms ± 4.5 ms [User: 87.3 ms, System: 128.7 ms]
Range (min … max): 202.4 ms … 219.6 ms 30 runs
Would be cool if someone could try this on macOS (maybe @ehuss - not sure if you have macOS or you only commented about its behavior on the Cargo issue :) ).
I also tried to prefilter the paths (not in this PR); right now we load everything and then we filter files with given prefixes, that's wasteful. Filtering just files starting with lib would get us down to ~150ms here. (The baseline without -L is ~80ms on my PC). The rest of the 70ms is essentially allocations from iterating the directory entries and sorting. That would be very hard to change - iterating the directory entries (de)allocates a lot of intermediate paths :( We'd have to implement the iteration by hand with either arena allocation, or at least some better management of memory.
r? @nnethercote
@rustbotrustbot
added

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
T-compiler
Relevant to the compiler team, which will review and decide on the PR/issue.
labels
Feb 26, 2026
@Kobzol
Kobzol
commented
Feb 26, 2026

Copy link


Copy Markdown

MemberAuthor
Not expecting any big wins there, but let's check:
@bors try @rust-timer queue
@rust-timer

This comment has been minimized.

Sign in to view
@rust-bors

This comment has been minimized.

Sign in to view
@rustbotrustbot
added
the 
S-waiting-on-perf
Status: Waiting on a perf run to be completed.
label
Feb 26, 2026
rust-borsBot
pushed a commit
that referenced
this pull request
Feb 26, 2026
@bors
Auto merge of#153131- Kobzol:filesearch-opt, r=<try>
71d4456
Optimize dependency file search
petrochenkov
petrochenkov
reviewed
Feb 26, 2026
View reviewed changes
Comment threadcompiler/rustc_session/src/search_paths.rs
@rust-bors
rust-borsBot
commented
Feb 26, 2026

Copy link


Copy Markdown

Contributor
☀️ Try build successful (CI)

Build commit: 71d4456 (71d445654a32204fbf86bfec11e657fb97adcd22, parent: 69b78537fac74de40f009b076bcbbf54b77683ad)
@rust-timer

This comment has been minimized.

Sign in to view
@ehuss
ehuss
commented
Feb 26, 2026

Copy link


Copy Markdown

Contributor
Unfortunately I am unable to test because of #153077, I am unable to get rustc to build.
@rust-timer
rust-timer
commented
Feb 26, 2026

Copy link


Copy Markdown

Collaborator
Finished benchmarking commit (71d4456): comparison URL.
Overall result: ❌✅ regressions and improvements - please read the text below
Benchmarking this pull request means it may be perf-sensitive – we'll automatically label it not fit for rolling up. You can override this, but we strongly advise not to, due to possible changes in compiler perf.
Next Steps: If you can justify the regressions found in this try perf run, please do so in sufficient writing along with @rustbot label: +perf-regression-triaged. If not, please fix the regressions and do another perf run. If its results are neutral or positive, the label will be automatically removed.
@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression
Instruction count
Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.
meanrangecount
Regressions ❌ 
 (primary)		0.3%[0.2%, 0.3%]4
Regressions ❌ 
 (secondary)		0.0%[0.0%, 0.0%]1
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-0.4%[-0.6%, -0.1%]7
All ❌✅ (primary)0.3%[0.2%, 0.3%]4
Max RSS (memory usage)
Results (secondary 0.1%)
A less reliable metric. May be of interest, but not used to determine the overall result above.
meanrangecount
Regressions ❌ 
 (primary)		--0
Regressions ❌ 
 (secondary)		6.8%[6.8%, 6.8%]1
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-6.7%[-6.7%, -6.7%]1
All ❌✅ (primary)--0
Cycles
This benchmark run did not return any relevant results for this metric.
Binary size
This benchmark run did not return any relevant results for this metric.
Bootstrap: 480.132s -> 480.258s (0.03%)
Artifact size: 395.80 MiB -> 395.77 MiB (-0.01%)
@rustbotrustbot
added

perf-regression
Performance regression.
and removed

S-waiting-on-perf
Status: Waiting on a perf run to be completed.
labels
Feb 26, 2026
@ehuss
ehuss
commented
Feb 26, 2026

Copy link


Copy Markdown

Contributor
OK, got some data:
With this PR:
Benchmark 1: ~/Proj/rust/rust/build/aarch64-apple-darwin/stage1/bin/rustc src/main.rs -L deps
Time (mean ± σ): 197.4 ms ± 3.9 ms [User: 143.7 ms, System: 117.9 ms]
Range (min … max): 192.4 ms … 207.7 ms 30 runs

Without:
Benchmark 1: ~/Proj/rust/rust/build/aarch64-apple-darwin/stage1/bin/rustc src/main.rs -L deps
Time (mean ± σ): 248.9 ms ± 5.1 ms [User: 189.5 ms, System: 124.2 ms]
Range (min … max): 241.1 ms … 262.6 ms 30 runs

The tests are super noisy, though (I get significantly different results between runs). But it seems like a pretty good win.
@nnethercote
nnethercote
commented
Feb 27, 2026

Copy link


Copy Markdown

Contributor
Nice wins.
@bors r+
@rust-bors
rust-borsBot
commented
Feb 27, 2026

Copy link


Copy Markdown

Contributor
📌 Commit f450289 has been approved by nnethercote
It is now in the queue for this repository.
@rust-borsrust-borsBot
added

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
and removed

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
labels
Feb 27, 2026
bjorn3
bjorn3
reviewed
Feb 27, 2026
View reviewed changes
Comment threadcompiler/rustc_session/src/search_paths.rs
 Err(..) => Default::default(),
 };
 files.sort_by(|lhs, rhs| lhs.file_name_str.cmp(&rhs.file_name_str));
 files.sort_unstable_by(|lhs, rhs| lhs.file_name_str.cmp(&rhs.file_name_str));
@bjorn3bjorn3Feb 27, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there no possibility of two files with the same file_name_str?
@bjorn3bjorn3Feb 27, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although I guess that would result in either one being picked deterministically (if it is the only matching one), which one is picked not mattering (if both files having the same crate hash) or an error about multiple candidates getting emitted.
@KobzolKobzolFeb 27, 2026

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We sorted (and searched) by the UTF-8 name before anyway, so there shouldn't hopefully be any change in behavior.
@rust-bors

This comment has been minimized.

Sign in to view
rust-borsBot
pushed a commit
that referenced
this pull request
Mar 1, 2026
@bors
Auto merge of#153131- Kobzol:filesearch-opt, r=nnethercote
07af97d
Optimize dependency file search
I tried to look into the slowdown reported in rust-lang/cargo#16665.
I created a Rust hello world program, and used this Python script to create a directory containing 200k files:
```python
from pathlib import Path
dir = Path("deps")
dir.mkdir(parents=True, exist_ok=True)
for i in range(200000):
path = dir / f"file{i:07}.o"
with open(path, "w") as f:
f.write("\n")
```
Then I tried to do various small microoptimalizations and simplifications to the code that iterates the search directories. Each individual commit improved performance, with the third one having the biggest effect.
Here are the results on `main` vs the last commit with the stage1 compiler on Linux, using `hyperfine "rustc +stage1 src/main.rs -L deps" -r 30` (there's IO involved, so it's good to let it run for a while):
```bash
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 299.4 ms ± 2.7 ms [User: 161.9 ms, System: 144.9 ms]
Range (min … max): 294.8 ms … 307.1 ms 30 runs
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 208.1 ms ± 4.5 ms [User: 87.3 ms, System: 128.7 ms]
Range (min … max): 202.4 ms … 219.6 ms 30 runs
```
Would be cool if someone could try this on macOS (maybe @ehuss - not sure if you have macOS or you only commented about its behavior on the Cargo issue :) ).
I also tried to prefilter the paths (not in this PR); right now we load everything and then we filter files with given prefixes, that's wasteful. Filtering just files starting with `lib` would get us down to ~150ms here. (The baseline without `-L` is ~80ms on my PC). The rest of the 70ms is essentially allocations from iterating the directory entries and sorting. That would be very hard to change - iterating the directory entries (de)allocates a lot of intermediate paths :( We'd have to implement the iteration by hand with either arena allocation, or at least some better management of memory.
r? @nnethercote
@rust-borsrust-borsBot
added

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
and removed

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
labels
Mar 1, 2026
@rust-bors
rust-borsBot
commented
Mar 1, 2026

Copy link


Copy Markdown

Contributor
💔 Test for 07af97d failed: CI. Failed job:
@rust-log-analyzer
rust-log-analyzer
commented
Mar 1, 2026

Copy link


Copy Markdown

Collaborator
The job aarch64-gnu failed! Check out the build log: (web)(plain enhanced)(plain)
Click to see the possible cause of the failure (guessed by this bot)
/dev/sda15 98M 6.4M 92M 7% /boot/efi
tmpfs 1.6G 16K 1.6G 1% /run/user/1001
================================================================================
Sufficient disk space available (120307208KB >= 52428800KB). Skipping cleanup.
##[group]Run src/ci/scripts/setup-environment.sh
src/ci/scripts/setup-environment.sh
shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
---
[1773/3894] Building ARMGenMCCodeEmitter.inc...
##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
[1774/3894] Building ARMGenMCPseudoLowering.inc...
Session terminated, killing shell...::group::Clock drift check
local time: Sun Mar 1 08:40:58 UTC 2026
network time: [1775/3894] Building ARMGenInstrInfo.inc...
Sun, 01 Mar 2026 08:40:58 GMT
##[endgroup]
[1776/3894] Building ARMGenRegisterBank.inc...

@Kobzol
Kobzol
commented
Mar 1, 2026

Copy link


Copy Markdown

MemberAuthor
@bors retry
Spurious worker restart.
@rust-borsrust-borsBot
added

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
and removed

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
labels
Mar 1, 2026

This was referenced Mar 1, 2026
 Merged

 Merged

@rust-bors

This comment has been minimized.

Sign in to view
@rust-borsrust-borsBot
added

merged-by-bors
This PR was explicitly merged by bors.
and removed

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
labels
Mar 2, 2026
@rust-bors
rust-borsBot
commented
Mar 2, 2026

Copy link


Copy Markdown

Contributor
☀️ Test successful - CI

Approved by: nnethercote

Duration: 3h 40m 3s

Pushing e7d90c6 to main...
@rust-bors
rust-borsBot
merged commit e7d90c6
into
rust-lang:mainMar 2, 2026

13 checks passed

@rustbotrustbot
added this to the 1.96.0 milestone
Mar 2, 2026
@github-actions
github-actionsBot
commented
Mar 2, 2026

Copy link


Copy Markdown

Contributor
What is this?
This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 8038127 (parent) -> e7d90c6 (this PR)
Test differences
Show 2 test diffs

2 doctest diffs were found. These are ignored, as they are noisy.
Test dashboard
Run
cargo run --manifest-path src/ci/citool/Cargo.toml -- \
test-dashboard e7d90c695a39426baf5ae705de2f9570a72229de --output-dir test-dashboard
And then open test-dashboard/index.html in your browser to see an overview of all executed tests.
Job duration changes
  1. dist-aarch64-llvm-mingw: 1h 33m -> 1h 46m (+14.1%)
  2. dist-aarch64-apple: 1h 56m -> 2h 12m (+13.1%)
  3. aarch64-apple: 3h 8m -> 3h 33m (+13.0%)
  4. dist-x86_64-apple: 2h 42m -> 2h 25m (-10.7%)
  5. arm-android: 1h 37m -> 1h 47m (+10.4%)
  6. dist-ohos-x86_64: 1h 14m -> 1h 21m (+9.7%)
  7. x86_64-gnu-aux: 2h 7m -> 2h 18m (+8.5%)
  8. x86_64-gnu-distcheck: 2h 7m -> 2h 18m (+8.4%)
  9. x86_64-gnu-miri: 1h 23m -> 1h 30m (+7.7%)
  10. dist-arm-linux-musl: 1h 38m -> 1h 30m (-7.3%)
How to interpret the job duration changes?
Job durations can vary a lot, based on the actual runner instance

that executed the job, system noise, invalidated caches, etc. The table above is provided

mostly for t-infra members, for simpler debugging of potential CI slow-downs.
@rust-timer
rust-timer
commented
Mar 2, 2026

Copy link


Copy Markdown

Collaborator
Finished benchmarking commit (e7d90c6): comparison URL.
Overall result: ❌✅ regressions and improvements - please read the text below
Our benchmarks found a performance regression caused by this PR.

This might be an actual regression, but it can also be just noise.
Next Steps:
  • If the regression was expected or you think it can be justified,
    
please write a comment with sufficient written justification, and add
    @rustbot label: +perf-regression-triaged to it, to mark the regression as triaged.
  • If you think that you know of a way to resolve the regression, try to create
    
a new PR with a fix for the regression.
  • If you do not understand the regression or you think that it is just noise,
    
you can ask the @rust-lang/wg-compiler-performance working group for help (members of this group
    
were already notified of this PR).
@rustbot label: +perf-regression

cc @rust-lang/wg-compiler-performance
Instruction count
Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.
meanrangecount
Regressions ❌ 
 (primary)		0.1%[0.1%, 0.1%]1
Regressions ❌ 
 (secondary)		--0
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-0.4%[-0.6%, -0.2%]7
All ❌✅ (primary)0.1%[0.1%, 0.1%]1
Max RSS (memory usage)
Results (secondary -4.0%)
A less reliable metric. May be of interest, but not used to determine the overall result above.
meanrangecount
Regressions ❌ 
 (primary)		--0
Regressions ❌ 
 (secondary)		--0
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-4.0%[-5.4%, -2.6%]3
All ❌✅ (primary)--0
Cycles
This benchmark run did not return any relevant results for this metric.
Binary size
Results (secondary 0.1%)
A less reliable metric. May be of interest, but not used to determine the overall result above.
meanrangecount
Regressions ❌ 
 (primary)		--0
Regressions ❌ 
 (secondary)		0.1%[0.1%, 0.1%]3
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		--0
All ❌✅ (primary)--0
Bootstrap: 480.971s -> 480.149s (-0.17%)
Artifact size: 396.96 MiB -> 396.96 MiB (0.00%)
@Kobzol
Kobzol
commented
Mar 2, 2026•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
The regression seems like noise. Tiny wins on large-workspace, matching pre-merge run.
@rustbot label: +perf-regression-triaged
@rustbotrustbot
added
the 
perf-regression-triaged
The performance regression has been triaged.
label
Mar 2, 2026
@osiewicz
osiewicz
commented
Mar 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Timings for Mac, as requested in OP (with the same scenario as the one exercised in OP):
➜ rustc-153131 hyperfine "rustc +nightly-2026-03-01 main.rs -L deps" -r 30
Benchmark 1: rustc +nightly-2026-03-01 main.rs -L deps
Time (mean ± σ): 215.3 ms ± 1.9 ms [User: 161.2 ms, System: 122.0 ms]
Range (min … max): 212.2 ms … 220.7 ms 30 runs

vs:
➜ rustc-153131 hyperfine "rustc +nightly-2026-03-06 main.rs -L deps" -r 30
Benchmark 1: rustc +nightly-2026-03-06 main.rs -L deps
Time (mean ± σ): 177.9 ms ± 1.6 ms [User: 128.5 ms, System: 120.6 ms]
Range (min … max): 175.2 ms … 183.2 ms 30 runs

And a baseline:
➜ rustc-153131 hyperfine "rustc +nightly-2026-03-06 main.rs" -r 30
Benchmark 1: rustc +nightly-2026-03-06 main.rs
Time (mean ± σ): 68.0 ms ± 1.1 ms [User: 87.0 ms, System: 52.7 ms]
Range (min … max): 66.8 ms … 71.9 ms 30 runs

@Kobzol
Kobzol
deleted the

filesearch-opt

branch
March 14, 2026 10:10
@KobzolKobzol
mentioned this pull request
Mar 15, 2026

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@petrochenkovpetrochenkovpetrochenkov left review comments
@bjorn3bjorn3bjorn3 left review comments
@nnethercotennethercotennethercote approved these changes

Assignees

@nnethercotennethercote

Labels

merged-by-borsThis PR was explicitly merged by bors.perf-regressionPerformance regression.perf-regression-triagedThe performance regression has been triaged.T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.

Projects


None yet


Milestone

1.96.0

Development

Successfully merging this pull request may close these issues.

9 participants

@Kobzol@rust-timer@ehuss@nnethercote@rust-log-analyzer@osiewicz@petrochenkov@bjorn3@rustbot

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Optimize dependency file search - #153131
Merged
rust-bors[bot] merged 4 commits into
rust-lang:mainfrom 
Kobzol:filesearch-opt
Mar 2, 2026
Merged
Optimize dependency file search#153131
rust-bors[bot] merged 4 commits into
rust-lang:mainfrom 
Kobzol:filesearch-opt
Conversation
@Kobzol
@KobzolKobzol
commented
Feb 26, 2026•

edited by rustbot

Loading

Copy link


Copy Markdown

Member
View all comments
I tried to look into the slowdown reported in rust-lang/cargo#16665.
I created a Rust hello world program, and used this Python script to create a directory containing 200k files:
frompathlibimportPathdir=Path("deps")
dir.mkdir(parents=True, exist_ok=True)
foriinrange(200000):
path=dir/f"file{i:07}.o"withopen(path, "w") asf:
f.write("\n")
Then I tried to do various small microoptimalizations and simplifications to the code that iterates the search directories. Each individual commit improved performance, with the third one having the biggest effect.
Here are the results on main vs the last commit with the stage1 compiler on Linux, using hyperfine "rustc +stage1 src/main.rs -L deps" -r 30 (there's IO involved, so it's good to let it run for a while):
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 299.4 ms ± 2.7 ms [User: 161.9 ms, System: 144.9 ms]
Range (min … max): 294.8 ms … 307.1 ms 30 runs
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 208.1 ms ± 4.5 ms [User: 87.3 ms, System: 128.7 ms]
Range (min … max): 202.4 ms … 219.6 ms 30 runs
Would be cool if someone could try this on macOS (maybe @ehuss - not sure if you have macOS or you only commented about its behavior on the Cargo issue :) ).
I also tried to prefilter the paths (not in this PR); right now we load everything and then we filter files with given prefixes, that's wasteful. Filtering just files starting with lib would get us down to ~150ms here. (The baseline without -L is ~80ms on my PC). The rest of the 70ms is essentially allocations from iterating the directory entries and sorting. That would be very hard to change - iterating the directory entries (de)allocates a lot of intermediate paths :( We'd have to implement the iteration by hand with either arena allocation, or at least some better management of memory.
r? @nnethercote
@rustbotrustbot
added

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
T-compiler
Relevant to the compiler team, which will review and decide on the PR/issue.
labels
Feb 26, 2026
@Kobzol
Kobzol
commented
Feb 26, 2026

Copy link


Copy Markdown

MemberAuthor
Not expecting any big wins there, but let's check:
@bors try @rust-timer queue
@rust-timer

This comment has been minimized.

Sign in to view
@rust-bors

This comment has been minimized.

Sign in to view
@rustbotrustbot
added
the 
S-waiting-on-perf
Status: Waiting on a perf run to be completed.
label
Feb 26, 2026
rust-borsBot
pushed a commit
that referenced
this pull request
Feb 26, 2026
@bors
Auto merge of#153131- Kobzol:filesearch-opt, r=<try>
71d4456
Optimize dependency file search
petrochenkov
petrochenkov
reviewed
Feb 26, 2026
View reviewed changes
Comment threadcompiler/rustc_session/src/search_paths.rs
@rust-bors
rust-borsBot
commented
Feb 26, 2026

Copy link


Copy Markdown

Contributor
☀️ Try build successful (CI)

Build commit: 71d4456 (71d445654a32204fbf86bfec11e657fb97adcd22, parent: 69b78537fac74de40f009b076bcbbf54b77683ad)
@rust-timer

This comment has been minimized.

Sign in to view
@ehuss
ehuss
commented
Feb 26, 2026

Copy link


Copy Markdown

Contributor
Unfortunately I am unable to test because of #153077, I am unable to get rustc to build.
@rust-timer
rust-timer
commented
Feb 26, 2026

Copy link


Copy Markdown

Collaborator
Finished benchmarking commit (71d4456): comparison URL.
Overall result: ❌✅ regressions and improvements - please read the text below
Benchmarking this pull request means it may be perf-sensitive – we'll automatically label it not fit for rolling up. You can override this, but we strongly advise not to, due to possible changes in compiler perf.
Next Steps: If you can justify the regressions found in this try perf run, please do so in sufficient writing along with @rustbot label: +perf-regression-triaged. If not, please fix the regressions and do another perf run. If its results are neutral or positive, the label will be automatically removed.
@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression
Instruction count
Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.
meanrangecount
Regressions ❌ 
 (primary)		0.3%[0.2%, 0.3%]4
Regressions ❌ 
 (secondary)		0.0%[0.0%, 0.0%]1
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-0.4%[-0.6%, -0.1%]7
All ❌✅ (primary)0.3%[0.2%, 0.3%]4
Max RSS (memory usage)
Results (secondary 0.1%)
A less reliable metric. May be of interest, but not used to determine the overall result above.
meanrangecount
Regressions ❌ 
 (primary)		--0
Regressions ❌ 
 (secondary)		6.8%[6.8%, 6.8%]1
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-6.7%[-6.7%, -6.7%]1
All ❌✅ (primary)--0
Cycles
This benchmark run did not return any relevant results for this metric.
Binary size
This benchmark run did not return any relevant results for this metric.
Bootstrap: 480.132s -> 480.258s (0.03%)
Artifact size: 395.80 MiB -> 395.77 MiB (-0.01%)
@rustbotrustbot
added

perf-regression
Performance regression.
and removed

S-waiting-on-perf
Status: Waiting on a perf run to be completed.
labels
Feb 26, 2026
@ehuss
ehuss
commented
Feb 26, 2026

Copy link


Copy Markdown

Contributor
OK, got some data:
With this PR:
Benchmark 1: ~/Proj/rust/rust/build/aarch64-apple-darwin/stage1/bin/rustc src/main.rs -L deps
Time (mean ± σ): 197.4 ms ± 3.9 ms [User: 143.7 ms, System: 117.9 ms]
Range (min … max): 192.4 ms … 207.7 ms 30 runs

Without:
Benchmark 1: ~/Proj/rust/rust/build/aarch64-apple-darwin/stage1/bin/rustc src/main.rs -L deps
Time (mean ± σ): 248.9 ms ± 5.1 ms [User: 189.5 ms, System: 124.2 ms]
Range (min … max): 241.1 ms … 262.6 ms 30 runs

The tests are super noisy, though (I get significantly different results between runs). But it seems like a pretty good win.
@nnethercote
nnethercote
commented
Feb 27, 2026

Copy link


Copy Markdown

Contributor
Nice wins.
@bors r+
@rust-bors
rust-borsBot
commented
Feb 27, 2026

Copy link


Copy Markdown

Contributor
📌 Commit f450289 has been approved by nnethercote
It is now in the queue for this repository.
@rust-borsrust-borsBot
added

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
and removed

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
labels
Feb 27, 2026
bjorn3
bjorn3
reviewed
Feb 27, 2026
View reviewed changes
Comment threadcompiler/rustc_session/src/search_paths.rs
 Err(..) => Default::default(),
 };
 files.sort_by(|lhs, rhs| lhs.file_name_str.cmp(&rhs.file_name_str));
 files.sort_unstable_by(|lhs, rhs| lhs.file_name_str.cmp(&rhs.file_name_str));
@bjorn3bjorn3Feb 27, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there no possibility of two files with the same file_name_str?
@bjorn3bjorn3Feb 27, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although I guess that would result in either one being picked deterministically (if it is the only matching one), which one is picked not mattering (if both files having the same crate hash) or an error about multiple candidates getting emitted.
@KobzolKobzolFeb 27, 2026

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We sorted (and searched) by the UTF-8 name before anyway, so there shouldn't hopefully be any change in behavior.
@rust-bors

This comment has been minimized.

Sign in to view
rust-borsBot
pushed a commit
that referenced
this pull request
Mar 1, 2026
@bors
Auto merge of#153131- Kobzol:filesearch-opt, r=nnethercote
07af97d
Optimize dependency file search
I tried to look into the slowdown reported in rust-lang/cargo#16665.
I created a Rust hello world program, and used this Python script to create a directory containing 200k files:
```python
from pathlib import Path
dir = Path("deps")
dir.mkdir(parents=True, exist_ok=True)
for i in range(200000):
path = dir / f"file{i:07}.o"
with open(path, "w") as f:
f.write("\n")
```
Then I tried to do various small microoptimalizations and simplifications to the code that iterates the search directories. Each individual commit improved performance, with the third one having the biggest effect.
Here are the results on `main` vs the last commit with the stage1 compiler on Linux, using `hyperfine "rustc +stage1 src/main.rs -L deps" -r 30` (there's IO involved, so it's good to let it run for a while):
```bash
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 299.4 ms ± 2.7 ms [User: 161.9 ms, System: 144.9 ms]
Range (min … max): 294.8 ms … 307.1 ms 30 runs
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 208.1 ms ± 4.5 ms [User: 87.3 ms, System: 128.7 ms]
Range (min … max): 202.4 ms … 219.6 ms 30 runs
```
Would be cool if someone could try this on macOS (maybe @ehuss - not sure if you have macOS or you only commented about its behavior on the Cargo issue :) ).
I also tried to prefilter the paths (not in this PR); right now we load everything and then we filter files with given prefixes, that's wasteful. Filtering just files starting with `lib` would get us down to ~150ms here. (The baseline without `-L` is ~80ms on my PC). The rest of the 70ms is essentially allocations from iterating the directory entries and sorting. That would be very hard to change - iterating the directory entries (de)allocates a lot of intermediate paths :( We'd have to implement the iteration by hand with either arena allocation, or at least some better management of memory.
r? @nnethercote
@rust-borsrust-borsBot
added

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
and removed

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
labels
Mar 1, 2026
@rust-bors
rust-borsBot
commented
Mar 1, 2026

Copy link


Copy Markdown

Contributor
💔 Test for 07af97d failed: CI. Failed job:
@rust-log-analyzer
rust-log-analyzer
commented
Mar 1, 2026

Copy link


Copy Markdown

Collaborator
The job aarch64-gnu failed! Check out the build log: (web)(plain enhanced)(plain)
Click to see the possible cause of the failure (guessed by this bot)
/dev/sda15 98M 6.4M 92M 7% /boot/efi
tmpfs 1.6G 16K 1.6G 1% /run/user/1001
================================================================================
Sufficient disk space available (120307208KB >= 52428800KB). Skipping cleanup.
##[group]Run src/ci/scripts/setup-environment.sh
src/ci/scripts/setup-environment.sh
shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
---
[1773/3894] Building ARMGenMCCodeEmitter.inc...
##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
[1774/3894] Building ARMGenMCPseudoLowering.inc...
Session terminated, killing shell...::group::Clock drift check
local time: Sun Mar 1 08:40:58 UTC 2026
network time: [1775/3894] Building ARMGenInstrInfo.inc...
Sun, 01 Mar 2026 08:40:58 GMT
##[endgroup]
[1776/3894] Building ARMGenRegisterBank.inc...

@Kobzol
Kobzol
commented
Mar 1, 2026

Copy link


Copy Markdown

MemberAuthor
@bors retry
Spurious worker restart.
@rust-borsrust-borsBot
added

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
and removed

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
labels
Mar 1, 2026

This was referenced Mar 1, 2026
 Merged

 Merged

@rust-bors

This comment has been minimized.

Sign in to view
@rust-borsrust-borsBot
added

merged-by-bors
This PR was explicitly merged by bors.
and removed

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
labels
Mar 2, 2026
@rust-bors
rust-borsBot
commented
Mar 2, 2026

Copy link


Copy Markdown

Contributor
☀️ Test successful - CI

Approved by: nnethercote

Duration: 3h 40m 3s

Pushing e7d90c6 to main...
@rust-bors
rust-borsBot
merged commit e7d90c6
into
rust-lang:mainMar 2, 2026

13 checks passed

@rustbotrustbot
added this to the 1.96.0 milestone
Mar 2, 2026
@github-actions
github-actionsBot
commented
Mar 2, 2026

Copy link


Copy Markdown

Contributor
What is this?
This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 8038127 (parent) -> e7d90c6 (this PR)
Test differences
Show 2 test diffs

2 doctest diffs were found. These are ignored, as they are noisy.
Test dashboard
Run
cargo run --manifest-path src/ci/citool/Cargo.toml -- \
test-dashboard e7d90c695a39426baf5ae705de2f9570a72229de --output-dir test-dashboard
And then open test-dashboard/index.html in your browser to see an overview of all executed tests.
Job duration changes
  1. dist-aarch64-llvm-mingw: 1h 33m -> 1h 46m (+14.1%)
  2. dist-aarch64-apple: 1h 56m -> 2h 12m (+13.1%)
  3. aarch64-apple: 3h 8m -> 3h 33m (+13.0%)
  4. dist-x86_64-apple: 2h 42m -> 2h 25m (-10.7%)
  5. arm-android: 1h 37m -> 1h 47m (+10.4%)
  6. dist-ohos-x86_64: 1h 14m -> 1h 21m (+9.7%)
  7. x86_64-gnu-aux: 2h 7m -> 2h 18m (+8.5%)
  8. x86_64-gnu-distcheck: 2h 7m -> 2h 18m (+8.4%)
  9. x86_64-gnu-miri: 1h 23m -> 1h 30m (+7.7%)
  10. dist-arm-linux-musl: 1h 38m -> 1h 30m (-7.3%)
How to interpret the job duration changes?
Job durations can vary a lot, based on the actual runner instance

that executed the job, system noise, invalidated caches, etc. The table above is provided

mostly for t-infra members, for simpler debugging of potential CI slow-downs.
@rust-timer
rust-timer
commented
Mar 2, 2026

Copy link


Copy Markdown

Collaborator
Finished benchmarking commit (e7d90c6): comparison URL.
Overall result: ❌✅ regressions and improvements - please read the text below
Our benchmarks found a performance regression caused by this PR.

This might be an actual regression, but it can also be just noise.
Next Steps:
  • If the regression was expected or you think it can be justified,
    
please write a comment with sufficient written justification, and add
    @rustbot label: +perf-regression-triaged to it, to mark the regression as triaged.
  • If you think that you know of a way to resolve the regression, try to create
    
a new PR with a fix for the regression.
  • If you do not understand the regression or you think that it is just noise,
    
you can ask the @rust-lang/wg-compiler-performance working group for help (members of this group
    
were already notified of this PR).
@rustbot label: +perf-regression

cc @rust-lang/wg-compiler-performance
Instruction count
Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.
meanrangecount
Regressions ❌ 
 (primary)		0.1%[0.1%, 0.1%]1
Regressions ❌ 
 (secondary)		--0
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-0.4%[-0.6%, -0.2%]7
All ❌✅ (primary)0.1%[0.1%, 0.1%]1
Max RSS (memory usage)
Results (secondary -4.0%)
A less reliable metric. May be of interest, but not used to determine the overall result above.
meanrangecount
Regressions ❌ 
 (primary)		--0
Regressions ❌ 
 (secondary)		--0
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-4.0%[-5.4%, -2.6%]3
All ❌✅ (primary)--0
Cycles
This benchmark run did not return any relevant results for this metric.
Binary size
Results (secondary 0.1%)
A less reliable metric. May be of interest, but not used to determine the overall result above.
meanrangecount
Regressions ❌ 
 (primary)		--0
Regressions ❌ 
 (secondary)		0.1%[0.1%, 0.1%]3
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		--0
All ❌✅ (primary)--0
Bootstrap: 480.971s -> 480.149s (-0.17%)
Artifact size: 396.96 MiB -> 396.96 MiB (0.00%)
@Kobzol
Kobzol
commented
Mar 2, 2026•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
The regression seems like noise. Tiny wins on large-workspace, matching pre-merge run.
@rustbot label: +perf-regression-triaged
@rustbotrustbot
added
the 
perf-regression-triaged
The performance regression has been triaged.
label
Mar 2, 2026
@osiewicz
osiewicz
commented
Mar 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Timings for Mac, as requested in OP (with the same scenario as the one exercised in OP):
➜ rustc-153131 hyperfine "rustc +nightly-2026-03-01 main.rs -L deps" -r 30
Benchmark 1: rustc +nightly-2026-03-01 main.rs -L deps
Time (mean ± σ): 215.3 ms ± 1.9 ms [User: 161.2 ms, System: 122.0 ms]
Range (min … max): 212.2 ms … 220.7 ms 30 runs

vs:
➜ rustc-153131 hyperfine "rustc +nightly-2026-03-06 main.rs -L deps" -r 30
Benchmark 1: rustc +nightly-2026-03-06 main.rs -L deps
Time (mean ± σ): 177.9 ms ± 1.6 ms [User: 128.5 ms, System: 120.6 ms]
Range (min … max): 175.2 ms … 183.2 ms 30 runs

And a baseline:
➜ rustc-153131 hyperfine "rustc +nightly-2026-03-06 main.rs" -r 30
Benchmark 1: rustc +nightly-2026-03-06 main.rs
Time (mean ± σ): 68.0 ms ± 1.1 ms [User: 87.0 ms, System: 52.7 ms]
Range (min … max): 66.8 ms … 71.9 ms 30 runs

@Kobzol
Kobzol
deleted the

filesearch-opt

branch
March 14, 2026 10:10
@KobzolKobzol
mentioned this pull request
Mar 15, 2026

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@petrochenkovpetrochenkovpetrochenkov left review comments
@bjorn3bjorn3bjorn3 left review comments
@nnethercotennethercotennethercote approved these changes

Assignees

@nnethercotennethercote

Labels

merged-by-borsThis PR was explicitly merged by bors.perf-regressionPerformance regression.perf-regression-triagedThe performance regression has been triaged.T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.

Projects


None yet


Milestone

1.96.0

Development

Successfully merging this pull request may close these issues.

9 participants

@Kobzol@rust-timer@ehuss@nnethercote@rust-log-analyzer@osiewicz@petrochenkov@bjorn3@rustbot

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Optimize dependency file search - #153131
Merged
rust-bors[bot] merged 4 commits into
rust-lang:mainfrom 
Kobzol:filesearch-opt
Mar 2, 2026
Merged
Optimize dependency file search#153131
rust-bors[bot] merged 4 commits into
rust-lang:mainfrom 
Kobzol:filesearch-opt
Conversation
@Kobzol
@KobzolKobzol
commented
Feb 26, 2026•

edited by rustbot

Loading

Copy link


Copy Markdown

Member
View all comments
I tried to look into the slowdown reported in rust-lang/cargo#16665.
I created a Rust hello world program, and used this Python script to create a directory containing 200k files:
frompathlibimportPathdir=Path("deps")
dir.mkdir(parents=True, exist_ok=True)
foriinrange(200000):
path=dir/f"file{i:07}.o"withopen(path, "w") asf:
f.write("\n")
Then I tried to do various small microoptimalizations and simplifications to the code that iterates the search directories. Each individual commit improved performance, with the third one having the biggest effect.
Here are the results on main vs the last commit with the stage1 compiler on Linux, using hyperfine "rustc +stage1 src/main.rs -L deps" -r 30 (there's IO involved, so it's good to let it run for a while):
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 299.4 ms ± 2.7 ms [User: 161.9 ms, System: 144.9 ms]
Range (min … max): 294.8 ms … 307.1 ms 30 runs
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 208.1 ms ± 4.5 ms [User: 87.3 ms, System: 128.7 ms]
Range (min … max): 202.4 ms … 219.6 ms 30 runs
Would be cool if someone could try this on macOS (maybe @ehuss - not sure if you have macOS or you only commented about its behavior on the Cargo issue :) ).
I also tried to prefilter the paths (not in this PR); right now we load everything and then we filter files with given prefixes, that's wasteful. Filtering just files starting with lib would get us down to ~150ms here. (The baseline without -L is ~80ms on my PC). The rest of the 70ms is essentially allocations from iterating the directory entries and sorting. That would be very hard to change - iterating the directory entries (de)allocates a lot of intermediate paths :( We'd have to implement the iteration by hand with either arena allocation, or at least some better management of memory.
r? @nnethercote
@rustbotrustbot
added

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
T-compiler
Relevant to the compiler team, which will review and decide on the PR/issue.
labels
Feb 26, 2026
@Kobzol
Kobzol
commented
Feb 26, 2026

Copy link


Copy Markdown

MemberAuthor
Not expecting any big wins there, but let's check:
@bors try @rust-timer queue
@rust-timer

This comment has been minimized.

Sign in to view
@rust-bors

This comment has been minimized.

Sign in to view
@rustbotrustbot
added
the 
S-waiting-on-perf
Status: Waiting on a perf run to be completed.
label
Feb 26, 2026
rust-borsBot
pushed a commit
that referenced
this pull request
Feb 26, 2026
@bors
Auto merge of#153131- Kobzol:filesearch-opt, r=<try>
71d4456
Optimize dependency file search
petrochenkov
petrochenkov
reviewed
Feb 26, 2026
View reviewed changes
Comment threadcompiler/rustc_session/src/search_paths.rs
@rust-bors
rust-borsBot
commented
Feb 26, 2026

Copy link


Copy Markdown

Contributor
☀️ Try build successful (CI)

Build commit: 71d4456 (71d445654a32204fbf86bfec11e657fb97adcd22, parent: 69b78537fac74de40f009b076bcbbf54b77683ad)
@rust-timer

This comment has been minimized.

Sign in to view
@ehuss
ehuss
commented
Feb 26, 2026

Copy link


Copy Markdown

Contributor
Unfortunately I am unable to test because of #153077, I am unable to get rustc to build.
@rust-timer
rust-timer
commented
Feb 26, 2026

Copy link


Copy Markdown

Collaborator
Finished benchmarking commit (71d4456): comparison URL.
Overall result: ❌✅ regressions and improvements - please read the text below
Benchmarking this pull request means it may be perf-sensitive – we'll automatically label it not fit for rolling up. You can override this, but we strongly advise not to, due to possible changes in compiler perf.
Next Steps: If you can justify the regressions found in this try perf run, please do so in sufficient writing along with @rustbot label: +perf-regression-triaged. If not, please fix the regressions and do another perf run. If its results are neutral or positive, the label will be automatically removed.
@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression
Instruction count
Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.
meanrangecount
Regressions ❌ 
 (primary)		0.3%[0.2%, 0.3%]4
Regressions ❌ 
 (secondary)		0.0%[0.0%, 0.0%]1
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-0.4%[-0.6%, -0.1%]7
All ❌✅ (primary)0.3%[0.2%, 0.3%]4
Max RSS (memory usage)
Results (secondary 0.1%)
A less reliable metric. May be of interest, but not used to determine the overall result above.
meanrangecount
Regressions ❌ 
 (primary)		--0
Regressions ❌ 
 (secondary)		6.8%[6.8%, 6.8%]1
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-6.7%[-6.7%, -6.7%]1
All ❌✅ (primary)--0
Cycles
This benchmark run did not return any relevant results for this metric.
Binary size
This benchmark run did not return any relevant results for this metric.
Bootstrap: 480.132s -> 480.258s (0.03%)
Artifact size: 395.80 MiB -> 395.77 MiB (-0.01%)
@rustbotrustbot
added

perf-regression
Performance regression.
and removed

S-waiting-on-perf
Status: Waiting on a perf run to be completed.
labels
Feb 26, 2026
@ehuss
ehuss
commented
Feb 26, 2026

Copy link


Copy Markdown

Contributor
OK, got some data:
With this PR:
Benchmark 1: ~/Proj/rust/rust/build/aarch64-apple-darwin/stage1/bin/rustc src/main.rs -L deps
Time (mean ± σ): 197.4 ms ± 3.9 ms [User: 143.7 ms, System: 117.9 ms]
Range (min … max): 192.4 ms … 207.7 ms 30 runs

Without:
Benchmark 1: ~/Proj/rust/rust/build/aarch64-apple-darwin/stage1/bin/rustc src/main.rs -L deps
Time (mean ± σ): 248.9 ms ± 5.1 ms [User: 189.5 ms, System: 124.2 ms]
Range (min … max): 241.1 ms … 262.6 ms 30 runs

The tests are super noisy, though (I get significantly different results between runs). But it seems like a pretty good win.
@nnethercote
nnethercote
commented
Feb 27, 2026

Copy link


Copy Markdown

Contributor
Nice wins.
@bors r+
@rust-bors
rust-borsBot
commented
Feb 27, 2026

Copy link


Copy Markdown

Contributor
📌 Commit f450289 has been approved by nnethercote
It is now in the queue for this repository.
@rust-borsrust-borsBot
added

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
and removed

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
labels
Feb 27, 2026
bjorn3
bjorn3
reviewed
Feb 27, 2026
View reviewed changes
Comment threadcompiler/rustc_session/src/search_paths.rs
 Err(..) => Default::default(),
 };
 files.sort_by(|lhs, rhs| lhs.file_name_str.cmp(&rhs.file_name_str));
 files.sort_unstable_by(|lhs, rhs| lhs.file_name_str.cmp(&rhs.file_name_str));
@bjorn3bjorn3Feb 27, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there no possibility of two files with the same file_name_str?
@bjorn3bjorn3Feb 27, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although I guess that would result in either one being picked deterministically (if it is the only matching one), which one is picked not mattering (if both files having the same crate hash) or an error about multiple candidates getting emitted.
@KobzolKobzolFeb 27, 2026

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We sorted (and searched) by the UTF-8 name before anyway, so there shouldn't hopefully be any change in behavior.
@rust-bors

This comment has been minimized.

Sign in to view
rust-borsBot
pushed a commit
that referenced
this pull request
Mar 1, 2026
@bors
Auto merge of#153131- Kobzol:filesearch-opt, r=nnethercote
07af97d
Optimize dependency file search
I tried to look into the slowdown reported in rust-lang/cargo#16665.
I created a Rust hello world program, and used this Python script to create a directory containing 200k files:
```python
from pathlib import Path
dir = Path("deps")
dir.mkdir(parents=True, exist_ok=True)
for i in range(200000):
path = dir / f"file{i:07}.o"
with open(path, "w") as f:
f.write("\n")
```
Then I tried to do various small microoptimalizations and simplifications to the code that iterates the search directories. Each individual commit improved performance, with the third one having the biggest effect.
Here are the results on `main` vs the last commit with the stage1 compiler on Linux, using `hyperfine "rustc +stage1 src/main.rs -L deps" -r 30` (there's IO involved, so it's good to let it run for a while):
```bash
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 299.4 ms ± 2.7 ms [User: 161.9 ms, System: 144.9 ms]
Range (min … max): 294.8 ms … 307.1 ms 30 runs
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 208.1 ms ± 4.5 ms [User: 87.3 ms, System: 128.7 ms]
Range (min … max): 202.4 ms … 219.6 ms 30 runs
```
Would be cool if someone could try this on macOS (maybe @ehuss - not sure if you have macOS or you only commented about its behavior on the Cargo issue :) ).
I also tried to prefilter the paths (not in this PR); right now we load everything and then we filter files with given prefixes, that's wasteful. Filtering just files starting with `lib` would get us down to ~150ms here. (The baseline without `-L` is ~80ms on my PC). The rest of the 70ms is essentially allocations from iterating the directory entries and sorting. That would be very hard to change - iterating the directory entries (de)allocates a lot of intermediate paths :( We'd have to implement the iteration by hand with either arena allocation, or at least some better management of memory.
r? @nnethercote
@rust-borsrust-borsBot
added

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
and removed

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
labels
Mar 1, 2026
@rust-bors
rust-borsBot
commented
Mar 1, 2026

Copy link


Copy Markdown

Contributor
💔 Test for 07af97d failed: CI. Failed job:
@rust-log-analyzer
rust-log-analyzer
commented
Mar 1, 2026

Copy link


Copy Markdown

Collaborator
The job aarch64-gnu failed! Check out the build log: (web)(plain enhanced)(plain)
Click to see the possible cause of the failure (guessed by this bot)
/dev/sda15 98M 6.4M 92M 7% /boot/efi
tmpfs 1.6G 16K 1.6G 1% /run/user/1001
================================================================================
Sufficient disk space available (120307208KB >= 52428800KB). Skipping cleanup.
##[group]Run src/ci/scripts/setup-environment.sh
src/ci/scripts/setup-environment.sh
shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
---
[1773/3894] Building ARMGenMCCodeEmitter.inc...
##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
[1774/3894] Building ARMGenMCPseudoLowering.inc...
Session terminated, killing shell...::group::Clock drift check
local time: Sun Mar 1 08:40:58 UTC 2026
network time: [1775/3894] Building ARMGenInstrInfo.inc...
Sun, 01 Mar 2026 08:40:58 GMT
##[endgroup]
[1776/3894] Building ARMGenRegisterBank.inc...

@Kobzol
Kobzol
commented
Mar 1, 2026

Copy link


Copy Markdown

MemberAuthor
@bors retry
Spurious worker restart.
@rust-borsrust-borsBot
added

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
and removed

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
labels
Mar 1, 2026

This was referenced Mar 1, 2026
 Merged

 Merged

@rust-bors

This comment has been minimized.

Sign in to view
@rust-borsrust-borsBot
added

merged-by-bors
This PR was explicitly merged by bors.
and removed

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
labels
Mar 2, 2026
@rust-bors
rust-borsBot
commented
Mar 2, 2026

Copy link


Copy Markdown

Contributor
☀️ Test successful - CI

Approved by: nnethercote

Duration: 3h 40m 3s

Pushing e7d90c6 to main...
@rust-bors
rust-borsBot
merged commit e7d90c6
into
rust-lang:mainMar 2, 2026

13 checks passed

@rustbotrustbot
added this to the 1.96.0 milestone
Mar 2, 2026
@github-actions
github-actionsBot
commented
Mar 2, 2026

Copy link


Copy Markdown

Contributor
What is this?
This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 8038127 (parent) -> e7d90c6 (this PR)
Test differences
Show 2 test diffs

2 doctest diffs were found. These are ignored, as they are noisy.
Test dashboard
Run
cargo run --manifest-path src/ci/citool/Cargo.toml -- \
test-dashboard e7d90c695a39426baf5ae705de2f9570a72229de --output-dir test-dashboard
And then open test-dashboard/index.html in your browser to see an overview of all executed tests.
Job duration changes
  1. dist-aarch64-llvm-mingw: 1h 33m -> 1h 46m (+14.1%)
  2. dist-aarch64-apple: 1h 56m -> 2h 12m (+13.1%)
  3. aarch64-apple: 3h 8m -> 3h 33m (+13.0%)
  4. dist-x86_64-apple: 2h 42m -> 2h 25m (-10.7%)
  5. arm-android: 1h 37m -> 1h 47m (+10.4%)
  6. dist-ohos-x86_64: 1h 14m -> 1h 21m (+9.7%)
  7. x86_64-gnu-aux: 2h 7m -> 2h 18m (+8.5%)
  8. x86_64-gnu-distcheck: 2h 7m -> 2h 18m (+8.4%)
  9. x86_64-gnu-miri: 1h 23m -> 1h 30m (+7.7%)
  10. dist-arm-linux-musl: 1h 38m -> 1h 30m (-7.3%)
How to interpret the job duration changes?
Job durations can vary a lot, based on the actual runner instance

that executed the job, system noise, invalidated caches, etc. The table above is provided

mostly for t-infra members, for simpler debugging of potential CI slow-downs.
@rust-timer
rust-timer
commented
Mar 2, 2026

Copy link


Copy Markdown

Collaborator
Finished benchmarking commit (e7d90c6): comparison URL.
Overall result: ❌✅ regressions and improvements - please read the text below
Our benchmarks found a performance regression caused by this PR.

This might be an actual regression, but it can also be just noise.
Next Steps:
  • If the regression was expected or you think it can be justified,
    
please write a comment with sufficient written justification, and add
    @rustbot label: +perf-regression-triaged to it, to mark the regression as triaged.
  • If you think that you know of a way to resolve the regression, try to create
    
a new PR with a fix for the regression.
  • If you do not understand the regression or you think that it is just noise,
    
you can ask the @rust-lang/wg-compiler-performance working group for help (members of this group
    
were already notified of this PR).
@rustbot label: +perf-regression

cc @rust-lang/wg-compiler-performance
Instruction count
Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.
meanrangecount
Regressions ❌ 
 (primary)		0.1%[0.1%, 0.1%]1
Regressions ❌ 
 (secondary)		--0
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-0.4%[-0.6%, -0.2%]7
All ❌✅ (primary)0.1%[0.1%, 0.1%]1
Max RSS (memory usage)
Results (secondary -4.0%)
A less reliable metric. May be of interest, but not used to determine the overall result above.
meanrangecount
Regressions ❌ 
 (primary)		--0
Regressions ❌ 
 (secondary)		--0
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-4.0%[-5.4%, -2.6%]3
All ❌✅ (primary)--0
Cycles
This benchmark run did not return any relevant results for this metric.
Binary size
Results (secondary 0.1%)
A less reliable metric. May be of interest, but not used to determine the overall result above.
meanrangecount
Regressions ❌ 
 (primary)		--0
Regressions ❌ 
 (secondary)		0.1%[0.1%, 0.1%]3
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		--0
All ❌✅ (primary)--0
Bootstrap: 480.971s -> 480.149s (-0.17%)
Artifact size: 396.96 MiB -> 396.96 MiB (0.00%)
@Kobzol
Kobzol
commented
Mar 2, 2026•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
The regression seems like noise. Tiny wins on large-workspace, matching pre-merge run.
@rustbot label: +perf-regression-triaged
@rustbotrustbot
added
the 
perf-regression-triaged
The performance regression has been triaged.
label
Mar 2, 2026
@osiewicz
osiewicz
commented
Mar 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Timings for Mac, as requested in OP (with the same scenario as the one exercised in OP):
➜ rustc-153131 hyperfine "rustc +nightly-2026-03-01 main.rs -L deps" -r 30
Benchmark 1: rustc +nightly-2026-03-01 main.rs -L deps
Time (mean ± σ): 215.3 ms ± 1.9 ms [User: 161.2 ms, System: 122.0 ms]
Range (min … max): 212.2 ms … 220.7 ms 30 runs

vs:
➜ rustc-153131 hyperfine "rustc +nightly-2026-03-06 main.rs -L deps" -r 30
Benchmark 1: rustc +nightly-2026-03-06 main.rs -L deps
Time (mean ± σ): 177.9 ms ± 1.6 ms [User: 128.5 ms, System: 120.6 ms]
Range (min … max): 175.2 ms … 183.2 ms 30 runs

And a baseline:
➜ rustc-153131 hyperfine "rustc +nightly-2026-03-06 main.rs" -r 30
Benchmark 1: rustc +nightly-2026-03-06 main.rs
Time (mean ± σ): 68.0 ms ± 1.1 ms [User: 87.0 ms, System: 52.7 ms]
Range (min … max): 66.8 ms … 71.9 ms 30 runs

@Kobzol
Kobzol
deleted the

filesearch-opt

branch
March 14, 2026 10:10
@KobzolKobzol
mentioned this pull request
Mar 15, 2026

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@petrochenkovpetrochenkovpetrochenkov left review comments
@bjorn3bjorn3bjorn3 left review comments
@nnethercotennethercotennethercote approved these changes

Assignees

@nnethercotennethercote

Labels

merged-by-borsThis PR was explicitly merged by bors.perf-regressionPerformance regression.perf-regression-triagedThe performance regression has been triaged.T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.

Projects


None yet


Milestone

1.96.0

Development

Successfully merging this pull request may close these issues.

9 participants

@Kobzol@rust-timer@ehuss@nnethercote@rust-log-analyzer@osiewicz@petrochenkov@bjorn3@rustbot

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Optimize dependency file search - #153131
Merged
rust-bors[bot] merged 4 commits into
rust-lang:mainfrom 
Kobzol:filesearch-opt
Mar 2, 2026
Merged
Optimize dependency file search#153131
rust-bors[bot] merged 4 commits into
rust-lang:mainfrom 
Kobzol:filesearch-opt
Conversation
@Kobzol
@KobzolKobzol
commented
Feb 26, 2026•

edited by rustbot

Loading

Copy link


Copy Markdown

Member
View all comments
I tried to look into the slowdown reported in rust-lang/cargo#16665.
I created a Rust hello world program, and used this Python script to create a directory containing 200k files:
frompathlibimportPathdir=Path("deps")
dir.mkdir(parents=True, exist_ok=True)
foriinrange(200000):
path=dir/f"file{i:07}.o"withopen(path, "w") asf:
f.write("\n")
Then I tried to do various small microoptimalizations and simplifications to the code that iterates the search directories. Each individual commit improved performance, with the third one having the biggest effect.
Here are the results on main vs the last commit with the stage1 compiler on Linux, using hyperfine "rustc +stage1 src/main.rs -L deps" -r 30 (there's IO involved, so it's good to let it run for a while):
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 299.4 ms ± 2.7 ms [User: 161.9 ms, System: 144.9 ms]
Range (min … max): 294.8 ms … 307.1 ms 30 runs
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 208.1 ms ± 4.5 ms [User: 87.3 ms, System: 128.7 ms]
Range (min … max): 202.4 ms … 219.6 ms 30 runs
Would be cool if someone could try this on macOS (maybe @ehuss - not sure if you have macOS or you only commented about its behavior on the Cargo issue :) ).
I also tried to prefilter the paths (not in this PR); right now we load everything and then we filter files with given prefixes, that's wasteful. Filtering just files starting with lib would get us down to ~150ms here. (The baseline without -L is ~80ms on my PC). The rest of the 70ms is essentially allocations from iterating the directory entries and sorting. That would be very hard to change - iterating the directory entries (de)allocates a lot of intermediate paths :( We'd have to implement the iteration by hand with either arena allocation, or at least some better management of memory.
r? @nnethercote
@rustbotrustbot
added

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
T-compiler
Relevant to the compiler team, which will review and decide on the PR/issue.
labels
Feb 26, 2026
@Kobzol
Kobzol
commented
Feb 26, 2026

Copy link


Copy Markdown

MemberAuthor
Not expecting any big wins there, but let's check:
@bors try @rust-timer queue
@rust-timer

This comment has been minimized.

Sign in to view
@rust-bors

This comment has been minimized.

Sign in to view
@rustbotrustbot
added
the 
S-waiting-on-perf
Status: Waiting on a perf run to be completed.
label
Feb 26, 2026
rust-borsBot
pushed a commit
that referenced
this pull request
Feb 26, 2026
@bors
Auto merge of#153131- Kobzol:filesearch-opt, r=<try>
71d4456
Optimize dependency file search
petrochenkov
petrochenkov
reviewed
Feb 26, 2026
View reviewed changes
Comment threadcompiler/rustc_session/src/search_paths.rs
@rust-bors
rust-borsBot
commented
Feb 26, 2026

Copy link


Copy Markdown

Contributor
☀️ Try build successful (CI)

Build commit: 71d4456 (71d445654a32204fbf86bfec11e657fb97adcd22, parent: 69b78537fac74de40f009b076bcbbf54b77683ad)
@rust-timer

This comment has been minimized.

Sign in to view
@ehuss
ehuss
commented
Feb 26, 2026

Copy link


Copy Markdown

Contributor
Unfortunately I am unable to test because of #153077, I am unable to get rustc to build.
@rust-timer
rust-timer
commented
Feb 26, 2026

Copy link


Copy Markdown

Collaborator
Finished benchmarking commit (71d4456): comparison URL.
Overall result: ❌✅ regressions and improvements - please read the text below
Benchmarking this pull request means it may be perf-sensitive – we'll automatically label it not fit for rolling up. You can override this, but we strongly advise not to, due to possible changes in compiler perf.
Next Steps: If you can justify the regressions found in this try perf run, please do so in sufficient writing along with @rustbot label: +perf-regression-triaged. If not, please fix the regressions and do another perf run. If its results are neutral or positive, the label will be automatically removed.
@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression
Instruction count
Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.
meanrangecount
Regressions ❌ 
 (primary)		0.3%[0.2%, 0.3%]4
Regressions ❌ 
 (secondary)		0.0%[0.0%, 0.0%]1
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-0.4%[-0.6%, -0.1%]7
All ❌✅ (primary)0.3%[0.2%, 0.3%]4
Max RSS (memory usage)
Results (secondary 0.1%)
A less reliable metric. May be of interest, but not used to determine the overall result above.
meanrangecount
Regressions ❌ 
 (primary)		--0
Regressions ❌ 
 (secondary)		6.8%[6.8%, 6.8%]1
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-6.7%[-6.7%, -6.7%]1
All ❌✅ (primary)--0
Cycles
This benchmark run did not return any relevant results for this metric.
Binary size
This benchmark run did not return any relevant results for this metric.
Bootstrap: 480.132s -> 480.258s (0.03%)
Artifact size: 395.80 MiB -> 395.77 MiB (-0.01%)
@rustbotrustbot
added

perf-regression
Performance regression.
and removed

S-waiting-on-perf
Status: Waiting on a perf run to be completed.
labels
Feb 26, 2026
@ehuss
ehuss
commented
Feb 26, 2026

Copy link


Copy Markdown

Contributor
OK, got some data:
With this PR:
Benchmark 1: ~/Proj/rust/rust/build/aarch64-apple-darwin/stage1/bin/rustc src/main.rs -L deps
Time (mean ± σ): 197.4 ms ± 3.9 ms [User: 143.7 ms, System: 117.9 ms]
Range (min … max): 192.4 ms … 207.7 ms 30 runs

Without:
Benchmark 1: ~/Proj/rust/rust/build/aarch64-apple-darwin/stage1/bin/rustc src/main.rs -L deps
Time (mean ± σ): 248.9 ms ± 5.1 ms [User: 189.5 ms, System: 124.2 ms]
Range (min … max): 241.1 ms … 262.6 ms 30 runs

The tests are super noisy, though (I get significantly different results between runs). But it seems like a pretty good win.
@nnethercote
nnethercote
commented
Feb 27, 2026

Copy link


Copy Markdown

Contributor
Nice wins.
@bors r+
@rust-bors
rust-borsBot
commented
Feb 27, 2026

Copy link


Copy Markdown

Contributor
📌 Commit f450289 has been approved by nnethercote
It is now in the queue for this repository.
@rust-borsrust-borsBot
added

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
and removed

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
labels
Feb 27, 2026
bjorn3
bjorn3
reviewed
Feb 27, 2026
View reviewed changes
Comment threadcompiler/rustc_session/src/search_paths.rs
 Err(..) => Default::default(),
 };
 files.sort_by(|lhs, rhs| lhs.file_name_str.cmp(&rhs.file_name_str));
 files.sort_unstable_by(|lhs, rhs| lhs.file_name_str.cmp(&rhs.file_name_str));
@bjorn3bjorn3Feb 27, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there no possibility of two files with the same file_name_str?
@bjorn3bjorn3Feb 27, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although I guess that would result in either one being picked deterministically (if it is the only matching one), which one is picked not mattering (if both files having the same crate hash) or an error about multiple candidates getting emitted.
@KobzolKobzolFeb 27, 2026

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We sorted (and searched) by the UTF-8 name before anyway, so there shouldn't hopefully be any change in behavior.
@rust-bors

This comment has been minimized.

Sign in to view
rust-borsBot
pushed a commit
that referenced
this pull request
Mar 1, 2026
@bors
Auto merge of#153131- Kobzol:filesearch-opt, r=nnethercote
07af97d
Optimize dependency file search
I tried to look into the slowdown reported in rust-lang/cargo#16665.
I created a Rust hello world program, and used this Python script to create a directory containing 200k files:
```python
from pathlib import Path
dir = Path("deps")
dir.mkdir(parents=True, exist_ok=True)
for i in range(200000):
path = dir / f"file{i:07}.o"
with open(path, "w") as f:
f.write("\n")
```
Then I tried to do various small microoptimalizations and simplifications to the code that iterates the search directories. Each individual commit improved performance, with the third one having the biggest effect.
Here are the results on `main` vs the last commit with the stage1 compiler on Linux, using `hyperfine "rustc +stage1 src/main.rs -L deps" -r 30` (there's IO involved, so it's good to let it run for a while):
```bash
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 299.4 ms ± 2.7 ms [User: 161.9 ms, System: 144.9 ms]
Range (min … max): 294.8 ms … 307.1 ms 30 runs
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 208.1 ms ± 4.5 ms [User: 87.3 ms, System: 128.7 ms]
Range (min … max): 202.4 ms … 219.6 ms 30 runs
```
Would be cool if someone could try this on macOS (maybe @ehuss - not sure if you have macOS or you only commented about its behavior on the Cargo issue :) ).
I also tried to prefilter the paths (not in this PR); right now we load everything and then we filter files with given prefixes, that's wasteful. Filtering just files starting with `lib` would get us down to ~150ms here. (The baseline without `-L` is ~80ms on my PC). The rest of the 70ms is essentially allocations from iterating the directory entries and sorting. That would be very hard to change - iterating the directory entries (de)allocates a lot of intermediate paths :( We'd have to implement the iteration by hand with either arena allocation, or at least some better management of memory.
r? @nnethercote
@rust-borsrust-borsBot
added

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
and removed

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
labels
Mar 1, 2026
@rust-bors
rust-borsBot
commented
Mar 1, 2026

Copy link


Copy Markdown

Contributor
💔 Test for 07af97d failed: CI. Failed job:
@rust-log-analyzer
rust-log-analyzer
commented
Mar 1, 2026

Copy link


Copy Markdown

Collaborator
The job aarch64-gnu failed! Check out the build log: (web)(plain enhanced)(plain)
Click to see the possible cause of the failure (guessed by this bot)
/dev/sda15 98M 6.4M 92M 7% /boot/efi
tmpfs 1.6G 16K 1.6G 1% /run/user/1001
================================================================================
Sufficient disk space available (120307208KB >= 52428800KB). Skipping cleanup.
##[group]Run src/ci/scripts/setup-environment.sh
src/ci/scripts/setup-environment.sh
shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
---
[1773/3894] Building ARMGenMCCodeEmitter.inc...
##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
[1774/3894] Building ARMGenMCPseudoLowering.inc...
Session terminated, killing shell...::group::Clock drift check
local time: Sun Mar 1 08:40:58 UTC 2026
network time: [1775/3894] Building ARMGenInstrInfo.inc...
Sun, 01 Mar 2026 08:40:58 GMT
##[endgroup]
[1776/3894] Building ARMGenRegisterBank.inc...

@Kobzol
Kobzol
commented
Mar 1, 2026

Copy link


Copy Markdown

MemberAuthor
@bors retry
Spurious worker restart.
@rust-borsrust-borsBot
added

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
and removed

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
labels
Mar 1, 2026

This was referenced Mar 1, 2026
 Merged

 Merged

@rust-bors

This comment has been minimized.

Sign in to view
@rust-borsrust-borsBot
added

merged-by-bors
This PR was explicitly merged by bors.
and removed

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
labels
Mar 2, 2026
@rust-bors
rust-borsBot
commented
Mar 2, 2026

Copy link


Copy Markdown

Contributor
☀️ Test successful - CI

Approved by: nnethercote

Duration: 3h 40m 3s

Pushing e7d90c6 to main...
@rust-bors
rust-borsBot
merged commit e7d90c6
into
rust-lang:mainMar 2, 2026

13 checks passed

@rustbotrustbot
added this to the 1.96.0 milestone
Mar 2, 2026
@github-actions
github-actionsBot
commented
Mar 2, 2026

Copy link


Copy Markdown

Contributor
What is this?
This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 8038127 (parent) -> e7d90c6 (this PR)
Test differences
Show 2 test diffs

2 doctest diffs were found. These are ignored, as they are noisy.
Test dashboard
Run
cargo run --manifest-path src/ci/citool/Cargo.toml -- \
test-dashboard e7d90c695a39426baf5ae705de2f9570a72229de --output-dir test-dashboard
And then open test-dashboard/index.html in your browser to see an overview of all executed tests.
Job duration changes
  1. dist-aarch64-llvm-mingw: 1h 33m -> 1h 46m (+14.1%)
  2. dist-aarch64-apple: 1h 56m -> 2h 12m (+13.1%)
  3. aarch64-apple: 3h 8m -> 3h 33m (+13.0%)
  4. dist-x86_64-apple: 2h 42m -> 2h 25m (-10.7%)
  5. arm-android: 1h 37m -> 1h 47m (+10.4%)
  6. dist-ohos-x86_64: 1h 14m -> 1h 21m (+9.7%)
  7. x86_64-gnu-aux: 2h 7m -> 2h 18m (+8.5%)
  8. x86_64-gnu-distcheck: 2h 7m -> 2h 18m (+8.4%)
  9. x86_64-gnu-miri: 1h 23m -> 1h 30m (+7.7%)
  10. dist-arm-linux-musl: 1h 38m -> 1h 30m (-7.3%)
How to interpret the job duration changes?
Job durations can vary a lot, based on the actual runner instance

that executed the job, system noise, invalidated caches, etc. The table above is provided

mostly for t-infra members, for simpler debugging of potential CI slow-downs.
@rust-timer
rust-timer
commented
Mar 2, 2026

Copy link


Copy Markdown

Collaborator
Finished benchmarking commit (e7d90c6): comparison URL.
Overall result: ❌✅ regressions and improvements - please read the text below
Our benchmarks found a performance regression caused by this PR.

This might be an actual regression, but it can also be just noise.
Next Steps:
  • If the regression was expected or you think it can be justified,
    
please write a comment with sufficient written justification, and add
    @rustbot label: +perf-regression-triaged to it, to mark the regression as triaged.
  • If you think that you know of a way to resolve the regression, try to create
    
a new PR with a fix for the regression.
  • If you do not understand the regression or you think that it is just noise,
    
you can ask the @rust-lang/wg-compiler-performance working group for help (members of this group
    
were already notified of this PR).
@rustbot label: +perf-regression

cc @rust-lang/wg-compiler-performance
Instruction count
Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.
meanrangecount
Regressions ❌ 
 (primary)		0.1%[0.1%, 0.1%]1
Regressions ❌ 
 (secondary)		--0
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-0.4%[-0.6%, -0.2%]7
All ❌✅ (primary)0.1%[0.1%, 0.1%]1
Max RSS (memory usage)
Results (secondary -4.0%)
A less reliable metric. May be of interest, but not used to determine the overall result above.
meanrangecount
Regressions ❌ 
 (primary)		--0
Regressions ❌ 
 (secondary)		--0
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-4.0%[-5.4%, -2.6%]3
All ❌✅ (primary)--0
Cycles
This benchmark run did not return any relevant results for this metric.
Binary size
Results (secondary 0.1%)
A less reliable metric. May be of interest, but not used to determine the overall result above.
meanrangecount
Regressions ❌ 
 (primary)		--0
Regressions ❌ 
 (secondary)		0.1%[0.1%, 0.1%]3
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		--0
All ❌✅ (primary)--0
Bootstrap: 480.971s -> 480.149s (-0.17%)
Artifact size: 396.96 MiB -> 396.96 MiB (0.00%)
@Kobzol
Kobzol
commented
Mar 2, 2026•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
The regression seems like noise. Tiny wins on large-workspace, matching pre-merge run.
@rustbot label: +perf-regression-triaged
@rustbotrustbot
added
the 
perf-regression-triaged
The performance regression has been triaged.
label
Mar 2, 2026
@osiewicz
osiewicz
commented
Mar 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Timings for Mac, as requested in OP (with the same scenario as the one exercised in OP):
➜ rustc-153131 hyperfine "rustc +nightly-2026-03-01 main.rs -L deps" -r 30
Benchmark 1: rustc +nightly-2026-03-01 main.rs -L deps
Time (mean ± σ): 215.3 ms ± 1.9 ms [User: 161.2 ms, System: 122.0 ms]
Range (min … max): 212.2 ms … 220.7 ms 30 runs

vs:
➜ rustc-153131 hyperfine "rustc +nightly-2026-03-06 main.rs -L deps" -r 30
Benchmark 1: rustc +nightly-2026-03-06 main.rs -L deps
Time (mean ± σ): 177.9 ms ± 1.6 ms [User: 128.5 ms, System: 120.6 ms]
Range (min … max): 175.2 ms … 183.2 ms 30 runs

And a baseline:
➜ rustc-153131 hyperfine "rustc +nightly-2026-03-06 main.rs" -r 30
Benchmark 1: rustc +nightly-2026-03-06 main.rs
Time (mean ± σ): 68.0 ms ± 1.1 ms [User: 87.0 ms, System: 52.7 ms]
Range (min … max): 66.8 ms … 71.9 ms 30 runs

@Kobzol
Kobzol
deleted the

filesearch-opt

branch
March 14, 2026 10:10
@KobzolKobzol
mentioned this pull request
Mar 15, 2026

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@petrochenkovpetrochenkovpetrochenkov left review comments
@bjorn3bjorn3bjorn3 left review comments
@nnethercotennethercotennethercote approved these changes

Assignees

@nnethercotennethercote

Labels

merged-by-borsThis PR was explicitly merged by bors.perf-regressionPerformance regression.perf-regression-triagedThe performance regression has been triaged.T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.

Projects


None yet


Milestone

1.96.0

Development

Successfully merging this pull request may close these issues.

9 participants

@Kobzol@rust-timer@ehuss@nnethercote@rust-log-analyzer@osiewicz@petrochenkov@bjorn3@rustbot

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Optimize dependency file search - #153131
Merged
rust-bors[bot] merged 4 commits into
rust-lang:mainfrom 
Kobzol:filesearch-opt
Mar 2, 2026
Merged
Optimize dependency file search#153131
rust-bors[bot] merged 4 commits into
rust-lang:mainfrom 
Kobzol:filesearch-opt
Conversation
@Kobzol
@KobzolKobzol
commented
Feb 26, 2026•

edited by rustbot

Loading

Copy link


Copy Markdown

Member
View all comments
I tried to look into the slowdown reported in rust-lang/cargo#16665.
I created a Rust hello world program, and used this Python script to create a directory containing 200k files:
frompathlibimportPathdir=Path("deps")
dir.mkdir(parents=True, exist_ok=True)
foriinrange(200000):
path=dir/f"file{i:07}.o"withopen(path, "w") asf:
f.write("\n")
Then I tried to do various small microoptimalizations and simplifications to the code that iterates the search directories. Each individual commit improved performance, with the third one having the biggest effect.
Here are the results on main vs the last commit with the stage1 compiler on Linux, using hyperfine "rustc +stage1 src/main.rs -L deps" -r 30 (there's IO involved, so it's good to let it run for a while):
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 299.4 ms ± 2.7 ms [User: 161.9 ms, System: 144.9 ms]
Range (min … max): 294.8 ms … 307.1 ms 30 runs
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 208.1 ms ± 4.5 ms [User: 87.3 ms, System: 128.7 ms]
Range (min … max): 202.4 ms … 219.6 ms 30 runs
Would be cool if someone could try this on macOS (maybe @ehuss - not sure if you have macOS or you only commented about its behavior on the Cargo issue :) ).
I also tried to prefilter the paths (not in this PR); right now we load everything and then we filter files with given prefixes, that's wasteful. Filtering just files starting with lib would get us down to ~150ms here. (The baseline without -L is ~80ms on my PC). The rest of the 70ms is essentially allocations from iterating the directory entries and sorting. That would be very hard to change - iterating the directory entries (de)allocates a lot of intermediate paths :( We'd have to implement the iteration by hand with either arena allocation, or at least some better management of memory.
r? @nnethercote
@rustbotrustbot
added

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
T-compiler
Relevant to the compiler team, which will review and decide on the PR/issue.
labels
Feb 26, 2026
@Kobzol
Kobzol
commented
Feb 26, 2026

Copy link


Copy Markdown

MemberAuthor
Not expecting any big wins there, but let's check:
@bors try @rust-timer queue
@rust-timer

This comment has been minimized.

Sign in to view
@rust-bors

This comment has been minimized.

Sign in to view
@rustbotrustbot
added
the 
S-waiting-on-perf
Status: Waiting on a perf run to be completed.
label
Feb 26, 2026
rust-borsBot
pushed a commit
that referenced
this pull request
Feb 26, 2026
@bors
Auto merge of#153131- Kobzol:filesearch-opt, r=<try>
71d4456
Optimize dependency file search
petrochenkov
petrochenkov
reviewed
Feb 26, 2026
View reviewed changes
Comment threadcompiler/rustc_session/src/search_paths.rs
@rust-bors
rust-borsBot
commented
Feb 26, 2026

Copy link


Copy Markdown

Contributor
☀️ Try build successful (CI)

Build commit: 71d4456 (71d445654a32204fbf86bfec11e657fb97adcd22, parent: 69b78537fac74de40f009b076bcbbf54b77683ad)
@rust-timer

This comment has been minimized.

Sign in to view
@ehuss
ehuss
commented
Feb 26, 2026

Copy link


Copy Markdown

Contributor
Unfortunately I am unable to test because of #153077, I am unable to get rustc to build.
@rust-timer
rust-timer
commented
Feb 26, 2026

Copy link


Copy Markdown

Collaborator
Finished benchmarking commit (71d4456): comparison URL.
Overall result: ❌✅ regressions and improvements - please read the text below
Benchmarking this pull request means it may be perf-sensitive – we'll automatically label it not fit for rolling up. You can override this, but we strongly advise not to, due to possible changes in compiler perf.
Next Steps: If you can justify the regressions found in this try perf run, please do so in sufficient writing along with @rustbot label: +perf-regression-triaged. If not, please fix the regressions and do another perf run. If its results are neutral or positive, the label will be automatically removed.
@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression
Instruction count
Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.
meanrangecount
Regressions ❌ 
 (primary)		0.3%[0.2%, 0.3%]4
Regressions ❌ 
 (secondary)		0.0%[0.0%, 0.0%]1
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-0.4%[-0.6%, -0.1%]7
All ❌✅ (primary)0.3%[0.2%, 0.3%]4
Max RSS (memory usage)
Results (secondary 0.1%)
A less reliable metric. May be of interest, but not used to determine the overall result above.
meanrangecount
Regressions ❌ 
 (primary)		--0
Regressions ❌ 
 (secondary)		6.8%[6.8%, 6.8%]1
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-6.7%[-6.7%, -6.7%]1
All ❌✅ (primary)--0
Cycles
This benchmark run did not return any relevant results for this metric.
Binary size
This benchmark run did not return any relevant results for this metric.
Bootstrap: 480.132s -> 480.258s (0.03%)
Artifact size: 395.80 MiB -> 395.77 MiB (-0.01%)
@rustbotrustbot
added

perf-regression
Performance regression.
and removed

S-waiting-on-perf
Status: Waiting on a perf run to be completed.
labels
Feb 26, 2026
@ehuss
ehuss
commented
Feb 26, 2026

Copy link


Copy Markdown

Contributor
OK, got some data:
With this PR:
Benchmark 1: ~/Proj/rust/rust/build/aarch64-apple-darwin/stage1/bin/rustc src/main.rs -L deps
Time (mean ± σ): 197.4 ms ± 3.9 ms [User: 143.7 ms, System: 117.9 ms]
Range (min … max): 192.4 ms … 207.7 ms 30 runs

Without:
Benchmark 1: ~/Proj/rust/rust/build/aarch64-apple-darwin/stage1/bin/rustc src/main.rs -L deps
Time (mean ± σ): 248.9 ms ± 5.1 ms [User: 189.5 ms, System: 124.2 ms]
Range (min … max): 241.1 ms … 262.6 ms 30 runs

The tests are super noisy, though (I get significantly different results between runs). But it seems like a pretty good win.
@nnethercote
nnethercote
commented
Feb 27, 2026

Copy link


Copy Markdown

Contributor
Nice wins.
@bors r+
@rust-bors
rust-borsBot
commented
Feb 27, 2026

Copy link


Copy Markdown

Contributor
📌 Commit f450289 has been approved by nnethercote
It is now in the queue for this repository.
@rust-borsrust-borsBot
added

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
and removed

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
labels
Feb 27, 2026
bjorn3
bjorn3
reviewed
Feb 27, 2026
View reviewed changes
Comment threadcompiler/rustc_session/src/search_paths.rs
 Err(..) => Default::default(),
 };
 files.sort_by(|lhs, rhs| lhs.file_name_str.cmp(&rhs.file_name_str));
 files.sort_unstable_by(|lhs, rhs| lhs.file_name_str.cmp(&rhs.file_name_str));
@bjorn3bjorn3Feb 27, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there no possibility of two files with the same file_name_str?
@bjorn3bjorn3Feb 27, 2026

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although I guess that would result in either one being picked deterministically (if it is the only matching one), which one is picked not mattering (if both files having the same crate hash) or an error about multiple candidates getting emitted.
@KobzolKobzolFeb 27, 2026

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We sorted (and searched) by the UTF-8 name before anyway, so there shouldn't hopefully be any change in behavior.
@rust-bors

This comment has been minimized.

Sign in to view
rust-borsBot
pushed a commit
that referenced
this pull request
Mar 1, 2026
@bors
Auto merge of#153131- Kobzol:filesearch-opt, r=nnethercote
07af97d
Optimize dependency file search
I tried to look into the slowdown reported in rust-lang/cargo#16665.
I created a Rust hello world program, and used this Python script to create a directory containing 200k files:
```python
from pathlib import Path
dir = Path("deps")
dir.mkdir(parents=True, exist_ok=True)
for i in range(200000):
path = dir / f"file{i:07}.o"
with open(path, "w") as f:
f.write("\n")
```
Then I tried to do various small microoptimalizations and simplifications to the code that iterates the search directories. Each individual commit improved performance, with the third one having the biggest effect.
Here are the results on `main` vs the last commit with the stage1 compiler on Linux, using `hyperfine "rustc +stage1 src/main.rs -L deps" -r 30` (there's IO involved, so it's good to let it run for a while):
```bash
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 299.4 ms ± 2.7 ms [User: 161.9 ms, System: 144.9 ms]
Range (min … max): 294.8 ms … 307.1 ms 30 runs
Benchmark 1: rustc +stage1 src/main.rs -L deps
Time (mean ± σ): 208.1 ms ± 4.5 ms [User: 87.3 ms, System: 128.7 ms]
Range (min … max): 202.4 ms … 219.6 ms 30 runs
```
Would be cool if someone could try this on macOS (maybe @ehuss - not sure if you have macOS or you only commented about its behavior on the Cargo issue :) ).
I also tried to prefilter the paths (not in this PR); right now we load everything and then we filter files with given prefixes, that's wasteful. Filtering just files starting with `lib` would get us down to ~150ms here. (The baseline without `-L` is ~80ms on my PC). The rest of the 70ms is essentially allocations from iterating the directory entries and sorting. That would be very hard to change - iterating the directory entries (de)allocates a lot of intermediate paths :( We'd have to implement the iteration by hand with either arena allocation, or at least some better management of memory.
r? @nnethercote
@rust-borsrust-borsBot
added

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
and removed

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
labels
Mar 1, 2026
@rust-bors
rust-borsBot
commented
Mar 1, 2026

Copy link


Copy Markdown

Contributor
💔 Test for 07af97d failed: CI. Failed job:
@rust-log-analyzer
rust-log-analyzer
commented
Mar 1, 2026

Copy link


Copy Markdown

Collaborator
The job aarch64-gnu failed! Check out the build log: (web)(plain enhanced)(plain)
Click to see the possible cause of the failure (guessed by this bot)
/dev/sda15 98M 6.4M 92M 7% /boot/efi
tmpfs 1.6G 16K 1.6G 1% /run/user/1001
================================================================================
Sufficient disk space available (120307208KB >= 52428800KB). Skipping cleanup.
##[group]Run src/ci/scripts/setup-environment.sh
src/ci/scripts/setup-environment.sh
shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
---
[1773/3894] Building ARMGenMCCodeEmitter.inc...
##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
[1774/3894] Building ARMGenMCPseudoLowering.inc...
Session terminated, killing shell...::group::Clock drift check
local time: Sun Mar 1 08:40:58 UTC 2026
network time: [1775/3894] Building ARMGenInstrInfo.inc...
Sun, 01 Mar 2026 08:40:58 GMT
##[endgroup]
[1776/3894] Building ARMGenRegisterBank.inc...

@Kobzol
Kobzol
commented
Mar 1, 2026

Copy link


Copy Markdown

MemberAuthor
@bors retry
Spurious worker restart.
@rust-borsrust-borsBot
added

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
and removed

S-waiting-on-review
Status: Awaiting review from the assignee but also interested parties.
labels
Mar 1, 2026

This was referenced Mar 1, 2026
 Merged

 Merged

@rust-bors

This comment has been minimized.

Sign in to view
@rust-borsrust-borsBot
added

merged-by-bors
This PR was explicitly merged by bors.
and removed

S-waiting-on-bors
Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
labels
Mar 2, 2026
@rust-bors
rust-borsBot
commented
Mar 2, 2026

Copy link


Copy Markdown

Contributor
☀️ Test successful - CI

Approved by: nnethercote

Duration: 3h 40m 3s

Pushing e7d90c6 to main...
@rust-bors
rust-borsBot
merged commit e7d90c6
into
rust-lang:mainMar 2, 2026

13 checks passed

@rustbotrustbot
added this to the 1.96.0 milestone
Mar 2, 2026
@github-actions
github-actionsBot
commented
Mar 2, 2026

Copy link


Copy Markdown

Contributor
What is this?
This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 8038127 (parent) -> e7d90c6 (this PR)
Test differences
Show 2 test diffs

2 doctest diffs were found. These are ignored, as they are noisy.
Test dashboard
Run
cargo run --manifest-path src/ci/citool/Cargo.toml -- \
test-dashboard e7d90c695a39426baf5ae705de2f9570a72229de --output-dir test-dashboard
And then open test-dashboard/index.html in your browser to see an overview of all executed tests.
Job duration changes
  1. dist-aarch64-llvm-mingw: 1h 33m -> 1h 46m (+14.1%)
  2. dist-aarch64-apple: 1h 56m -> 2h 12m (+13.1%)
  3. aarch64-apple: 3h 8m -> 3h 33m (+13.0%)
  4. dist-x86_64-apple: 2h 42m -> 2h 25m (-10.7%)
  5. arm-android: 1h 37m -> 1h 47m (+10.4%)
  6. dist-ohos-x86_64: 1h 14m -> 1h 21m (+9.7%)
  7. x86_64-gnu-aux: 2h 7m -> 2h 18m (+8.5%)
  8. x86_64-gnu-distcheck: 2h 7m -> 2h 18m (+8.4%)
  9. x86_64-gnu-miri: 1h 23m -> 1h 30m (+7.7%)
  10. dist-arm-linux-musl: 1h 38m -> 1h 30m (-7.3%)
How to interpret the job duration changes?
Job durations can vary a lot, based on the actual runner instance

that executed the job, system noise, invalidated caches, etc. The table above is provided

mostly for t-infra members, for simpler debugging of potential CI slow-downs.
@rust-timer
rust-timer
commented
Mar 2, 2026

Copy link


Copy Markdown

Collaborator
Finished benchmarking commit (e7d90c6): comparison URL.
Overall result: ❌✅ regressions and improvements - please read the text below
Our benchmarks found a performance regression caused by this PR.

This might be an actual regression, but it can also be just noise.
Next Steps:
  • If the regression was expected or you think it can be justified,
    
please write a comment with sufficient written justification, and add
    @rustbot label: +perf-regression-triaged to it, to mark the regression as triaged.
  • If you think that you know of a way to resolve the regression, try to create
    
a new PR with a fix for the regression.
  • If you do not understand the regression or you think that it is just noise,
    
you can ask the @rust-lang/wg-compiler-performance working group for help (members of this group
    
were already notified of this PR).
@rustbot label: +perf-regression

cc @rust-lang/wg-compiler-performance
Instruction count
Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.
meanrangecount
Regressions ❌ 
 (primary)		0.1%[0.1%, 0.1%]1
Regressions ❌ 
 (secondary)		--0
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-0.4%[-0.6%, -0.2%]7
All ❌✅ (primary)0.1%[0.1%, 0.1%]1
Max RSS (memory usage)
Results (secondary -4.0%)
A less reliable metric. May be of interest, but not used to determine the overall result above.
meanrangecount
Regressions ❌ 
 (primary)		--0
Regressions ❌ 
 (secondary)		--0
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		-4.0%[-5.4%, -2.6%]3
All ❌✅ (primary)--0
Cycles
This benchmark run did not return any relevant results for this metric.
Binary size
Results (secondary 0.1%)
A less reliable metric. May be of interest, but not used to determine the overall result above.
meanrangecount
Regressions ❌ 
 (primary)		--0
Regressions ❌ 
 (secondary)		0.1%[0.1%, 0.1%]3
Improvements ✅ 
 (primary)		--0
Improvements ✅ 
 (secondary)		--0
All ❌✅ (primary)--0
Bootstrap: 480.971s -> 480.149s (-0.17%)
Artifact size: 396.96 MiB -> 396.96 MiB (0.00%)
@Kobzol
Kobzol
commented
Mar 2, 2026•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
The regression seems like noise. Tiny wins on large-workspace, matching pre-merge run.
@rustbot label: +perf-regression-triaged
@rustbotrustbot
added
the 
perf-regression-triaged
The performance regression has been triaged.
label
Mar 2, 2026
@osiewicz
osiewicz
commented
Mar 14, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Timings for Mac, as requested in OP (with the same scenario as the one exercised in OP):
➜ rustc-153131 hyperfine "rustc +nightly-2026-03-01 main.rs -L deps" -r 30
Benchmark 1: rustc +nightly-2026-03-01 main.rs -L deps
Time (mean ± σ): 215.3 ms ± 1.9 ms [User: 161.2 ms, System: 122.0 ms]
Range (min … max): 212.2 ms … 220.7 ms 30 runs

vs:
➜ rustc-153131 hyperfine "rustc +nightly-2026-03-06 main.rs -L deps" -r 30
Benchmark 1: rustc +nightly-2026-03-06 main.rs -L deps
Time (mean ± σ): 177.9 ms ± 1.6 ms [User: 128.5 ms, System: 120.6 ms]
Range (min … max): 175.2 ms … 183.2 ms 30 runs

And a baseline:
➜ rustc-153131 hyperfine "rustc +nightly-2026-03-06 main.rs" -r 30
Benchmark 1: rustc +nightly-2026-03-06 main.rs
Time (mean ± σ): 68.0 ms ± 1.1 ms [User: 87.0 ms, System: 52.7 ms]
Range (min … max): 66.8 ms … 71.9 ms 30 runs

@Kobzol
Kobzol
deleted the

filesearch-opt

branch
March 14, 2026 10:10
@KobzolKobzol
mentioned this pull request
Mar 15, 2026

Open

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@petrochenkovpetrochenkovpetrochenkov left review comments
@bjorn3bjorn3bjorn3 left review comments
@nnethercotennethercotennethercote approved these changes

Assignees

@nnethercotennethercote

Labels

merged-by-borsThis PR was explicitly merged by bors.perf-regressionPerformance regression.perf-regression-triagedThe performance regression has been triaged.T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.

Projects


None yet


Milestone

1.96.0

Development

Successfully merging this pull request may close these issues.

9 participants

@Kobzol@rust-timer@ehuss@nnethercote@rust-log-analyzer@osiewicz@petrochenkov@bjorn3@rustbot