Let io::copy reuse BufWriter buffers by the8472 · Pull Request #78641 · rust-lang/rust

the8472 · 2020-11-01T19:28:33Z

This optimization will allow users to implicitly set the buffer size for io::copy by wrapping the writer into a BufWriter if the default block size is insufficient, which should fix#49921

Due to min_specialization limitations this approach only works with BufWriter but not for BufReader<R> since R is unconstrained and thus the necessary specialization on R: Read is not always applicable. Once specialization becomes more powerful this optimization could be extended to look at the reader and writer side and use whichever buffer is larger.

rust-highfive · 2020-11-01T19:28:37Z

r? @shepmaster

(rust_highfive has picked a reviewer for you, use r? to override)

bors · 2020-11-14T14:57:53Z

☔ The latest upstream changes (presumably #75272) made this pull request unmergeable. Please resolve the merge conflicts.

Note that reviewers usually do not review pull requests until merge conflicts are resolved! Once you resolve the conflicts, you should change the labels applied by bors to indicate that your PR is ready for review. Post this as a comment to change the labels:

@rustbot modify labels: +S-waiting-on-review -S-waiting-on-author

the8472 · 2020-11-18T20:53:44Z

The resulting code feels a bit gnarly but I didn't want to duplicate the read-write loop because it contains all that unsafe code around uninitialized memory.

If it's accaptable to duplicate that code then the generic and the BufWriter-specific version could individually be made cleaner.

the8472 · 2020-12-12T14:55:06Z

I'll hold off fixing the nits until I get a higher-level review.

shepmaster · 2021-01-01T20:02:21Z

I'ven't been able to give this the time it deserves.

r? @sfackler

sfackler · 2021-01-01T20:32:58Z

It seems strange for there to be specialization in the function used "when specializations are not available or not applicable".

That refers to another set of specializations. I'll try to reword it.

sfackler · 2021-02-01T01:09:52Z

This looks good, thanks!

@bors r+

bors · 2021-02-01T01:09:53Z

📌 Commit 4105506 has been approved by sfackler

sfackler · 2021-02-01T01:10:22Z

+ // Safety: The initializer contract guarantees that either it or `read`
+ // will have initialized these bytes. And we just checked that the number
+ // of bytes is within the buffer capacity.
+ unsafe { buf.set_len(buf.len() + bytes_read) };


Huh, is std being built with polonious?

Not sure what is causing surprise here. The x.foo(x.bar())? Isn't that just two-phase borrows?

bors · 2021-02-01T05:27:21Z

⌛ Testing commit 4105506 with merge 8bd506e0f39d10dcc588ae469c9c6c73d0c568c6...

bors · 2021-02-01T09:42:22Z

💥 Test timed out

bjorn3 · 2021-02-01T09:47:10Z

The macOS builder is still running and showing new logs.

rust-log-analyzer · 2021-02-01T10:16:54Z

A job failed! Check out the build log: (web)(plain)

Click to see the possible cause of the failure (guessed by this bot)

bjorn3 · 2021-02-01T10:20:38Z

@bors r+ retry

bors · 2021-02-01T10:20:39Z

💡 This pull request was already approved, no need to approve it again.

This pull request previously failed. You should add more commits to fix the bug, or use retry to trigger a build again.
There's another pull request that is currently being tested, blocking this pull request: Rollup of 8 pull requests #81620

bors · 2021-02-01T10:20:40Z

📌 Commit 4105506 has been approved by bjorn3

bjorn3 · 2021-02-01T11:54:28Z

Oops, should have done

@bors r=sfackler

bors · 2021-02-01T11:54:29Z

💡 This pull request was already approved, no need to approve it again.

There's another pull request that is currently being tested, blocking this pull request: Implement Rust 2021 panic #80851

bors · 2021-02-01T11:54:30Z

📌 Commit 4105506 has been approved by sfackler

Let io::copy reuse BufWriter buffers This optimization will allow users to implicitly set the buffer size for io::copy by wrapping the writer into a `BufWriter` if the default block size is insufficient, which should fixrust-lang#49921 Due to min_specialization limitations this approach only works with `BufWriter` but not for `BufReader<R>` since `R` is unconstrained and thus the necessary specialization on `R: Read` is not always applicable. Once specialization becomes more powerful this optimization could be extended to look at the reader and writer side and use whichever buffer is larger.

…as-schievink Rollup of 12 pull requests Successful merges: - rust-lang#78641 (Let io::copy reuse BufWriter buffers) - rust-lang#79291 (Add error message for private fn) - rust-lang#81364 (Improve `rustc_mir_build::matches` docs) - rust-lang#81387 (Move some tests to more reasonable directories - 3) - rust-lang#81463 (Rename NLL* to Nll* accordingly to C-CASE) - rust-lang#81504 (Suggest accessing field when appropriate) - rust-lang#81529 (Fix invalid camel case suggestion involving unicode idents) - rust-lang#81536 (Indicate both start and end of pass RSS in time-passes output) - rust-lang#81592 (Rustdoc UI fixes) - rust-lang#81594 (Avoid building LLVM just for llvm-dwp) - rust-lang#81598 (Fix calling convention for CRT startup) - rust-lang#81618 (Sync rustc_codegen_cranelift) Failed merges: r? `@ghost` `@rustbot` modify labels: rollup

pickfire · 2021-02-18T02:03:51Z

+
+ writer.flush_buf()?;


I think putting this in an else branch will be clearer, if the previous conditions are hit all will either continue or return, it does not reach this part.

rust-highfive assigned shepmaster Nov 1, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Nov 1, 2020

the8472 commented Nov 1, 2020
View reviewed changes

Comment threadlibrary/std/src/io/copy.rs Outdated

the8472 force-pushed the buffered-copy branch from 9f4919c to 8623f04Compare November 1, 2020 19:51

jyn514 added the T-libs-api [DEPRECATED; DO NOT USE] label Nov 1, 2020

the8472 marked this pull request as draft November 13, 2020 20:09

shepmaster added S-blocked Status: Blocked on something else such as an RFC or other implementation work. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 17, 2020

the8472 force-pushed the buffered-copy branch from 8623f04 to 221c8fcCompare November 17, 2020 19:01

the8472 marked this pull request as ready for review November 17, 2020 19:02

This comment has been minimized.
Sign in to view

rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Nov 17, 2020

This comment has been minimized.
Sign in to view

rustbot removed the S-blocked Status: Blocked on something else such as an RFC or other implementation work. label Nov 26, 2020

the8472 mentioned this pull request Dec 11, 2020
Optimize BufWriter #79930
Merged

pickfire reviewed Dec 12, 2020
View reviewed changes

Comment threadlibrary/std/src/io/copy.rs Outdated

pickfire reviewed Dec 12, 2020
View reviewed changes

Comment threadlibrary/std/src/io/copy.rs Outdated

pickfire reviewed Dec 12, 2020
View reviewed changes

Comment threadlibrary/std/src/io/copy.rs Outdated

This comment has been minimized.
Sign in to view

rust-highfive assigned sfackler and unassigned shepmaster Jan 1, 2021

sfackler reviewed Jan 1, 2021
View reviewed changes

Comment threadlibrary/std/src/io/buffered/bufwriter.rs Outdated

sfackler reviewed Jan 1, 2021
View reviewed changes

Comment threadlibrary/std/src/io/copy.rs Outdated

sfackler reviewed Jan 1, 2021
View reviewed changes

Comment threadlibrary/std/src/io/copy.rs Outdated

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 1, 2021

sfackler reviewed Feb 1, 2021
View reviewed changes

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Feb 1, 2021

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 1, 2021

henryboisdequin mentioned this pull request Feb 1, 2021
Rollup of 7 pull requests #81621
Closed

jonas-schievink mentioned this pull request Feb 1, 2021
Rollup of 13 pull requests #81622
Closed

jonas-schievink mentioned this pull request Feb 1, 2021
Rollup of 12 pull requests #81625
Merged

bors merged commit a7a6f01 into rust-lang:masterFeb 1, 2021

rustbot added this to the 1.51.0 milestone Feb 1, 2021

the8472 mentioned this pull request Feb 1, 2021
std::io::copy performance may be 20x slower than it could be? #49921
Closed

pickfire reviewed Feb 18, 2021
View reviewed changes

Uh oh!

Uh oh!

Conversation

the8472 commented Nov 1, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rust-highfive commented Nov 1, 2020

Uh oh!

Uh oh!

bors commented Nov 14, 2020

Uh oh!

This comment has been minimized.

the8472 commented Nov 18, 2020

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

the8472 commented Dec 12, 2020

Uh oh!

This comment has been minimized.

shepmaster commented Jan 1, 2021

Uh oh!

Uh oh!

sfacklerJan 1, 2021

Choose a reason for hiding this comment

Uh oh!

the8472Jan 1, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sfackler commented Feb 1, 2021

Uh oh!

bors commented Feb 1, 2021

Uh oh!

sfacklerFeb 1, 2021

Choose a reason for hiding this comment

Uh oh!

the8472Feb 1, 2021

Choose a reason for hiding this comment

Uh oh!

bors commented Feb 1, 2021

Uh oh!

bors commented Feb 1, 2021

Uh oh!

bjorn3 commented Feb 1, 2021

Uh oh!

rust-log-analyzer commented Feb 1, 2021

Uh oh!

bjorn3 commented Feb 1, 2021

Uh oh!

bors commented Feb 1, 2021

Uh oh!

bors commented Feb 1, 2021

Uh oh!

bjorn3 commented Feb 1, 2021

Uh oh!

bors commented Feb 1, 2021

Uh oh!

bors commented Feb 1, 2021

Uh oh!

pickfireFeb 18, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

the8472 commented Nov 1, 2020•
edited
Loading