Workarounds for copy_file_range issues by the8472 · Pull Request #75428 · rust-lang/rust

the8472 · 2020-08-11T23:31:52Z

…ttemted on a NFS mount under RHEL/CentOS 7. The syscall is supposed to return ENOSYS in most cases but when calling it on NFS it may leak through EOPNOTSUPP even though that's supposed to be handled by the kernel and not returned to userspace. Since it returns ENOSYS in some cases anyway this will trip the HAS_COPY_FILE_RANGE detection anyway, so treat EOPNOTSUPP as if it were a ENOSYS. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.8_release_notes/deprecated_functionality#the_literal_copy_file_range_literal_call_has_been_disabled_on_local_file_systems_and_in_nfs https://bugzilla.redhat.com/show_bug.cgi?id=1783554

rust-highfive · 2020-08-11T23:31:55Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @joshtriplett (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

joshtriplett · 2020-08-12T01:06:37Z

@bors r+

bors · 2020-08-12T01:06:38Z

📌 Commit 1316c78 has been approved by joshtriplett

bors · 2020-08-12T11:37:16Z

⌛ Testing commit 1316c78 with merge c0e06fb4d88cf4063d23e407c155a2096b4ac611...

rust-log-analyzer · 2020-08-12T11:41:27Z

Your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @rust-lang/infra. (Feature Requests)

bors · 2020-08-12T11:57:18Z

💔 Test failed - checks-actions

the8472 · 2020-08-12T18:11:40Z

The bors error is not helpful, spurious failure?

… on file size This solves several problems - race conditions where a file is truncated while copying from it. if we blindly trusted the file size this would lead to an infinite loop - proc files appearing empty to copy_file_range but not to read/write coreutils/coreutils@4b04a0c - copy_file_range returning 0 for some filesystems (overlay? bind mounts?) inside docker, again leading to an infinite loop

the8472 · 2020-08-14T21:19:10Z

Updated to address an additional copy_file_range issue.

the8472 · 2020-08-14T21:59:55Z

ping @joshtriplett

tavianator · 2020-08-15T19:42:37Z

+ while written < max_len {
 let copy_result = if has_copy_file_range {
- let bytes_to_copy = cmp::min(len - written, usize::MAX as u64) as usize;
+ let bytes_to_copy = cmp::min(max_len - written, usize::MAX as u64) as usize;


Is it really necessary to keep track of max_len - written accurately at this point? It's probably okay just to pass a large constant size on every call.

It avoids overflowing an u64. Plus I want to reuse the code in #75272 which will require exact max sizes

bors · 2020-09-05T04:15:45Z

⌛ Testing commit 4ddedd5 with merge 596b8ed70633bf181a3b8966fb235e91a67ae146...

bors · 2020-09-05T08:30:45Z

💥 Test timed out

Dylan-DPC-zz · 2020-09-05T12:43:14Z

@bors retry

…triplett Workarounds for copy_file_range issues fixesrust-lang#75387fixesrust-lang#75446

bors · 2020-09-05T13:55:35Z

⌛ Testing commit 4ddedd5 with merge f9b36230f023ae6f396f3cc4284cd41a0583b819...

bors · 2020-09-05T14:59:24Z

💔 Test failed - checks-actions

the8472 · 2020-09-05T16:43:43Z

I don't see how linux-only code changes can make a mingw build fail. Doubly so when it's doc tests.

nagisa · 2020-09-05T18:52:01Z

@bors retry

bors · 2020-09-05T19:09:27Z

⌛ Testing commit 4ddedd5 with merge de921ab...

bors · 2020-09-05T21:02:11Z

☀️ Test successful - checks-actions, checks-azure
Approved by: joshtriplett
Pushing de921ab to master...

implement better availability probing for copy_file_range Followup to rust-lang#75428 (comment) Previously syscall detection was overly pessimistic. Any attempt to copy to an immutable file (EPERM) would disable copy_file_range support for the whole process. The change tries to copy_file_range on invalid file descriptors which will never run into the immutable file case and thus we can clearly distinguish syscall availability.

rust-highfive assigned joshtriplett Aug 11, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Aug 11, 2020

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 12, 2020

tesuji reviewed Aug 12, 2020
View reviewed changes

Comment threadlibrary/std/src/sys/unix/fs.rs Outdated

tesuji reviewed Aug 12, 2020
View reviewed changes

Comment threadlibrary/std/src/sys/unix/fs.rs Outdated

Tavi-Kohn reviewed Aug 12, 2020
View reviewed changes

Comment threadlibrary/std/src/sys/unix/fs.rs

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Aug 12, 2020

more concise error matching
f078363

nagisa reviewed Aug 13, 2020
View reviewed changes

Comment threadlibrary/std/src/sys/unix/fs.rs

the8472 changed the title ~~Workaround for copy_file_range spuriously returning EOPNOTSUPP on CentOS 7~~Workarounds for copy_file_range issuesAug 14, 2020

tavianator reviewed Aug 15, 2020
View reviewed changes

the8472 mentioned this pull request Sep 4, 2020
specialize io::copy to use copy_file_range, splice or sendfile #75272
Merged

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 5, 2020

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Sep 5, 2020

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 5, 2020

Dylan-DPC-zz pushed a commit to Dylan-DPC-zz/rust that referenced this pull request Sep 5, 2020
Rollup merge ofrust-lang#75428- the8472:fix-copy-eopnotsupp, r=josh…
5e60cf8
…triplett Workarounds for copy_file_range issues fixesrust-lang#75387fixesrust-lang#75446

Dylan-DPC-zz mentioned this pull request Sep 5, 2020
Rollup of 14 pull requests #76373
Closed

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Sep 5, 2020

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 5, 2020

bors added the merged-by-bors This PR was explicitly merged by bors. label Sep 5, 2020

bors merged commit de921ab into rust-lang:masterSep 5, 2020

rustbot added this to the 1.48.0 milestone Sep 5, 2020

the8472 mentioned this pull request Nov 21, 2020
implement better availability probing for copy_file_range #79274
Merged

leomao10 mentioned this pull request May 19, 2021
Very specific set of circumstances leads to zero-byte (empty) file being created docker/for-linux#1015
Closed
3 tasks

the8472 mentioned this pull request Jun 29, 2021
Use copy_file_range(2) only on Linux >= 5.3 #86718
Closed

Uh oh!

Uh oh!

Conversation

the8472 commented Aug 11, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rust-highfive commented Aug 11, 2020

Uh oh!

joshtriplett commented Aug 12, 2020

Uh oh!

bors commented Aug 12, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bors commented Aug 12, 2020

Uh oh!

rust-log-analyzer commented Aug 12, 2020

Uh oh!

bors commented Aug 12, 2020

Uh oh!

the8472 commented Aug 12, 2020

Uh oh!

Uh oh!

the8472 commented Aug 14, 2020

Uh oh!

the8472 commented Aug 14, 2020

Uh oh!

tavianatorAug 15, 2020

Choose a reason for hiding this comment

Uh oh!

the8472Aug 15, 2020

Choose a reason for hiding this comment

Uh oh!

bors commented Sep 5, 2020

Uh oh!

bors commented Sep 5, 2020

Uh oh!

Dylan-DPC-zz commented Sep 5, 2020

Uh oh!

bors commented Sep 5, 2020

Uh oh!

bors commented Sep 5, 2020

Uh oh!

the8472 commented Sep 5, 2020

Uh oh!

nagisa commented Sep 5, 2020

Uh oh!

bors commented Sep 5, 2020

Uh oh!

bors commented Sep 5, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

the8472 commented Aug 11, 2020•
edited
Loading