Implement `read_offset` and `write_offset` by tbu- · Pull Request #35704 · rust-lang/rust

tbu- · 2016-08-16T00:18:17Z

These functions allow to read from and write to a file from multiple
threads without changing the per-file cursor, avoiding the race between
the seek and the read.

rust-highfive · 2016-08-16T00:18:21Z

r? @aturon

(rust_highfive has picked a reviewer for you, use r? to override)

sfackler · 2016-08-16T19:16:42Z

 })
 }
+
+ /// Reads a number of bytes starting from a given offset.


It's specifically an offset from the start of the file, not the current cursor position, right? Could you add that to the docs?

alexcrichton · 2016-08-16T20:37:24Z

Can you also be sure to add tests for this? Especially with how it relates to the "current file pointer" and such

tbu- · 2016-08-22T23:00:50Z

Added docs and tests.

alexcrichton · 2016-08-23T05:46:27Z

Discussed during @rust-lang/libs triage today the conclusion was that we probably want to call these {read,write}_at but beyond that they're good to go!

Could you also file a tracking issue and update the tracking issue here?

tbu- · 2016-08-23T08:36:51Z

Done.

sfackler · 2016-08-23T09:00:34Z

@bors r+ 4e48924

Implement `read_offset` and `write_offset` These functions allow to read from and write to a file from multiple threads without changing the per-file cursor, avoiding the race between the seek and the read.

bors · 2016-08-23T18:30:41Z

⌛ Testing commit 4e48924 with merge 6ba6e27...

bors · 2016-08-23T18:34:32Z

💔 Test failed - auto-linux-64-cross-freebsd

eddyb · 2016-08-25T01:01:54Z

@bors retry

bors · 2016-08-25T04:49:25Z

⌛ Testing commit 4e48924 with merge 2d35fe1...

bors · 2016-08-25T05:59:33Z

💔 Test failed - auto-win-msvc-64-opt-rustbuild

tbu- · 2016-08-25T07:40:02Z

This failure is definitely my fault.

tbu- · 2016-08-25T10:11:51Z

@alexcrichton New situation, the Windows version of this differs from the Unix one: WriteFile updates the file pointer, pwrite doesn't.

Do we still want this and document the difference between platforms?

I don't see a way to emulate either platform's behavior on the other one atomically (atomicity is what this function is about).

sfackler · 2016-08-25T10:22:44Z

That seems like a pretty significant platform difference. :(

alexcrichton · 2016-08-25T17:31:32Z

Yeah that seems significant enough that we wouldn't provide a cross platform API for this. Perhaps a platform-specific one in std::os::*::fs, though.

ranma42 · 2016-08-29T09:34:52Z

@tbu- my understanding of the Windows API is that it would be possible to avoid the update of the file pointer if the file had been opened with the flag FILE_FLAG_OVERLAPPED. This seems to be required anyway if multiple threads are accessing the file concurrently (without this flag, the operations would happen sequentially).

Would it be acceptable to only allow read_offset and write_offset on OVERLAPPED files on Windows?

arthurprs · 2016-09-04T12:55:27Z

@ranma42 do we have a way to open files with OVERLAPPED flag using just the stdlib?

ranma42 · 2016-09-04T14:04:47Z

@arthurprs Right now the OVERLAPPED flag is available through the Windows-specific extension for OpenOption: #27720

This is already used in the stdlib to open pipes as overlapped files:

rust/src/libstd/sys/windows/pipe.rs

Line 97 in e77d86c

opts.attributes(c::FILE_FLAG_OVERLAPPED);

tbu- · 2016-10-12T11:08:31Z

Lints should be the last stage before tests, fixed.

alexcrichton · 2016-10-12T14:13:01Z

@bors: r+

bors · 2016-10-12T14:13:02Z

📌 Commit 744aecf has been approved by alexcrichton

bors · 2016-10-12T16:04:04Z

⌛ Testing commit 744aecf with merge 2481c23...

bors · 2016-10-12T16:58:05Z

💔 Test failed - auto-linux-64-x-android-t

tbu- · 2016-10-12T23:31:47Z

It seems the signature of ftruncate in libc is wrong, or the wrong signature is being used here (I removed the local definition of ftruncate. What is the preferred course of action?

Is off_t always i64 on Android and we don't need this fallback? Is it only sometimes i64, and we should only use this signature when it's not? Or is the definition in libc completely wrong, and it's always i32?

alexcrichton · 2016-10-13T03:26:59Z

IIRC it's because this is an older version of Android compatibility. I think newer versions changed the definition of ftruncate (e.g. the ABI changed). I forget the exact details here though. For now seems fine to just leave it.

tbu- · 2016-10-14T12:25:28Z

I changed it to only look for {ftruncate,pread,pwrite}64 on 32 bit systems, on 64 bit systems, off_t should be i64.

alexcrichton · 2016-10-14T18:18:28Z

@bors: r+

bors · 2016-10-14T18:18:29Z

📌 Commit 94aa08b has been approved by alexcrichton

Implement `read_offset` and `write_offset` These functions allow to read from and write to a file from multiple threads without changing the per-file cursor, avoiding the race between the seek and the read.

bors · 2016-10-14T18:58:44Z

⌛ Testing commit 94aa08b with merge 8c09454...

bors · 2016-10-14T20:40:05Z

💔 Test failed - auto-linux-64-x-android-t

alexcrichton · 2016-10-14T21:06:01Z

@bors: r+

bors · 2016-10-14T21:06:02Z

📌 Commit 1554993 has been approved by alexcrichton

bors · 2016-10-15T02:33:04Z

⌛ Testing commit 1554993 with merge e1b6777...

Implement `read_offset` and `write_offset` These functions allow to read from and write to a file from multiple threads without changing the per-file cursor, avoiding the race between the seek and the read.

bors · 2016-10-15T08:48:40Z

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-cargotest, auto-linux-64-cross-freebsd, auto-linux-64-debug-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-opt-rustbuild, auto-linux-64-x-android-t, auto-linux-cross-opt, auto-linux-musl-64-opt, auto-mac-32-opt, auto-mac-64-opt, auto-mac-64-opt-rustbuild, auto-mac-cross-ios-opt, auto-win-gnu-32-opt, auto-win-gnu-32-opt-rustbuild, auto-win-gnu-64-opt, auto-win-msvc-32-opt, auto-win-msvc-64-cargotest, auto-win-msvc-64-opt, auto-win-msvc-64-opt-rustbuild
Approved by: alexcrichton
Pushing e1b6777 to master...

tbu- · 2016-10-15T09:02:59Z

Finally! Thanks for your patience, @alexcrichton.

tbu- · 2016-10-28T14:42:30Z

Test.

Currently the documentation of `FileExt::seek_write` on Windows indicates that writes beyond the end of the file leave intermediate bytes uninitialized. This commentary dates back to the original inclusion of these functions in rust-lang#35704 (wow blast from the past!). At the time the functionality here was implemented using `WriteFile`, but nowadays the `NtWriteFile` method is used instead. The documentation for `NtWriteFile` explicitly states: > If Length and ByteOffset specify a write operation past the current > end-of-file mark, NtWriteFile automatically extends the file and updates > the end-of-file mark; any bytes that are not explicitly written between > such old and new end-of-file marks are defined to be zero. This commentary has had a downstream impact in the `system-interface` crate where it tries to handle this by explicitly writing zeros, but I don't believe that's necessary any more. I'm sending a PR upstream here to avoid future confusion and codify that zeros are written in the intermediate bytes matching what Windows currently provides.

…rite-docs, r=ChrisDenton std: Update documentation of seek_write on Windows Currently the documentation of `FileExt::seek_write` on Windows indicates that writes beyond the end of the file leave intermediate bytes uninitialized. This commentary dates back to the original inclusion of these functions in rust-lang#35704 (wow blast from the past!). At the time the functionality here was implemented using `WriteFile`, but nowadays the `NtWriteFile` method is used instead. The documentation for `NtWriteFile` explicitly states: > If Length and ByteOffset specify a write operation past the current > end-of-file mark, NtWriteFile automatically extends the file and updates > the end-of-file mark; any bytes that are not explicitly written between > such old and new end-of-file marks are defined to be zero. This commentary has had a downstream impact in the `system-interface` crate where it tries to handle this by explicitly writing zeros, but I don't believe that's necessary any more. I'm sending a PR upstream here to avoid future confusion and codify that zeros are written in the intermediate bytes matching what Windows currently provides.

Rollup merge of rust-lang#120452 - alexcrichton:update-windows-seek-write-docs, r=ChrisDenton std: Update documentation of seek_write on Windows Currently the documentation of `FileExt::seek_write` on Windows indicates that writes beyond the end of the file leave intermediate bytes uninitialized. This commentary dates back to the original inclusion of these functions in rust-lang#35704 (wow blast from the past!). At the time the functionality here was implemented using `WriteFile`, but nowadays the `NtWriteFile` method is used instead. The documentation for `NtWriteFile` explicitly states: > If Length and ByteOffset specify a write operation past the current > end-of-file mark, NtWriteFile automatically extends the file and updates > the end-of-file mark; any bytes that are not explicitly written between > such old and new end-of-file marks are defined to be zero. This commentary has had a downstream impact in the `system-interface` crate where it tries to handle this by explicitly writing zeros, but I don't believe that's necessary any more. I'm sending a PR upstream here to avoid future confusion and codify that zeros are written in the intermediate bytes matching what Windows currently provides.

rust-highfive assigned aturon Aug 16, 2016

sfackler added the T-libs-api [DEPRECATED; DO NOT USE] label Aug 16, 2016

sfackler reviewed Aug 16, 2016
View reviewed changes

tbu- mentioned this pull request Aug 23, 2016
Tracking issue for File::read_at, File::write_at (file_offset) #35918
Closed

eddyb mentioned this pull request Aug 23, 2016
Rollup of 15 pull requests #35944
Closed

eddyb mentioned this pull request Aug 23, 2016
Rollup of 13 pull requests #35945
Closed

Only use Android fallback for {ftruncate,pread,pwrite} on 32 bit
94aa08b

Android: Fix unused-imports warning
1554993

bors merged commit 1554993 into rust-lang:masterOct 15, 2016

PSeitz mentioned this pull request Feb 21, 2017
rustc suggests std::os::ext::fs::FileExt then can't find it #38978
Closed

PSeitz mentioned this pull request Jan 6, 2018
File read_at, non-mutable platform independent version #47229
Closed

alexcrichton mentioned this pull request Jan 28, 2024
std: Update documentation of seek_write on Windows #120452
Merged

Uh oh!

Uh oh!

Conversation

tbu- commented Aug 16, 2016

Uh oh!

rust-highfive commented Aug 16, 2016

Uh oh!

sfacklerAug 16, 2016

Choose a reason for hiding this comment

Uh oh!

alexcrichton commented Aug 16, 2016

Uh oh!

tbu- commented Aug 22, 2016

Uh oh!

alexcrichton commented Aug 23, 2016

Uh oh!

tbu- commented Aug 23, 2016

Uh oh!

sfackler commented Aug 23, 2016

Uh oh!

bors commented Aug 23, 2016

Uh oh!

bors commented Aug 23, 2016

Uh oh!

eddyb commented Aug 25, 2016

Uh oh!

bors commented Aug 25, 2016

Uh oh!

bors commented Aug 25, 2016

Uh oh!

tbu- commented Aug 25, 2016

Uh oh!

tbu- commented Aug 25, 2016

Uh oh!

sfackler commented Aug 25, 2016

Uh oh!

alexcrichton commented Aug 25, 2016

Uh oh!

ranma42 commented Aug 29, 2016

Uh oh!

arthurprs commented Sep 4, 2016

Uh oh!

ranma42 commented Sep 4, 2016

Uh oh!

tbu- commented Oct 12, 2016

Uh oh!

alexcrichton commented Oct 12, 2016

Uh oh!

bors commented Oct 12, 2016

Uh oh!

bors commented Oct 12, 2016

Uh oh!

bors commented Oct 12, 2016

Uh oh!

tbu- commented Oct 12, 2016

Uh oh!

alexcrichton commented Oct 13, 2016

Uh oh!

tbu- commented Oct 14, 2016

Uh oh!

alexcrichton commented Oct 14, 2016

Uh oh!

bors commented Oct 14, 2016

Uh oh!

bors commented Oct 14, 2016

Uh oh!

bors commented Oct 14, 2016

Uh oh!

alexcrichton commented Oct 14, 2016

Uh oh!

bors commented Oct 14, 2016

Uh oh!

bors commented Oct 15, 2016

Uh oh!

bors commented Oct 15, 2016

Uh oh!

tbu- commented Oct 15, 2016

Uh oh!

tbu- commented Oct 28, 2016

Uh oh!