Skip to content

fs: added fs.writev() which exposes syscalls writev() - #25925

Closed
AnasAboreeda wants to merge 3 commits into
nodejs:masterfrom
AnasAboreeda:expose-syscall-writev-to-fs
Closed

fs: added fs.writev() which exposes syscalls writev()#25925
AnasAboreeda wants to merge 3 commits into
nodejs:masterfrom
AnasAboreeda:expose-syscall-writev-to-fs

Conversation

@AnasAboreeda

@AnasAboreedaAnasAboreeda commented Feb 4, 2019

Copy link
Copy Markdown
Contributor

fs with writev allow many buffers to be pushed to underlying OS
APIs in one batch, so this should improve write speed to files.

I have tried to follow the old fs.write() function signature
and make all the features of it available when users use fs.writev()

Fixes: #2298

: expose-syscall-writev-to-fs

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines

@nodejs-github-botnodejs-github-bot added doc Issues and PRs related to the documentations. fs Issues and PRs related to the fs subsystem / file system. tools Issues and PRs related to the tools directory. labels Feb 4, 2019
mscdex
mscdex reviewed Feb 4, 2019
View reviewed changes
Comment threadlib/fs.js Outdated
mscdex
mscdex reviewed Feb 4, 2019
View reviewed changes
Comment threaddoc/api/fs.md Outdated
mscdex
mscdex reviewed Feb 4, 2019
View reviewed changes
Comment threaddoc/api/fs.md Outdated
@mscdex

mscdex commented Feb 4, 2019

Copy link
Copy Markdown
Contributor

I don't understand why if a single buffer object is passed, it would be split up in chunks just to use binding.writeBuffers()? That seems inefficient. I would expect the function to throw an error on a non-array or an array containing a non-Buffer-like object.

@AnasAboreeda

AnasAboreeda commented Feb 4, 2019

Copy link
Copy Markdown
ContributorAuthor

@mscdex I am trying to support all types that are supported by fs.write(). Do you think it's a good idea to just support buffer[] as an input?

@mscdex

mscdex commented Feb 4, 2019

Copy link
Copy Markdown
Contributor

Do you think it's a good idea to just support buffer[] as an input?

I don't see a reason to restrict it like that. Any data type supported by the language or node.js that stores binary data should be allowed IMO.

On another note, I think if we are going to support strings, I think it might be better to support them on the C++ side like we do for socket writev to avoid having to possibly cross the JS/C++ boundary for each string. One thing that you might want to benchmark though with that C++ implementation is whether it'd be faster to do the allBuffers check in JS first or just always have the C++ check the types itself (throwing an error when a non-string and non-buffer is found).

@addaleaxaddaleax added semver-minor PRs that contain new features and should be released in the next minor version. and removed tools Issues and PRs related to the tools directory. labels Feb 4, 2019
@addaleax

addaleax commented Feb 4, 2019

Copy link
Copy Markdown
Member

I would suggest supporting ArrayBufferView[] for now; it doesn’t have to be just Buffers, but supporting a single Buffer or strings is probably not the best idea for now, especially if we’d convert them anyway.

bnoordhuis
bnoordhuis reviewed Feb 5, 2019
View reviewed changes
Comment threadlib/fs.js Outdated
Comment threadlib/fs.js Outdated
Comment threadlib/fs.js Outdated
Comment threadlib/fs.js Outdated
Comment threadtest/parallel/test-fs-writev.js Outdated
@AnasAboreeda

AnasAboreeda commented Feb 6, 2019

Copy link
Copy Markdown
ContributorAuthor

@addaleax & @mscdex Now fs.writev() only accepts an array of chunks as an input. Can you please check again?

addaleax
addaleax approved these changes Feb 6, 2019
View reviewed changes

@addaleaxaddaleax left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

Comment threadlib/fs.js Outdated
Comment threadlib/fs.js Outdated
Comment threadtest/parallel/test-fs-writev.js Outdated
Comment threadtools/doc/type-parser.js Outdated
@AnasAboreeda
AnasAboreedaforce-pushed the expose-syscall-writev-to-fs branch from 1519909 to 9a39133CompareFebruary 6, 2019 15:31
jasnell
jasnell reviewed Feb 6, 2019
View reviewed changes
Comment threaddoc/api/fs.md Outdated
jasnell
jasnell reviewed Feb 6, 2019
View reviewed changes
Comment threaddoc/api/fs.md Outdated
jasnell
jasnell reviewed Feb 6, 2019
View reviewed changes
Comment threadlib/fs.js Outdated

@jasnelljasnellFeb 6, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A Promisified version of this should be added to fs.promises also

@mscdexmscdexFeb 6, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parameter name should be buffers to match the comment and the intention.

mscdex
mscdex reviewed Feb 6, 2019
View reviewed changes
Comment threadtest/parallel/test-fs-writev.js Outdated
mscdex
mscdex reviewed Feb 6, 2019
View reviewed changes
Comment threadtest/parallel/test-fs-writev.js Outdated
@AnasAboreeda
AnasAboreedaforce-pushed the expose-syscall-writev-to-fs branch from 2e3f895 to 137dbcbCompareFebruary 6, 2019 20:07
mscdex
mscdex reviewed Feb 7, 2019
View reviewed changes
Comment threaddoc/api/fs.md Outdated
mscdex
mscdex reviewed Feb 7, 2019
View reviewed changes
Comment threaddoc/api/fs.md Outdated
mscdex
mscdex reviewed Feb 7, 2019
View reviewed changes
Comment threaddoc/api/fs.md Outdated
mscdex
mscdex reviewed Feb 7, 2019
View reviewed changes
Comment threadtest/parallel/test-fs-writev.js Outdated
@AnasAboreeda
AnasAboreedaforce-pushed the expose-syscall-writev-to-fs branch from 137dbcb to 35e8aa8CompareFebruary 7, 2019 06:37
mscdex
mscdex reviewed Feb 7, 2019
View reviewed changes
Comment threadtest/parallel/test-fs-writev.js Outdated
mscdex
mscdex reviewed Feb 7, 2019
View reviewed changes
Comment threadtest/parallel/test-fs-writev.js Outdated
@mscdex

mscdex commented Feb 7, 2019•
edited
Loading

Copy link
Copy Markdown
Contributor

I think we should probably also add fs.writevSync() for completeness.

targos pushed a commit that referenced this pull request Aug 19, 2019
@AnasAboreeda@targos
fs: add fs.writev() which exposes syscall writev()
bb19d82
fs with writev allow many buffers to be pushed to underlying OS
APIs in one batch, so this should improve write speed to files.
Refs: #2298
PR-URL: #25925Fixes: #2298
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
@targostargos mentioned this pull request Aug 19, 2019
Merged
targos added a commit that referenced this pull request Aug 19, 2019
@targos
2019-08-20, Version 12.9.0 (Current)
a87f0e1
Notable changes:
* **crypto**:
* Added an oaepHash option to asymmetric encryption which allows users to specify a hash function when using OAEP padding (Tobias Nießen) [#28335](#28335).
* **deps**:
* Updated V8 to 7.6.303.29 (Michaël Zasso) [#28955](#28955).
* Improves the performance of various APIs such as `JSON.parse` and methods
called on frozen arrays.
* Adds the [`Promise.allSettled`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/allSettled) method.
* Improves support of `BigInt` in `Intl` methods.
* For more information: https://v8.dev/blog/v8-release-76
* Updated libuv to 1.31.0 (cjihrig) [#29070](#29070).
* `UV_FS_O_FILEMAP` has been added for faster access to memory mapped files on Windows.
* `uv_fs_mkdir()` now returns `UV_EINVAL` for invalid filenames on Windows. It previously returned `UV_ENOENT`.
* The `uv_fs_statfs()` API has been added.
* The `uv_os_environ()` and `uv_os_free_environ()` APIs have been added.
* **fs**:
* Added `fs.writev` and `fs.writevSync` methods. They allow to write an array of `ArrayBufferView`s to a file descriptor (Anas Aboureada) [#25925](#25925).
* **http**:
* Added three properties to `OutgoingMessage.prototype`: `writableObjectMode`, `writableLength` and `writableHighWaterMark` [#29018](#29018).
* **stream**:
* Added an new property `writableEnded` to writable streams. Its value is set to `true` after `writable.end()` has been called. (Robert Nagy) [#28934](#28934).
PR-URL: #29210
cjihrig added a commit to cjihrig/node that referenced this pull request Aug 19, 2019
@cjihrig
fs: add writev() promises version
3273d0e
nodejs#25925 added fs.writev()
and fs.writevSync(), but did not include a Promises based
equivalent. This commit adds the missing method.
Refs: nodejs#25925
PR-URL: nodejs#29186
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
targos pushed a commit that referenced this pull request Aug 20, 2019
@cjihrig@targos
fs: validate writev fds consistently
4356836
This commit updates the recently added writev methods
to validate file descriptors like the other fs methods do.
PR-URL: #29185
Refs: #25925
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Yongsheng Zhang <zyszys98@gmail.com>
Reviewed-By: Jiawen Geng <technicalcute@gmail.com>
Reviewed-By: Anto Aravinth <anto.aravinth.cse@gmail.com>
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
targos pushed a commit that referenced this pull request Aug 20, 2019
@cjihrig@targos
fs: add writev() promises version
81f3eb5
#25925 added fs.writev()
and fs.writevSync(), but did not include a Promises based
equivalent. This commit adds the missing method.
Refs: #25925
PR-URL: #29186
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
targos added a commit that referenced this pull request Aug 20, 2019
@targos
2019-08-20, Version 12.9.0 (Current)
43d6619
Notable changes:
* crypto:
* Added an oaepHash option to asymmetric encryption which allows
users to specify a hash function when using OAEP padding.
#28335
* deps:
* Updated V8 to 7.6.303.29. #28955
* Improves the performance of various APIs such as `JSON.parse` and
methods called on frozen arrays.
* Adds the Promise.allSettled method.
* Improves support of `BigInt` in `Intl` methods.
* For more information: https://v8.dev/blog/v8-release-76
* Updated libuv to 1.31.0. #29070
* `UV_FS_O_FILEMAP` has been added for faster access to memory
mapped files on Windows.
* `uv_fs_mkdir()` now returns `UV_EINVAL` for invalid filenames on
Windows. It previously returned `UV_ENOENT`.
* The `uv_fs_statfs()` API has been added.
* The `uv_os_environ()` and `uv_os_free_environ()` APIs have been
added.
* fs:
* Added `fs.writev`, `fs.writevSync` and `filehandle.writev` (promise
version) methods. They allow to write an array of `ArrayBufferView`s
to a file descriptor. #25925https://github.com/nodejs/node/pull/29186/files
* http:
* Added three properties to `OutgoingMessage.prototype`:
`writableObjectMode`, `writableLength` and `writableHighWaterMark`
#29018
* stream:
* Added an new property `readableEnded` to readable streams. Its value
is set to `true` when the `'end'` event is emitted.
#28814
* Added an new property `writableEnded` to writable streams. Its value
is set to `true` after `writable.end()` has been called.
#28934
PR-URL: #29210
targos added a commit to targos/node that referenced this pull request Aug 20, 2019
@targos
2019-08-20, Version 12.9.0 (Current)
a9adfdb
Notable changes:
* crypto:
* Added an oaepHash option to asymmetric encryption which allows
users to specify a hash function when using OAEP padding.
nodejs#28335
* deps:
* Updated V8 to 7.6.303.29. nodejs#28955
* Improves the performance of various APIs such as `JSON.parse` and
methods called on frozen arrays.
* Adds the Promise.allSettled method.
* Improves support of `BigInt` in `Intl` methods.
* For more information: https://v8.dev/blog/v8-release-76
* Updated libuv to 1.31.0. nodejs#29070
* `UV_FS_O_FILEMAP` has been added for faster access to memory
mapped files on Windows.
* `uv_fs_mkdir()` now returns `UV_EINVAL` for invalid filenames on
Windows. It previously returned `UV_ENOENT`.
* The `uv_fs_statfs()` API has been added.
* The `uv_os_environ()` and `uv_os_free_environ()` APIs have been
added.
* fs:
* Added `fs.writev`, `fs.writevSync` and `filehandle.writev` (promise
version) methods. They allow to write an array of `ArrayBufferView`s
to a file descriptor. nodejs#25925https://github.com/nodejs/node/pull/29186/files
* http:
* Added three properties to `OutgoingMessage.prototype`:
`writableObjectMode`, `writableLength` and `writableHighWaterMark`
nodejs#29018
* stream:
* Added an new property `readableEnded` to readable streams. Its value
is set to `true` when the `'end'` event is emitted.
nodejs#28814
* Added an new property `writableEnded` to writable streams. Its value
is set to `true` after `writable.end()` has been called.
nodejs#28934
PR-URL: nodejs#29210
targos added a commit that referenced this pull request Aug 20, 2019
@targos
2019-08-20, Version 12.9.0 (Current)
e6872f5
Notable changes:
* crypto:
* Added an oaepHash option to asymmetric encryption which allows
users to specify a hash function when using OAEP padding.
#28335
* deps:
* Updated V8 to 7.6.303.29. #28955
* Improves the performance of various APIs such as `JSON.parse` and
methods called on frozen arrays.
* Adds the Promise.allSettled method.
* Improves support of `BigInt` in `Intl` methods.
* For more information: https://v8.dev/blog/v8-release-76
* Updated libuv to 1.31.0. #29070
* `UV_FS_O_FILEMAP` has been added for faster access to memory
mapped files on Windows.
* `uv_fs_mkdir()` now returns `UV_EINVAL` for invalid filenames on
Windows. It previously returned `UV_ENOENT`.
* The `uv_fs_statfs()` API has been added.
* The `uv_os_environ()` and `uv_os_free_environ()` APIs have been
added.
* fs:
* Added `fs.writev`, `fs.writevSync` and `filehandle.writev` (promise
version) methods. They allow to write an array of `ArrayBufferView`s
to a file descriptor. #25925#29186
* http:
* Added three properties to `OutgoingMessage.prototype`:
`writableObjectMode`, `writableLength` and `writableHighWaterMark`
#29018
* stream:
* Added an new property `readableEnded` to readable streams. Its value
is set to `true` when the `'end'` event is emitted.
#28814
* Added an new property `writableEnded` to writable streams. Its value
is set to `true` after `writable.end()` has been called.
#28934
PR-URL: #29210
targos added a commit that referenced this pull request Aug 20, 2019
@targos
2019-08-20, Version 12.9.0 (Current)
edc83a9
Notable changes:
* crypto:
* Added an oaepHash option to asymmetric encryption which allows
users to specify a hash function when using OAEP padding.
#28335
* deps:
* Updated V8 to 7.6.303.29. #28955
* Improves the performance of various APIs such as `JSON.parse` and
methods called on frozen arrays.
* Adds the Promise.allSettled method.
* Improves support of `BigInt` in `Intl` methods.
* For more information: https://v8.dev/blog/v8-release-76
* Updated libuv to 1.31.0. #29070
* `UV_FS_O_FILEMAP` has been added for faster access to memory
mapped files on Windows.
* `uv_fs_mkdir()` now returns `UV_EINVAL` for invalid filenames on
Windows. It previously returned `UV_ENOENT`.
* The `uv_fs_statfs()` API has been added.
* The `uv_os_environ()` and `uv_os_free_environ()` APIs have been
added.
* fs:
* Added `fs.writev`, `fs.writevSync` and `filehandle.writev` (promise
version) methods. They allow to write an array of `ArrayBufferView`s
to a file descriptor. #25925#29186
* http:
* Added three properties to `OutgoingMessage.prototype`:
`writableObjectMode`, `writableLength` and `writableHighWaterMark`
#29018
* stream:
* Added an new property `readableEnded` to readable streams. Its value
is set to `true` when the `'end'` event is emitted.
#28814
* Added an new property `writableEnded` to writable streams. Its value
is set to `true` after `writable.end()` has been called.
#28934
PR-URL: #29210
This was referenced Aug 20, 2019
Closed
Closed
Closed
Closed
Closed
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bnoordhuisbnoordhuisbnoordhuis left review comments
@targostargostargos left review comments
@jasnelljasnelljasnell approved these changes
@TrottTrottTrott approved these changes
@addaleaxaddaleaxaddaleax approved these changes
+4 more reviewers
@mscdexmscdexmscdex left review comments
@thefourtheyethefourtheyethefourtheye left review comments
@reqsharkreqsharkreqshark left review comments
@vsemozhetbytvsemozhetbytvsemozhetbyt left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

author readyPRs that have at least one approval, no pending requests for changes, and a CI started.docIssues and PRs related to the documentations.fsIssues and PRs related to the fs subsystem / file system.semver-minorPRs that contain new features and should be released in the next minor version.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

fs: expose scatter/gather syscalls writev() and readv()

12 participants

@AnasAboreeda@mscdex@addaleax@reqshark@joe9111@nodejs-github-bot@Trott@bnoordhuis@jasnell@thefourtheye@targos@vsemozhetbyt