Skip to content

fs: add bufferSize option to fs.opendir() - #30114

Closed
addaleax wants to merge 7 commits into
nodejs:masterfrom
addaleax:fs-opendir-buffersize
Closed

fs: add bufferSize option to fs.opendir()#30114
addaleax wants to merge 7 commits into
nodejs:masterfrom
addaleax:fs-opendir-buffersize

Conversation

@addaleax

@addaleaxaddaleax commented Oct 25, 2019

Copy link
Copy Markdown
Member

Add an option that controls the size of the internal
buffer.

Fixes: #29941

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines

@addaleax
fs: add bufferSize option to fs.opendir()
ac9b36b
Add an option that controls the size of the internal
buffer.
Fixes: nodejs#29941
@nodejs-github-botnodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. fs Issues and PRs related to the fs subsystem / file system. labels Oct 25, 2019
@duffleit

duffleit commented Oct 25, 2019

Copy link
Copy Markdown

What a mob programming session 🙌🏻😀

@jdjuan

jdjuan commented Oct 25, 2019

Copy link
Copy Markdown

300 people watched the process of this PR during @scriptconf Great job!!!!!

richardlau
richardlau reviewed Oct 25, 2019
View reviewed changes
Comment threaddoc/api/fs.md Outdated
Comment threaddoc/api/fs.md Outdated
Comment threaddoc/api/fs.md Outdated
Comment threaddoc/api/fs.md Outdated
Comment threaddoc/api/fs.md Outdated
Comment threaddoc/api/fs.md Outdated
@drdreo

drdreo commented Oct 25, 2019

Copy link
Copy Markdown

Can approve that all tests were done

@nodejs-github-bot

nodejs-github-bot commented Oct 25, 2019

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/26194/

@Trott

Trott commented Oct 25, 2019

Copy link
Copy Markdown
Member

bufferSize will need to be added to test/benchmark/test-benchmark-fs.js

Speaking of which: Needs tests. (Just forgot to git add them maybe? You checked the box for tests and I know it's not like you to not have tests. :-D )

cjihrig
cjihrig reviewed Oct 25, 2019
View reviewed changes
Comment threaddoc/api/fs.md Outdated
@addaleax

addaleax commented Oct 25, 2019

Copy link
Copy Markdown
MemberAuthor

bufferSize will need to be added to test/benchmark/test-benchmark-fs.js

Done, thanks!

Speaking of which: Needs tests. (Just forgot to git add them maybe? You checked the box for tests and I know it's not like you to not have tests. :-D )

I checked the box because I added benchmarks that measure the performance impact – I don’t think this is something that we could test for well without digging into internals.

@addaleaxaddaleax added the semver-minor PRs that contain new features and should be released in the next minor version. label Oct 25, 2019
@richardlau

richardlau commented Oct 25, 2019

Copy link
Copy Markdown
Member

bufferSize will need to be added to test/benchmark/test-benchmark-fs.js

Done, thanks!

Speaking of which: Needs tests. (Just forgot to git add them maybe? You checked the box for tests and I know it's not like you to not have tests. :-D )

I checked the box because I added benchmarks that measure the performance impact – I don’t think this is something that we could test for well without digging into internals.

Test passing non-numbers/negative numbers/zero/non-integers as bufferSize?

@Trott

Trott commented Oct 25, 2019•
edited
Loading

Copy link
Copy Markdown
Member

Test passing non-numbers/negative numbers/zero/non-integers as bufferSize?

And maybe one test for 32 and one other reasonable positive integer to confirm that it doesn't throw in those cases?

@addaleax

addaleax commented Oct 25, 2019

Copy link
Copy Markdown
MemberAuthor

@Trott@richardlau I guess, yes – added tests for the error condition/that it works when using a positive integer value.

richardlau
richardlau approved these changes Oct 25, 2019
View reviewed changes
Comment threadtest/parallel/test-fs-opendir.js Outdated
@addaleax@richardlau
fixup! fs: add bufferSize option to fs.opendir()
2119b4c
Co-Authored-By: Richard Lau <riclau@uk.ibm.com>
@addaleaxaddaleax added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Oct 26, 2019
@nodejs-github-bot

nodejs-github-bot commented Oct 26, 2019

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/26206/

cjihrig
cjihrig approved these changes Oct 26, 2019
View reviewed changes
Comment threadlib/internal/fs/dir.js Outdated
@nodejs-github-bot

nodejs-github-bot commented Oct 26, 2019

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/26218/

@nodejs-github-bot

nodejs-github-bot commented Oct 28, 2019

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/26232/

@nodejs-github-bot

nodejs-github-bot commented Oct 28, 2019

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/26234/

@addaleax

addaleax commented Oct 28, 2019

Copy link
Copy Markdown
MemberAuthor

Landed in b35181f 🎉

addaleax added a commit that referenced this pull request Oct 28, 2019
@addaleax
fs: add bufferSize option to fs.opendir()
b35181f
Add an option that controls the size of the internal
buffer.
Fixes: #29941
PR-URL: #30114
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
@addaleax
addaleax deleted the fs-opendir-buffersize branch October 28, 2019 13:35
This was referenced Oct 31, 2019
Closed
Closed
targos pushed a commit that referenced this pull request Nov 5, 2019
@addaleax@targos
fs: add bufferSize option to fs.opendir()
07b5584
Add an option that controls the size of the internal
buffer.
Fixes: #29941
PR-URL: #30114
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
targos added a commit that referenced this pull request Nov 5, 2019
@targos
2019-11-05, Version 13.1.0 (Current)
64eacd8
Notable changes:
* cli:
* Added a new flag (`--trace-uncaught`) that makes Node.js print the
stack trace at the time of throwing uncaught exceptions, rather than
at the creation of the `Error` object, if there is any. This is
disabled by default because it affects GC behavior.
#30025
* crypto
* Added `Hash.prototype.copy()` method. It returns a new `Hash` object
with its internal state cloned from the original one.
#29910
* dgram
* Added source-specific multicast support. This adds methods to
Datagram sockets to support RFC 4607
(https://tools.ietf.org/html/rfc4607) for IPv4 and IPv6.
#15735
* fs
* Added a `bufferSize` option to `fs.opendir()`. It allows to control
the number of entries that are buffered internally when reading from
the directory. #30114
* meta
* Added Chengzhong Wu (https://github.com/legendecas) to
collaborators. #30115
PR-URL: #30262
@targostargos mentioned this pull request Nov 5, 2019
Merged
targos added a commit that referenced this pull request Nov 5, 2019
@targos
2019-11-05, Version 13.1.0 (Current)
309e1ea
Notable changes:
* cli:
* Added a new flag (`--trace-uncaught`) that makes Node.js print the
stack trace at the time of throwing uncaught exceptions, rather than
at the creation of the `Error` object, if there is any. This is
disabled by default because it affects GC behavior.
#30025
* crypto
* Added `Hash.prototype.copy()` method. It returns a new `Hash` object
with its internal state cloned from the original one.
#29910
* dgram
* Added source-specific multicast support. This adds methods to
Datagram sockets to support RFC 4607
(https://tools.ietf.org/html/rfc4607) for IPv4 and IPv6.
#15735
* fs
* Added a `bufferSize` option to `fs.opendir()`. It allows to control
the number of entries that are buffered internally when reading from
the directory. #30114
* meta
* Added Chengzhong Wu (https://github.com/legendecas) to
collaborators. #30115
PR-URL: #30262
MylesBorins pushed a commit that referenced this pull request Jan 12, 2020
@addaleax@MylesBorins
fs: add bufferSize option to fs.opendir()
ba5683c
Add an option that controls the size of the internal
buffer.
Fixes: #29941
PR-URL: #30114
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
@targostargos mentioned this pull request Jan 15, 2020
Closed
BethGriggs pushed a commit that referenced this pull request Feb 6, 2020
@addaleax@BethGriggs
fs: add bufferSize option to fs.opendir()
785aa86
Add an option that controls the size of the internal
buffer.
Fixes: #29941
PR-URL: #30114
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
@MylesBorinsMylesBorins mentioned this pull request Feb 8, 2020
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jasnelljasnelljasnell approved these changes
@TrottTrottTrott approved these changes
@cjihrigcjihrigcjihrig approved these changes
@richardlaurichardlaurichardlau approved these changes
@Fishrock123Fishrock123Awaiting requested review from Fishrock123

Assignees

No one assigned

Labels

author readyPRs that have at least one approval, no pending requests for changes, and a CI started.c++Issues and PRs that require attention from people who are familiar with C++.fsIssues and PRs related to the fs subsystem / file system.semver-minorPRs that contain new features and should be released in the next minor version.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

fs.Dir.read() is very slow

10 participants

@addaleax@duffleit@jdjuan@drdreo@nodejs-github-bot@Trott@richardlau@jasnell@cjihrig@z1c0