fs: improve performance for sync stat() functions by mscdex · Pull Request #11522 · nodejs/node

mscdex · 2017-02-23T18:07:15Z

Results from benchmarks:

 improvement confidence p.value
fs/bench-statSync.js kind="fstatSync" n=1000000 13.51 % *** 4.033815e-18
fs/bench-statSync.js kind="statSync" n=1000000 7.09 % *** 6.502451e-15
fs/bench-statSync.js kind="lstatSync" n=1000000 7.43 % *** 9.979362e-16

CI: https://ci.nodejs.org/job/node-test-pull-request/6568/

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

Affected core subsystem(s)

fs

bnoordhuis

LGTM. Is there value in keeping the sync non-float64array code paths in the binding methods?

mscdex · 2017-02-23T18:44:30Z

@bnoordhuis I left them there for backwards compatibility for now, even though third-party use of process.binding() is frowned upon/not supported. *shrug*

bnoordhuis · 2017-02-23T19:51:55Z

I left them there for backwards compatibility for now, even though third-party use of process.binding() is frowned upon/not supported.

I figured that was the reason but I'd worry that people construe that as a nod of approval that it's okay to use the bindings layer directly.

mscdex · 2017-02-23T21:06:22Z

@bnoordhuis I don't mind either way. If most @nodejs/collaborators agree to remove support for the old method, then I will remove it.

cjihrig · 2017-02-23T23:56:59Z

Indentation looks off here.

jasnell · 2017-02-24T00:15:16Z

The recently added deprecation policy makes it fairly clear that process.binding() is not covered under our API guarantees. I know that it's a recent change but I believe it gives sufficient coverage on this.

mscdex · 2017-02-24T02:27:31Z

I've removed the non-Float64Array sync() branches.

mscdex · 2017-02-24T02:28:29Z

CI: https://ci.nodejs.org/job/node-test-pull-request/6573/

mcollina

CI does not seem green.
LGTM when it does.

bnoordhuis · 2017-02-24T08:43:30Z

fs.exists() and fs.existsSync() use binding.stat() directly, they'll need to be updated.

mscdex · 2017-02-24T09:10:15Z

Fixed.

CI again: https://ci.nodejs.org/job/node-test-pull-request/6576/

PR-URL: nodejs#11522 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

Node 7.7 changed the behavior of the `binding.{stat,lstat,fstat}` functions (see nodejs/node#11522). This updates the binding functions used in mock-fs to match the new Node binding behavior, while still maintaining compatibility with old Node versions. The new behavior is detected when the second argument to `binding.{stat,lstat,fstat}` is a `Float64Array`, which would be an invalid argument for previous versions of the binding.

…haub#197) Node 7.7 changed the behavior of the `binding.{stat,lstat,fstat}` functions (see nodejs/node#11522). This updates the binding functions used in mock-fs to match the new Node binding behavior, while still maintaining compatibility with old Node versions. The new behavior is detected when the second argument to `binding.{stat,lstat,fstat}` is a `Float64Array`, which would be an invalid argument for previous versions of the binding.

jasnell · 2017-03-07T04:57:05Z

@nodejs/lts @mscdex ... should we pull this into v6.x-staging?

mscdex · 2017-03-07T05:06:33Z

@jasnell Fine by me, it should still provide a speedup even though I haven't explicitly benchmarked v6.x.

Why the dont-land-on-v4.x?

jasnell · 2017-03-07T05:09:07Z

like the other one, the don't land is largely precautionary

PR-URL: #11522 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com>

Notable changes * performance: The performance of several APIs has been improved. - `Buffer.compare()` is up to 35% faster on average. (Brian White) #10927 - `buffer.toJSON()` is up to 2859% faster on average. (Brian White) #10895 - `fs.*statSync()` functions are now up to 9.3% faster on average. (Brian White) #11522 - `os.loadavg` is up to 151% faster. (Brian White) #11516 - `process.memoryUsage()` is up to 34% faster. (Brian White) #11497 - `querystring.unescape()` for `Buffer`s is 15% faster on average. (Brian White) #10837 - `querystring.stringify()` is up to 7.8% faster on average. (Brian White) #10852 - `querystring.parse()` is up to 21% faster on average. (Brian White) #10874 * IPC: - Batched writes have been enabled for process IPC on platforms that support Unix Domain Sockets. (Alexey Orlenko) #10677 - Performance gains may be up to 40% for some workloads. * child_process: - `spawnSync` now returns a null `status` when child is terminated by a signal. (cjihrig) #11288 - This fixes the behavior to act like `spawn()` does. * http: - Control characters are now always rejected when using `http.request()`. (Ben Noordhuis) #8923 - Debug messages have been added for cases when headers contain invalid values. (Evan Lucas) #9195 * node: - Heap statistics now support values larger than 4GB. (Ben Noordhuis) #10186 * timers: - Timer callbacks now always maintain order when interacting with domain error handling. (John Barboza) #10522 PR-URL: #11759

Notable changes * performance: The performance of several APIs has been improved. - `Buffer.compare()` is up to 35% faster on average. (Brian White) nodejs/node#10927 - `buffer.toJSON()` is up to 2859% faster on average. (Brian White) nodejs/node#10895 - `fs.*statSync()` functions are now up to 9.3% faster on average. (Brian White) nodejs/node#11522 - `os.loadavg` is up to 151% faster. (Brian White) nodejs/node#11516 - `process.memoryUsage()` is up to 34% faster. (Brian White) nodejs/node#11497 - `querystring.unescape()` for `Buffer`s is 15% faster on average. (Brian White) nodejs/node#10837 - `querystring.stringify()` is up to 7.8% faster on average. (Brian White) nodejs/node#10852 - `querystring.parse()` is up to 21% faster on average. (Brian White) nodejs/node#10874 * IPC: - Batched writes have been enabled for process IPC on platforms that support Unix Domain Sockets. (Alexey Orlenko) nodejs/node#10677 - Performance gains may be up to 40% for some workloads. * child_process: - `spawnSync` now returns a null `status` when child is terminated by a signal. (cjihrig) nodejs/node#11288 - This fixes the behavior to act like `spawn()` does. * http: - Control characters are now always rejected when using `http.request()`. (Ben Noordhuis) nodejs/node#8923 - Debug messages have been added for cases when headers contain invalid values. (Evan Lucas) nodejs/node#9195 * node: - Heap statistics now support values larger than 4GB. (Ben Noordhuis) nodejs/node#10186 * timers: - Timer callbacks now always maintain order when interacting with domain error handling. (John Barboza) nodejs/node#10522 PR-URL: nodejs/node#11759 Signed-off-by: Ilkka Myller <ilkka.myller@nodefield.com>

…haub#197) Node 7.7 changed the behavior of the `binding.{stat,lstat,fstat}` functions (see nodejs/node#11522). This updates the binding functions used in mock-fs to match the new Node binding behavior, while still maintaining compatibility with old Node versions. The new behavior is detected when the second argument to `binding.{stat,lstat,fstat}` is a `Float64Array`, which would be an invalid argument for previous versions of the binding.

mscdex added c++ Issues and PRs that require attention from people who are familiar with C++. fs Issues and PRs related to the fs subsystem / file system. performance Issues and PRs related to the performance of Node.js. labels Feb 23, 2017

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. fs Issues and PRs related to the fs subsystem / file system. labels Feb 23, 2017

mscdex force-pushed the fs-statsync-perf branch from 92a8350 to 99f37aaCompare February 23, 2017 18:11

bnoordhuis approved these changes Feb 23, 2017
View reviewed changes

jasnell approved these changes Feb 23, 2017
View reviewed changes

addaleax approved these changes Feb 23, 2017
View reviewed changes

cjihrig reviewed Feb 23, 2017
View reviewed changes

mscdex force-pushed the fs-statsync-perf branch 2 times, most recently from ef8da93 to 0f3da51Compare February 24, 2017 02:26

mcollina approved these changes Feb 24, 2017
View reviewed changes

mscdex force-pushed the fs-statsync-perf branch from 0f3da51 to 08db001Compare February 24, 2017 09:09

bnoordhuis approved these changes Feb 24, 2017
View reviewed changes

cjihrig approved these changes Feb 24, 2017
View reviewed changes

mscdex force-pushed the fs-statsync-perf branch from 08db001 to 22abb39Compare February 26, 2017 02:01

mscdex merged commit 22abb39 into nodejs:masterFeb 26, 2017

mscdex deleted the fs-statsync-perf branch February 26, 2017 02:05

italoacasas mentioned this pull request Feb 27, 2017
7.7.0 Proposal #11553
Merged

not-an-aardvark mentioned this pull request Mar 2, 2017
Calling fs.statSync with an active mock causes an error in Node 7.7.1 tschaub/mock-fs#197
Closed

not-an-aardvark mentioned this pull request Mar 2, 2017
Update fs.stat(), fs.lstat(), and fs.fstat() for Node 7.7+ (fixes #197) tschaub/mock-fs#198
Merged

mscdex mentioned this pull request Mar 3, 2017
fs: include more fs.stat*() optimizations #11665
Closed
3 tasks

jasnell added the dont-land-on-v4.x label Mar 7, 2017

MylesBorins mentioned this pull request Mar 9, 2017
v6.10.1 proposal #11759
Merged

mscdex mentioned this pull request Apr 15, 2017
fs.stat and fs.statSync may not return identical struct #12419
Closed

JLHwung mentioned this pull request Nov 3, 2017
backport: 11665 to v6 #16667
Closed
4 tasks

Uh oh!

Uh oh!

Conversation

mscdex commented Feb 23, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Affected core subsystem(s)

Uh oh!

bnoordhuis left a comment

Choose a reason for hiding this comment

Uh oh!

mscdex commented Feb 23, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bnoordhuis commented Feb 23, 2017

Uh oh!

mscdex commented Feb 23, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cjihrigFeb 23, 2017

Choose a reason for hiding this comment

Uh oh!

mscdexFeb 24, 2017

Choose a reason for hiding this comment

Uh oh!

jasnell commented Feb 24, 2017

Uh oh!

mscdex commented Feb 24, 2017

Uh oh!

mscdex commented Feb 24, 2017

Uh oh!

mcollina left a comment

Choose a reason for hiding this comment

Uh oh!

bnoordhuis commented Feb 24, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mscdex commented Feb 24, 2017

Uh oh!

jasnell commented Mar 7, 2017

Uh oh!

mscdex commented Mar 7, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jasnell commented Mar 7, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

mscdex commented Feb 23, 2017•
edited
Loading

mscdex commented Feb 23, 2017•
edited
Loading

mscdex commented Feb 23, 2017•
edited
Loading

bnoordhuis commented Feb 24, 2017•
edited
Loading

mscdex commented Mar 7, 2017•
edited
Loading