fs: optimize `fs.cpSync` js calls by anonrig · Pull Request #53614 · nodejs/node

anonrig · 2024-06-27T18:38:31Z

#53612 is required to do a benchmark

A different approach to #53541

cc @nodejs/performance @nodejs/cpp-reviewers @billywhizz @jasnell @nodejs/fs

Benchmark CI: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1573/

Small win on happy path, and probably high win on error path:

 confidence improvement accuracy (*) (**) (***)
fs/bench-cpSync.js n=1 *** 21.96 % ±6.85% ±9.14% ±11.97%
fs/bench-cpSync.js n=100 *** 14.91 % ±2.86% ±3.81% ±4.97%
fs/bench-cpSync.js n=10000 *** 9.74 % ±1.28% ±1.71% ±2.23%

anonrig · 2024-06-27T21:32:11Z

cc @nodejs/tsc @nodejs/fs appreciate your input on which path to take.

previous pr:

is a sem-ver major
introduces 2 different implementations (js only and cpp)
had limitations due to std::filesystem::copy

this pr:

is not a sem-ver major
optimizes all flags passed to cpSync
optimizes error cases as same as the previous pr
has a lot more optimization to do

mcollina

lgtm

nodejs-github-bot · 2024-07-08T18:29:44Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60175/

nodejs-github-bot · 2024-07-09T06:01:34Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60191/

nodejs-github-bot · 2024-07-09T06:01:35Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60190/

nodejs-github-bot · 2024-07-11T19:04:07Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60257/

nodejs-github-bot · 2024-07-12T10:48:52Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60274/

nodejs-github-bot · 2024-07-12T15:29:49Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60277/

nodejs-github-bot · 2024-07-16T19:58:32Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60366/

aduh95 · 2024-07-16T21:58:04Z

Windows compilation is failing:

C:\workspace\node-compile-windows-debug\node\deps\v8\src\base\bits.h(477,31): warning C4146: unary minus operator applied to unsigned type, result still unsigned [C:\workspace\node-compile-windows-debug\node\tools\v8_gypfiles\mksnapshot.vcxproj]
embedded-file-writer.cc
mksnapshot.cc
embedded-empty.cc
platform-embedded-file-writer-base.cc
platform-embedded-file-writer-generic.cc
platform-embedded-file-writer-win.cc
platform-embedded-file-writer-aix.cc
platform-embedded-file-writer-mac.cc
snapshot-empty.cc
static-roots-gen.cc
Creating library ..\..\out\Debug\mksnapshot.lib and object ..\..\out\Debug\mksnapshot.exp
mksnapshot.vcxproj -> ..\..\out\Debug\\mksnapshot.exe
generating: "..\..\out\Debug\obj\v8_snapshot\/snapshot.cc" "..\..\out\Debug\obj\v8_snapshot\/embedded.S"
Assembling ..\..\out\Debug\obj\v8_snapshot\\embedded.S...
setup-isolate-deserialize.cc
C:\workspace\node-compile-windows-debug\node\deps\v8\src\base\bits.h(477,31): warning C4146: unary minus operator applied to unsigned type, result still unsigned [C:\workspace\node-compile-windows-debug\node\tools\v8_gypfiles\v8_snapshot.vcxproj]
snapshot.cc
C:\workspace\node-compile-windows-debug\node\deps\v8\src\base\bits.h(477,31): warning C4146: unary minus operator applied to unsigned type, result still unsigned [C:\workspace\node-compile-windows-debug\node\tools\v8_gypfiles\v8_snapshot.vcxproj]
v8_snapshot.vcxproj -> ..\..\out\Debug\lib\v8_snapshot.lib

anonrig · 2024-07-17T00:35:24Z

@nodejs/platform-windows any idea why this build is failing?

H4ad · 2024-07-17T01:43:14Z

Building locally, I got this:

C:\Users\vinic\Desktop\Projects\node\src\node_file.cc(3036,15): error C2664: 'void node::Environment::ThrowUVException(int,const char *,const char *,const char *,const char *)': cannot convert argument 4 from
'const std::filesystem::path::value_type *' to 'const char *' [C:\Users\vinic\Desktop\Projects\node\libnode.vcxproj]

StefanStojanovic · 2024-07-17T06:31:13Z

Building locally, I got this:

C:\Users\vinic\Desktop\Projects\node\src\node_file.cc(3036,15): error C2664: 'void node::Environment::ThrowUVException(int,const char *,const char *,const char *,const char *)': cannot convert argument 4 from
'const std::filesystem::path::value_type *' to 'const char *' [C:\Users\vinic\Desktop\Projects\node\libnode.vcxproj]

This is also in the log from the CI. Based on the error message, this looks very similar to what we had in #53600 because std::filesystem::path::value_type is different on POSIX and Windows.

anonrig · 2024-07-21T21:19:19Z

This is also in the log from the CI. Based on the error message, this looks very similar to what we had in #53600 because std::filesystem::path::value_type is different on POSIX and Windows.

It seems I forgot a src_path.c_str() there. Thank you! @StefanStojanovic @H4ad

nodejs-github-bot · 2024-07-21T23:41:28Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60519/

nodejs-github-bot · 2024-07-22T06:15:53Z

CI: https://ci.nodejs.org/job/node-test-pull-request/60528/

jasnell

Still LGTM

nodejs-github-bot · 2024-07-22T16:00:41Z

Landed in 88027e8

PR-URL: #53614 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

joyeecheung · 2024-08-22T09:10:08Z

+ ? std::filesystem::symlink_status(src_path, error_code)
+ : std::filesystem::status(src_path, error_code);
+ if (error_code) {
+ return env->ThrowUVException(EEXIST, "lstat", nullptr, src.out());


This looks wrong, the error code should be properly translated instead of being directly mapped to EEXIST, there are many other reasons why this could fail.

joyeecheung · 2024-08-22T09:20:52Z

+ ToNamespacedPath(env, &src);
+ THROW_IF_INSUFFICIENT_PERMISSIONS(
+ env, permission::PermissionScope::kFileSystemRead, src.ToStringView());
+ auto src_path = std::filesystem::path(src.ToStringView());


BufferValue converts the string to UTF8, on Windows this requires automatic conversion for char8_t to work which may not be supported by the version of MSVC we use, I think that may be the source of bugs like #54476

Actually no, this is simply broken on Windows, because this just gets returned in char. It needs to be returned as std::u8string/std::u8string_view instead.

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Jun 27, 2024

anonrig mentioned this pull request Jun 27, 2024
fs: improve cpSync performance #53541
Closed

anonrig force-pushed the optimize-cp-sync branch from 02e0883 to d7d4ae8Compare June 27, 2024 18:44

anonrig requested a review from lemire June 27, 2024 18:44

anonrig force-pushed the optimize-cp-sync branch 2 times, most recently from bebf131 to 2d5fb50Compare June 27, 2024 21:23

anonrig force-pushed the optimize-cp-sync branch from 2d5fb50 to 6a216bdCompare June 27, 2024 21:33

This comment was marked as outdated.
Sign in to view

mcollina approved these changes Jun 28, 2024
View reviewed changes

benjamingr approved these changes Jun 28, 2024
View reviewed changes

anonrig force-pushed the optimize-cp-sync branch from 6a216bd to 371d7a7Compare July 11, 2024 18:54

anonrig force-pushed the optimize-cp-sync branch from 371d7a7 to e2c27aeCompare July 21, 2024 21:16

anonrig requested a review from jasnell July 21, 2024 21:17

This comment was marked as outdated.
Sign in to view

jasnell approved these changes Jul 21, 2024
View reviewed changes

anonrig added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Jul 21, 2024

fs: optimize fs.cpSync js calls
a590e84

anonrig force-pushed the optimize-cp-sync branch from e2c27ae to a590e84Compare July 21, 2024 23:40

jasnell approved these changes Jul 22, 2024
View reviewed changes

anonrig added the commit-queue Add this label to land a pull request using GitHub Actions. label Jul 22, 2024

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Jul 22, 2024

nodejs-github-bot merged commit 88027e8 into nodejs:mainJul 22, 2024

targos pushed a commit that referenced this pull request Jul 30, 2024
fs: optimize fs.cpSync js calls
1b0d5d4
PR-URL: #53614 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

RafaelGSS mentioned this pull request Jul 30, 2024
v22.6.0 proposal #54123
Merged

RafaelGSS pushed a commit that referenced this pull request Aug 5, 2024
fs: optimize fs.cpSync js calls
5e03c17
PR-URL: #53614 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com>

joyeecheung reviewed Aug 22, 2024
View reviewed changes

joyeecheung mentioned this pull request Aug 22, 2024
v22.6.0 fs.cpSync crash #54476
Closed

markandrus mentioned this pull request Sep 11, 2024
[Bug?]: fsBinding.cpSyncCheckPaths error in Node 22 yarnpkg/berry#6488
Open
1 task

jazelly mentioned this pull request Sep 12, 2024
fs: translate error code properly in cpSync #54906
Merged

joyeecheung mentioned this pull request Sep 20, 2024
src: revert filesystem::path changes #55015
Merged

targos added the dont-land-on-v20.x label Sep 21, 2024

Uh oh!

Uh oh!

Conversation

anonrig commented Jun 27, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anonrig commented Jun 27, 2024

Uh oh!

This comment was marked as outdated.

mcollina left a comment

Choose a reason for hiding this comment

Uh oh!

nodejs-github-bot commented Jul 8, 2024

Uh oh!

nodejs-github-bot commented Jul 9, 2024

Uh oh!

nodejs-github-bot commented Jul 9, 2024

Uh oh!

nodejs-github-bot commented Jul 11, 2024

Uh oh!

nodejs-github-bot commented Jul 12, 2024

Uh oh!

nodejs-github-bot commented Jul 12, 2024

Uh oh!

nodejs-github-bot commented Jul 16, 2024

Uh oh!

aduh95 commented Jul 16, 2024

Uh oh!

anonrig commented Jul 17, 2024

Uh oh!

H4ad commented Jul 17, 2024

Uh oh!

StefanStojanovic commented Jul 17, 2024

Uh oh!

This comment was marked as outdated.

anonrig commented Jul 21, 2024

Uh oh!

nodejs-github-bot commented Jul 21, 2024

Uh oh!

nodejs-github-bot commented Jul 22, 2024

Uh oh!

jasnell left a comment

Choose a reason for hiding this comment

Uh oh!

nodejs-github-bot commented Jul 22, 2024

Uh oh!

joyeecheungAug 22, 2024

Choose a reason for hiding this comment

Uh oh!

joyeecheungAug 22, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joyeecheungAug 22, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

anonrig commented Jun 27, 2024•
edited
Loading

joyeecheungAug 22, 2024•
edited
Loading

joyeecheungAug 22, 2024•
edited
Loading