build: speedup compilation of mksnapshot output by kvakil · Pull Request #48162 · nodejs/node

kvakil · 2023-05-25T02:45:02Z

Incremental compilation of Node.js is slow. Currently on a powerful
Linux machine, it takes about 5.8 seconds to compile
gen/node_snapshot.cc with g++.

As in the previous PR which dealt with node_js2c, we add a new build
define NODE_MKSNAPSHOT_USE_STRING_LITERALS which is used by
node_mksnapshot. When this flag is set, we emit string literals
instead of array literals for the snapshot blob and for the code cache,
i.e.:

// old: static const uint8_t X[] = { ... };staticconstuint8_t *X = "...";

I only enabled the new flag on Linux/macOS, since those are systems that
I have available for testing. On my Linux system with gcc, it speeds up
compilation of this file by 3.7s (5.8s -> 2.1s). On my Mac system with
clang, it speeds up compilation by 1.7s (3.4s -> 1.7s).

Again, the right thing here is probably to generate separate files for
the snapshot blob and for each code cache output, but this is a nice
intermediate speedup.

Refs: #47984
Refs: #48160

nodejs-github-bot · 2023-05-25T02:45:06Z

Review requested:

@nodejs/gyp
@nodejs/startup

bnoordhuis · 2023-05-25T08:19:47Z

If we're doing toolchain-specific speed hacks: a trick I've used in other projects is to emit the .o directly. Much (much!) faster than first generating, then parsing a huge C source file.

I only had to care about x86_64 Linux though. For node it'd be a bit more complex.

aduh95 · 2023-05-25T14:47:58Z

This makes the test-linux GHA ends 30 minutes earlier than on other PRs, which is impressive and seems to be worth it by itself even if it had no effect on incremental compilation!

Incremental compilation of Node.js is slow. Currently on a powerful Linux machine, it takes about 5.8 seconds to compile `gen/node_snapshot.cc` with g++. As in the previous PR which dealt with `node_js2c`, we add a new build define `NODE_MKSNAPSHOT_USE_STRING_LITERALS` which is used by `node_mksnapshot`. When this flag is set, we emit string literals instead of array literals for the snapshot blob and for the code cache, i.e.: ```c++ // old: static const uint8_t X[] = { ... }; static const uint8_t *X = "..."; ``` I only enabled the new flag on Linux/macOS, since those are systems that I have available for testing. On my Linux system with gcc, it speeds up compilation of this file by 3.7s (5.8s -> 2.1s). On my Mac system with clang, it speeds up compilation by 1.7s (3.4s -> 1.7s). Again, the right thing here is probably to generate separate files for the snapshot blob and for each code cache output, but this is a nice intermediate speedup. Refs: nodejs#47984 Refs: nodejs#48160

kvakil · 2023-05-25T17:39:47Z

If we're doing toolchain-specific speed hacks: a trick I've used in other projects is to emit the .o directly. Much (much!) faster than first generating, then parsing a huge C source file.
I only had to care about x86_64 Linux though. For node it'd be a bit more complex.

Wow, generating a .o with objcopy is really fast. There's more good suggestions in this blog, apparently using .incbin with an assembler is about equally fast as objcopy/ld.

I guess, the advantage of this approach is that it's simpler (no extra files). I may try to put up another diff using incbin or similar.

This makes the test-linux GHA ends 30 minutes earlier than on other PRs, which is impressive and seems to be worth it by itself even if it had no effect on incremental compilation!

That's got to be some sort of fluke, I don't expect it to help that much.

Thanks Joyee for the suggestion to switch to a build flag, which makes this diff much shorter. (I'm going to apply a similar treatment to #48160.)

nodejs-github-bot · 2023-05-25T18:02:39Z

CI: https://ci.nodejs.org/job/node-test-pull-request/51964/

nodejs-github-bot · 2023-05-25T18:14:34Z

CI: https://ci.nodejs.org/job/node-test-pull-request/51965/

nodejs-github-bot · 2023-05-31T16:28:01Z

CI: https://ci.nodejs.org/job/node-test-pull-request/52028/

nodejs-github-bot · 2023-05-31T16:41:43Z

CI: https://ci.nodejs.org/job/node-test-pull-request/52031/

anonrig · 2023-05-31T17:08:43Z

@kvakil the build is failing

nodejs-github-bot · 2023-06-01T05:43:28Z

CI: https://ci.nodejs.org/job/node-test-pull-request/52044/

RaisinTen · 2023-06-01T05:43:59Z

It's probably a flake because the error was about missing .tap files. I have started a fresh build.

nodejs-github-bot · 2023-06-01T07:25:19Z

CI: https://ci.nodejs.org/job/node-test-pull-request/52046/

nodejs-github-bot · 2023-06-01T15:41:54Z

Landed in 287dada

Incremental compilation of Node.js is slow. Currently on a powerful Linux machine, it takes about 5.8 seconds to compile `gen/node_snapshot.cc` with g++. As in the previous PR which dealt with `node_js2c`, we add a new build define `NODE_MKSNAPSHOT_USE_STRING_LITERALS` which is used by `node_mksnapshot`. When this flag is set, we emit string literals instead of array literals for the snapshot blob and for the code cache, i.e.: ```c++ // old: static const uint8_t X[] = { ... }; static const uint8_t *X = "..."; ``` I only enabled the new flag on Linux/macOS, since those are systems that I have available for testing. On my Linux system with gcc, it speeds up compilation of this file by 3.7s (5.8s -> 2.1s). On my Mac system with clang, it speeds up compilation by 1.7s (3.4s -> 1.7s). Again, the right thing here is probably to generate separate files for the snapshot blob and for each code cache output, but this is a nice intermediate speedup. Refs: #47984 Refs: #48160 PR-URL: #48162 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

Incremental compilation of Node.js is slow. Currently on a powerful Linux machine, it takes about 5.8 seconds to compile `gen/node_snapshot.cc` with g++. As in the previous PR which dealt with `node_js2c`, we add a new build define `NODE_MKSNAPSHOT_USE_STRING_LITERALS` which is used by `node_mksnapshot`. When this flag is set, we emit string literals instead of array literals for the snapshot blob and for the code cache, i.e.: ```c++ // old: static const uint8_t X[] = { ... }; static const uint8_t *X = "..."; ``` I only enabled the new flag on Linux/macOS, since those are systems that I have available for testing. On my Linux system with gcc, it speeds up compilation of this file by 3.7s (5.8s -> 2.1s). On my Mac system with clang, it speeds up compilation by 1.7s (3.4s -> 1.7s). Again, the right thing here is probably to generate separate files for the snapshot blob and for each code cache output, but this is a nice intermediate speedup. Refs: nodejs#47984 Refs: nodejs#48160 PR-URL: nodejs#48162 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels May 25, 2023

anonrig reviewed May 25, 2023
View reviewed changes

Comment threadsrc/node_snapshotable.cc Outdated
Comment threadsrc/node_snapshotable.cc Outdated
Comment threadtools/snapshot/node_mksnapshot.cc Outdated

kvakil force-pushed the speedup-snapshot-compile branch from bdd18a5 to 3836ab5Compare May 25, 2023 17:09

anonrig approved these changes May 25, 2023
View reviewed changes

anonrig added the request-ci Add this label to start a Jenkins CI on a PR. label May 25, 2023

github-actionsBot removed the request-ci Add this label to start a Jenkins CI on a PR. label May 25, 2023

kvakil added the request-ci Add this label to start a Jenkins CI on a PR. label May 25, 2023

anonrig added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label May 25, 2023

github-actionsBot removed the request-ci Add this label to start a Jenkins CI on a PR. label May 25, 2023

joyeecheung approved these changes May 31, 2023
View reviewed changes

RaisinTen added the commit-queue Add this label to land a pull request using GitHub Actions. label Jun 1, 2023

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Jun 1, 2023

nodejs-github-bot merged commit 287dada into nodejs:mainJun 1, 2023

This was referenced Jun 2, 2023
CI Reliability 2023-06-02 nodejs/reliability#583
Open
CI Reliability 2023-06-03 nodejs/reliability#584
Open
CI Reliability 2023-06-04 nodejs/reliability#585
Open

targos mentioned this pull request Jun 4, 2023
v20.3.0 release proposal #48332
Merged

This was referenced Jun 5, 2023
CI Reliability 2023-06-05 nodejs/reliability#586
Open
CI Reliability 2023-06-06 nodejs/reliability#587
Open

danielleadams added the backport-requested-v18.x label Jul 4, 2023

Uh oh!

Uh oh!

Conversation

kvakil commented May 25, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nodejs-github-bot commented May 25, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bnoordhuis commented May 25, 2023

Uh oh!

aduh95 commented May 25, 2023

Uh oh!

kvakil commented May 25, 2023

Uh oh!

nodejs-github-bot commented May 25, 2023

Uh oh!

nodejs-github-bot commented May 25, 2023

Uh oh!

nodejs-github-bot commented May 31, 2023

Uh oh!

nodejs-github-bot commented May 31, 2023

Uh oh!

anonrig commented May 31, 2023

Uh oh!

nodejs-github-bot commented Jun 1, 2023

Uh oh!

RaisinTen commented Jun 1, 2023

Uh oh!

nodejs-github-bot commented Jun 1, 2023

Uh oh!

nodejs-github-bot commented Jun 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

kvakil commented May 25, 2023•
edited
Loading