Skip to content

benchmark: rewrite the startup performance benchmark - #45746

Merged
nodejs-github-bot merged 5 commits into
nodejs:mainfrom
joyeecheung:startup-bench
Dec 8, 2022
Merged

benchmark: rewrite the startup performance benchmark#45746
nodejs-github-bot merged 5 commits into
nodejs:mainfrom
joyeecheung:startup-bench

Conversation

@joyeecheung

Copy link
Copy Markdown
Member

benchmark: make benchmarks runnable in older versions of Node.js

Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.

benchmark: rewrite the startup performance benchmark

The previous benchmark was inaccurate in several ways:

  • It previously measured "how many workers/child processes can finish
    loading in parallel in 1 second" which is prone to wrong
    interpretations. It's also not typical for users to spin up >20
    workers/child processes in parallel, which was what the benchmark
    usually ended up doing. The parallelism can also be greatly affected
    by system configurations.
  • The time spent on loading a Node.js binary for the first time varies
    greatly depending on system configurations, but it was inevitable
    for the benchmark to load Node.js multiple times to reduce the
    influence of fluctuations.

This patch rewrites the benchmark so that:

  • It now measures "how long it takes to finish 30 workers/child processes
    in serial" which is generally more in line with how other benchmarks
    are written and with the figures one gets from doing something like
    time node index.js or hyperfine 'node index.js'
  • It now warms up the loading of the Node.js instance to reduce the
    statistical outliers, so that we get more meaningful figures when
    trying to compare the startup performance of different Node.js
    binaries.

Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.
The previous benchmark was inaccurate in several ways:
- It previously measured "how many workers/child processes can finish
loading in parallel in 1 second" which is prone to wrong
interpretations. It's also not typical for users to spin up >20
workers/child processes in parallel, which was what the benchmark
usually ended up doing. The parallelism can also be greatly affected
by system configurations.
- The time spent on loading a Node.js binary for the first time varies
greatly depending on system configurations, but it was inevitable
for the benchmark to load Node.js multiple times to reduce the
influence of fluctuations.
This patch rewrites the benchmark so that:
- It now measures "how long it takes to finish 30 workers/child processes
in serial" which is generally more in line with how other benchmarks
are written and with the figures one gets from doing something like
`time node index.js` or `hyperfine 'node index.js'`
- It now warms up the loading of the Node.js instance to reduce the
statistical outliers, so that we get more meaningful figures when
trying to compare the startup performance of different Node.js
binaries.
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/startup

@nodejs-github-botnodejs-github-bot added the benchmark Issues and PRs related to the benchmark subsystem. label Dec 5, 2022
@joyeecheung

joyeecheung commented Dec 5, 2022

Copy link
Copy Markdown
MemberAuthor

New benchmark results from #45659, which is more in line with the results I see from hyperfine:

 confidence improvement accuracy (*) (**) (***)
misc/startup.js count=30 mode='process' script='benchmark/fixtures/require-builtins' -0.25 % ±2.05% ±2.78% ±3.71%
misc/startup.js count=30 mode='process' script='test/fixtures/semicolon' *** 14.04 % ±2.80% ±3.79% ±5.06%
misc/startup.js count=30 mode='worker' script='benchmark/fixtures/require-builtins' 0.19 % ±2.10% ±2.84% ±3.79%
misc/startup.js count=30 mode='worker' script='test/fixtures/semicolon' *** 16.86 % ±2.67% ±3.61% ±4.82%

I'll recompile and see the new results from #45716

@joyeecheung

Copy link
Copy Markdown
MemberAuthor

New benchmark numbers from #45716


confidence improvement accuracy (*) (**) (***)
misc/startup.js count=30 mode='process' script='benchmark/fixtures/require-builtins' *** 12.28 % ±1.13% ±1.52% ±2.00%
misc/startup.js count=30 mode='process' script='test/fixtures/semicolon' *** 17.39 % ±1.12% ±1.50% ±1.97%
misc/startup.js count=30 mode='worker' script='benchmark/fixtures/require-builtins' *** 14.54 % ±0.77% ±1.03% ±1.35%
misc/startup.js count=30 mode='worker' script='test/fixtures/semicolon' *** 18.02 % ±1.02% ±1.36% ±1.79%

Comment threadbenchmark/misc/startup.js Outdated
@legendecas

Copy link
Copy Markdown
Member

Linter is still complaining.

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@joyeecheungjoyeecheung added the review wanted PRs that need reviews. label Dec 8, 2022
@joyeecheung

Copy link
Copy Markdown
MemberAuthor

@nodejs/performance Can I have some reviews please?

@anonriganonrig added performance Issues and PRs related to the performance of Node.js. author ready PRs that have at least one approval, no pending requests for changes, and a CI started. commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. commit-queue Add this label to land a pull request using GitHub Actions. labels Dec 8, 2022
@nodejs-github-botnodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Dec 8, 2022
@nodejs-github-bot
nodejs-github-bot merged commit 2b9d3b2 into nodejs:mainDec 8, 2022
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Landed in 2b9d3b2

targos pushed a commit that referenced this pull request Dec 12, 2022
Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.
PR-URL: #45746
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
@targostargos mentioned this pull request Dec 12, 2022
targos pushed a commit that referenced this pull request Dec 13, 2022
Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.
PR-URL: #45746
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
danielleadams pushed a commit that referenced this pull request Dec 30, 2022
Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.
PR-URL: #45746
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
danielleadams pushed a commit that referenced this pull request Jan 3, 2023
Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.
PR-URL: #45746
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
danielleadams pushed a commit that referenced this pull request Jan 5, 2023
Also remove the require-cachable.js benchmarks because now all builtin
modules are cacheable, it would be comparing oranges to apples when
we try to compare the performance of loading all cacheable modules
in different Node.js binaries since the set of modules are just
different. Comparison of startup performance that involves loading
of the long-standing, stable builtins is already covered by the
require-builtins benchmark.
PR-URL: #45746
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author readyPRs that have at least one approval, no pending requests for changes, and a CI started.benchmarkIssues and PRs related to the benchmark subsystem.commit-queue-squashAdd this label to instruct the Commit Queue to squash all the PR commits into the first one.performanceIssues and PRs related to the performance of Node.js.review wantedPRs that need reviews.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@joyeecheung@nodejs-github-bot@legendecas@addaleax@anonrig