benchmark: add benchmarks for the test_runner by rluvaton · Pull Request #48931 · nodejs/node

rluvaton · 2023-07-26T19:38:02Z

I'm trying to benchmark tests creation and execution

this is needed as we need to evaluate how a change in the test runner affects the performance:

test_runner: cleanup test timeout abort listener #48915 (comment)

rluvaton · 2023-07-26T19:38:53Z

+ });
+ }
+
+ await Promise.all(promises);


Should we use finished on the reporter or this?

anonrig

I mostly didn't understand what we are benchmarking. Can you leave some descriptive comments on what we are benchmarking? PS: The file names does not help either. (For example: plain-tests-as-fast-as-can)

anonrig · 2023-07-26T19:53:15Z

cc @nodejs/performance @MoLow @cjihrig

MoLow · 2023-07-26T20:04:52Z

I mostly didn't understand what we are benchmarking. Can you leave some descriptive comments on what we are benchmarking? PS: The file names does not help either. (For example: plain-tests-as-fast-as-can)

we want to compare how changes in test runner code affect its performance, but I agree naming can be improved

@rluvaton in regarding to #47945 (comment) - @cjihrig and myself were talking about what we want to benchmark - we probably also want to run tests with --test (probably one case should run a single file with --test and another should run multiple files (perhaps all of the test-runner fixture folder))
can you add those benchmarks to this pr? if not, Il do it in a follow-up PR

rluvaton · 2023-07-26T20:10:39Z

[...] and another should run multiple files (perhaps all of the test-runner fixture folder)

I don't think running the test-runner fixture folder is a good idea as it changes, while I think benchmarks are best when they are static

we probably also want to run tests with --test

I avoided that on purpose to reduce the scope of the benchmark, adding the --test as well will also benchmark the process and read files and more

MoLow · 2023-07-26T20:15:12Z

[...] and another should run multiple files (perhaps all of the test-runner fixture folder)
I don't think running the test-runner fixture folder is a good idea as it changes, while I think benchmarks are best when they are static

it is ok since the benchmarks are compared before and after a change we want to understand the impact of. no baseline or benchmark is saved according to @cjihrig

we probably also want to run tests with --test
I avoided that on purpose to reduce the scope of the benchmark, adding the --test as well will also benchmark the process and read files and more

that is a good point, but I think that should only mean we should rely on this specific benchmark only if it is significantly different, not that we shouldn't test it at all

rluvaton · 2023-07-28T06:28:49Z

can you add those benchmarks to this pr? if not, Il do it in a follow-up PR

I prefer if you add those as you also need to add only and also I faced some problems with adding the --test flag, if it's really important to you I can put some time to fix those...

MoLow

LGTM

nodejs-github-bot · 2023-07-29T18:07:45Z

Landed in f458e5b

PR-URL: nodejs#48931 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: Debadree Chatterjee <debadree333@gmail.com>

PR-URL: #48931 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: Debadree Chatterjee <debadree333@gmail.com>

PR-URL: nodejs#48931 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: Debadree Chatterjee <debadree333@gmail.com>

PR-URL: #48931 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: Debadree Chatterjee <debadree333@gmail.com>

PR-URL: nodejs/node#48931 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: Debadree Chatterjee <debadree333@gmail.com>

Update the hooks benchmark so registered hooks perform the same anti-optimization assignment as test bodies instead of calling a noop. This keeps the measured hook path from using an empty callback. Links for more information on this actions: nodejs#48931 (comment)https://www.mail-archive.com/v8-users@googlegroups.com/msg05521.html Signed-off-by: Luan Muniz <luan@luanmuniz.com.br>

benchmark: add benchmarks for the test_runner
8700926

nodejs-github-bot added benchmark Issues and PRs related to the benchmark subsystem. test_runner Issues and PRs related to the test runner subsystem. labels Jul 26, 2023

remove todo
4781ca7

rluvaton commented Jul 26, 2023
View reviewed changes

remove 1K suites due to OOM in conjunction with 1K tests per suite
d9e9b1c

rluvaton mentioned this pull request Jul 26, 2023
test_runner: cleanup test timeout abort listener #48915
Merged

anonrig reviewed Jul 26, 2023
View reviewed changes

rename files
a67b01d

MoLow reviewed Jul 26, 2023
View reviewed changes

Comment threadbenchmark/test_runner/global-concurrent-tests.js Outdated

MoLow reviewed Jul 26, 2023
View reviewed changes

Comment threadbenchmark/test_runner/global-sequential-tests.js Outdated

MoLow reviewed Jul 26, 2023
View reviewed changes

Comment threadbenchmark/test_runner/suite-tests.js Outdated

sync and async tests
82e20f7

rluvaton added 2 commits July 26, 2023 23:15

sync and async tests
1308c10

add concurrency options
929e0d2

MoLow mentioned this pull request Jul 26, 2023
test_runner: propagate only to test ancestors #48932
Closed
5 tasks

H4ad reviewed Jul 26, 2023
View reviewed changes

Comment threadbenchmark/test_runner/global-concurrent-tests.js

rluvaton added 2 commits July 28, 2023 11:28

trigger GitHub Actions CI
dbb1380

avoid v8 optimization
994e575

MoLow approved these changes Jul 28, 2023
View reviewed changes

H4ad approved these changes Jul 28, 2023
View reviewed changes

atlowChemi approved these changes Jul 28, 2023
View reviewed changes

debadree25 approved these changes Jul 29, 2023
View reviewed changes

debadree25 added author ready PRs that have at least one approval, no pending requests for changes, and a CI started. commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. labels Jul 29, 2023

debadree25 added the commit-queue Add this label to land a pull request using GitHub Actions. label Jul 29, 2023

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Jul 29, 2023

nodejs-github-bot merged commit f458e5b into nodejs:mainJul 29, 2023

rluvaton deleted the add-test-runner-benchmarks branch July 29, 2023 18:09

RafaelGSS pushed a commit that referenced this pull request Aug 15, 2023
benchmark: add benchmarks for the test_runner
2608c9f
PR-URL: #48931 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: Debadree Chatterjee <debadree333@gmail.com>

UlisesGascon mentioned this pull request Aug 15, 2023
v20.6.0 proposal #49185
Merged

RafaelGSS pushed a commit that referenced this pull request Aug 17, 2023
benchmark: add benchmarks for the test_runner
771abcb
PR-URL: #48931 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: Debadree Chatterjee <debadree333@gmail.com>

targos pushed a commit that referenced this pull request Nov 27, 2023
benchmark: add benchmarks for the test_runner
0d0eb47
PR-URL: #48931 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: Debadree Chatterjee <debadree333@gmail.com>

targos mentioned this pull request Nov 28, 2023
v18.19.0 release proposal #50953
Merged

luanmuniz mentioned this pull request Jun 8, 2026
benchmark: add test runner hooks and options #63754
Open

Navigation Menu

Uh oh!

Uh oh!

benchmark: add benchmarks for the test_runner - #48931

benchmark: add benchmarks for the test_runner#48931
nodejs-github-bot merged 9 commits into
nodejs:mainfrom
rluvaton:add-test-runner-benchmarks

rluvaton commented Jul 26, 2023•
edited
Loading

Uh oh!

rluvatonJul 26, 2023

Uh oh!

anonrig left a comment

Uh oh!

anonrig commented Jul 26, 2023

Uh oh!

MoLow commented Jul 26, 2023•
edited
Loading

Uh oh!

Uh oh!

rluvaton commented Jul 26, 2023

Uh oh!

Uh oh!

Uh oh!

MoLow commented Jul 26, 2023

Uh oh!

Uh oh!

rluvaton commented Jul 28, 2023

Uh oh!

MoLow left a comment

Uh oh!

nodejs-github-bot commented Jul 29, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Uh oh!

Conversation

rluvaton commented Jul 26, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rluvatonJul 26, 2023

Choose a reason for hiding this comment

Uh oh!

anonrig left a comment

Choose a reason for hiding this comment

Uh oh!

anonrig commented Jul 26, 2023

Uh oh!

MoLow commented Jul 26, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

rluvaton commented Jul 26, 2023

Uh oh!

Uh oh!

Uh oh!

MoLow commented Jul 26, 2023

Uh oh!

Uh oh!

rluvaton commented Jul 28, 2023

Uh oh!

MoLow left a comment

Choose a reason for hiding this comment

Uh oh!

nodejs-github-bot commented Jul 29, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

rluvaton commented Jul 26, 2023•
edited
Loading

MoLow commented Jul 26, 2023•
edited
Loading