Skip to content

test_runner: improve enqueue and dequeue - #49751

Closed
MoLow wants to merge 3 commits into
nodejs:mainfrom
MoLow:test-runner-enqueu-dequeue
Closed

test_runner: improve enqueue and dequeue#49751
MoLow wants to merge 3 commits into
nodejs:mainfrom
MoLow:test-runner-enqueu-dequeue

Conversation

@MoLow

Copy link
Copy Markdown
Member

extracting parts out of #48932 to land first, will make it easier for me to complete that big PR

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/test_runner

@nodejs-github-botnodejs-github-bot added needs-ci PRs that need a full CI run. test_runner Issues and PRs related to the test runner subsystem. labels Sep 21, 2023
@MoLow

Copy link
Copy Markdown
MemberAuthor

@MoLow
MoLow marked this pull request as ready for review September 21, 2023 19:17
@rluvaton

Copy link
Copy Markdown
Member

Benchmark output:

 confidence improvement accuracy (*) (**) (***)
test_runner/global-concurrent-tests.js type='async' n=100 ** -5.62 % ±3.69% ±4.92% ±6.40%
test_runner/global-concurrent-tests.js type='async' n=1000 -3.01 % ±6.33% ±8.43% ±10.97%
test_runner/global-concurrent-tests.js type='async' n=10000 *** -11.68 % ±1.86% ±2.48% ±3.24%
test_runner/global-concurrent-tests.js type='sync' n=100 ** -8.29 % ±5.45% ±7.25% ±9.46%
test_runner/global-concurrent-tests.js type='sync' n=1000 -3.87 % ±5.29% ±7.04% ±9.17%
test_runner/global-concurrent-tests.js type='sync' n=10000 *** -12.91 % ±1.85% ±2.46% ±3.21%
test_runner/global-sequential-tests.js type='async' n=100 -0.55 % ±1.62% ±2.16% ±2.81%
test_runner/global-sequential-tests.js type='async' n=1000 0.71 % ±2.49% ±3.32% ±4.33%
test_runner/global-sequential-tests.js type='async' n=10000 0.68 % ±1.50% ±2.00% ±2.62%
test_runner/global-sequential-tests.js type='sync' n=100 0.01 % ±1.30% ±1.74% ±2.26%
test_runner/global-sequential-tests.js type='sync' n=1000 1.20 % ±1.40% ±1.86% ±2.42%
test_runner/global-sequential-tests.js type='sync' n=10000 1.13 % ±2.15% ±2.88% ±3.77%
test_runner/multi-files.js concurrency='no' patterns='test/fixtures/test-runner/**/*.?(c|m)js' 0.02 % ±0.64% ±0.85% ±1.11%
test_runner/multi-files.js concurrency='no' patterns='test/parallel/test-runner-*' -0.37 % ±0.99% ±1.32% ±1.72%
test_runner/multi-files.js concurrency='yes' patterns='test/fixtures/test-runner/**/*.?(c|m)js' -0.06 % ±0.45% ±0.60% ±0.78%
test_runner/multi-files.js concurrency='yes' patterns='test/parallel/test-runner-*' 0.80 % ±4.69% ±6.29% ±8.29%
test_runner/suite-tests.js concurrency='no' testType='async' testsPerSuite=10 numberOfSuites=10 *** -16.52 % ±7.34% ±9.79% ±12.79%
test_runner/suite-tests.js concurrency='no' testType='async' testsPerSuite=10 numberOfSuites=100 -5.35 % ±6.64% ±8.83% ±11.49%
test_runner/suite-tests.js concurrency='no' testType='async' testsPerSuite=100 numberOfSuites=10 * -7.93 % ±6.35% ±8.46% ±11.03%
test_runner/suite-tests.js concurrency='no' testType='async' testsPerSuite=100 numberOfSuites=100 *** -15.74 % ±3.83% ±5.12% ±6.69%
test_runner/suite-tests.js concurrency='no' testType='sync' testsPerSuite=10 numberOfSuites=10 ** -9.80 % ±5.75% ±7.66% ±9.97%
test_runner/suite-tests.js concurrency='no' testType='sync' testsPerSuite=10 numberOfSuites=100 -6.93 % ±7.02% ±9.34% ±12.16%
test_runner/suite-tests.js concurrency='no' testType='sync' testsPerSuite=100 numberOfSuites=10 ** -10.70 % ±6.46% ±8.62% ±11.27%
test_runner/suite-tests.js concurrency='no' testType='sync' testsPerSuite=100 numberOfSuites=100 *** -15.67 % ±4.25% ±5.67% ±7.40%
test_runner/suite-tests.js concurrency='yes' testType='async' testsPerSuite=10 numberOfSuites=10 ** -9.35 % ±5.73% ±7.62% ±9.92%
test_runner/suite-tests.js concurrency='yes' testType='async' testsPerSuite=10 numberOfSuites=100 -3.61 % ±7.13% ±9.49% ±12.35%
test_runner/suite-tests.js concurrency='yes' testType='async' testsPerSuite=100 numberOfSuites=10 -5.42 % ±6.65% ±8.86% ±11.53%
test_runner/suite-tests.js concurrency='yes' testType='async' testsPerSuite=100 numberOfSuites=100 *** -15.05 % ±4.11% ±5.48% ±7.14%
test_runner/suite-tests.js concurrency='yes' testType='sync' testsPerSuite=10 numberOfSuites=10 *** -15.59 % ±6.39% ±8.52% ±11.13%
test_runner/suite-tests.js concurrency='yes' testType='sync' testsPerSuite=10 numberOfSuites=100 -3.53 % ±6.85% ±9.12% ±11.87%
test_runner/suite-tests.js concurrency='yes' testType='sync' testsPerSuite=100 numberOfSuites=10 -2.89 % ±3.45% ±4.63% ±6.11%
test_runner/suite-tests.js concurrency='yes' testType='sync' testsPerSuite=100 numberOfSuites=100 *** -14.85 % ±4.63% ±6.17% ±8.04%
Be aware that when doing many comparisons the risk of a false-positive
result increases. In this case, there are 32 comparisons, you can thus
expect the following amount of false-positive results:
1.60 false positives, when considering a 5% risk acceptance (*, **, ***),
0.32 false positives, when considering a 1% risk acceptance (**, ***),
0.03 false positives, when considering a 0.1% risk acceptance (***)

Comment threadbenchmark/test_runner/suite-tests.js Outdated
Comment threadbenchmark/test_runner/suite-tests.js Outdated
Comment threadtest/fixtures/test-runner/output/describe_it.js Outdated
Comment threadtest/fixtures/test-runner/output/describe_it.snapshot Outdated
@RelxOff

Copy link
Copy Markdown

💯

@MoLow
MoLowforce-pushed the test-runner-enqueu-dequeue branch from 6626cd4 to 37e1373CompareSeptember 23, 2023 20:16
@MoLowMoLow closed this Mar 17, 2024
@MoLow
MoLow deleted the test-runner-enqueu-dequeue branch May 24, 2024 09:01
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-ciPRs that need a full CI run.test_runnerIssues and PRs related to the test runner subsystem.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@MoLow@nodejs-github-bot@rluvaton@RelxOff