perf_hooks: add statistical hypothesis testing to histogram - #65416

Closed
jasnell wants to merge 2 commits into
nodejs:mainfrom
jasnell:jasnell/more-histogram-fun
Closed

perf_hooks: add statistical hypothesis testing to histogram#65416
jasnell wants to merge 2 commits into
nodejs:mainfrom
jasnell:jasnell/more-histogram-fun

Conversation

@jasnell

@jasnelljasnell commented Aug 20, 2026

Copy link
Copy Markdown
Member

Continue expanding the capabilities of the Histogram API... and use it to enable a dependency-free benchmark/compare.js option... i.e. no longer requiring R-script to show benchmark analysis... just run benchmark/compare.js with the --analyze option.

The added algorithms cover analysis calculations that would be fairly common in perf analysis (latency, burn rates, detecting regressions, etc). Prior to this, really the only way to get these were to use aging outside dependencies (5+ years old) or R-script.

Just scratching an old itch I've been wanting since I added Histogram back in v11. I've been doing a bunch of benchmarking lately with the QUIC and DTLS impls and just got tired of having to use outside deps for this stuff.

/cc @mcollina

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/performance
  • @nodejs/tsc

@jasnell
jasnell requested review from Qard and mcollinaAugust 20, 2026 06:25
@nodejs-github-botnodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. labels Aug 20, 2026
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from f866877 to bc4d71eCompareAugust 20, 2026 13:24
@codecov

codecovBot commented Aug 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.44262% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.14%. Comparing base (cc57cb7) to head (fedf4f2).
⚠️ Report is 20 commits behind head on main.

Files with missing linesPatch %Lines
src/histogram.cc90.76%4 Missing and 30 partials ⚠️
src/histogram-inl.h86.36%3 Missing and 3 partials ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #65416 +/- ##
==========================================
+ Coverage 90.12% 90.14% +0.02% 
==========================================
Files 751 751 Lines 252536 253137 +601 Branches 47519 47631 +112 ==========================================
+ Hits 227598 228191 +593 + Misses 16250 16202 -48 - Partials 8688 8744 +56 
Files with missing linesCoverage Δ
lib/internal/histogram.js96.09% <100.00%> (+1.32%)⬆️
src/histogram.h68.42% <ø> (ø)
src/histogram-inl.h84.45% <86.36%> (+2.21%)⬆️
src/histogram.cc84.62% <90.76%> (+2.83%)⬆️

... and 35 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mcollinamcollina left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rslgtm

@nodejs-github-bot

This comment was marked as outdated.

@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from bc4d71e to 54c4c56CompareAugust 20, 2026 19:19
@nodejs-github-bot

This comment was marked as outdated.

@jasnell

Copy link
Copy Markdown
MemberAuthor

/cc @nodejs/performance

@jasnelljasnell added the author ready PRs with CI started, the required approvals, and no outstanding review comments. label Aug 20, 2026
Comment threadlib/internal/histogram.js Outdated
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from 54c4c56 to 0793579CompareAugust 20, 2026 20:35
@nodejs-github-bot

This comment was marked as outdated.

@jasnelljasnell added the commit-queue-rebase PRs the Commit Queue should land as multiple self-contained commits. label Aug 20, 2026
@jasnelljasnell added the notable-change PRs with changes that should be highlighted in changelogs. label Aug 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

The notable-changePRs with changes that should be highlighted in changelogs. label has been added by @jasnell.

Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section.

@jasnell

Copy link
Copy Markdown
MemberAuthor

Notes for the notable change:

This PR expands the analytical capabilities of Histogram further by adding EWMA, cliffsD, cohensD, SLO burn rate, mannWhitneyTest, and percentile confidence intervals to the API. Notably, this provides the ability to perform performance benchmark analysis directly within Node.js with no dependencies.

@jasnelljasnell added the semver-minor PRs that contain new features and should be released in the next minor version. label Aug 20, 2026
@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from 0793579 to fedf4f2CompareAugust 22, 2026 23:07
@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

jasnell added a commit that referenced this pull request Aug 23, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
jasnell added a commit that referenced this pull request Aug 23, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
@jasnell

Copy link
Copy Markdown
MemberAuthor

Landed in c28857e...bf67fdc

@jasnelljasnell closed this Aug 23, 2026
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
nodejs-github-bot added a commit that referenced this pull request Aug 25, 2026
Notable changes:
benchmark:
* (SEMVER-MINOR) add --analyze mode to compare.js (James M Snell) #65416
crypto:
* update root certificates to NSS 3.126 (Node.js GitHub Bot) #65495
* (SEMVER-MINOR) enable SIV and GCM-SIV modes in Cipher/Decipher APIs (Filip Skokan) #63411
diagnostics_channel:
* mark TracingChannel as stable (Abdelrahman Awad) #64525
lib,src:
* (SEMVER-MINOR) improve histogram implementation (James M Snell) #65024
net:
* (SEMVER-MINOR) improve performance of net.BlockList (James M Snell) #64974
perf_hooks:
* (SEMVER-MINOR) add statistical hypothesis testing to histogram (James M Snell) #65416
repl:
* add basic syntax highlighting (Aviv Keller) #64591
sqlite:
* (SEMVER-MINOR) add StatementSync.prototype.close() (Guilherme Araújo) #64232
* (SEMVER-MINOR) add StatementSync.prototype[Symbol.dispose]() (Guilherme Araújo) #64232
util:
* (SEMVER-MINOR) add non-throwing MIMEType.parse (James M Snell) #64965
zlib:
* (SEMVER-MINOR) add ZipEntry, ZipFile, and ZipBuffer (Philipp Dunkel) #64339
PR-URL: #65551
aduh95 pushed a commit that referenced this pull request Aug 26, 2026
Notable changes:
benchmark:
* (SEMVER-MINOR) add --analyze mode to compare.js (James M Snell) #65416
crypto:
* update root certificates to NSS 3.126 (Node.js GitHub Bot) #65495
* (SEMVER-MINOR) enable SIV and GCM-SIV modes in Cipher/Decipher APIs (Filip Skokan) #63411
diagnostics_channel:
* mark TracingChannel as stable (Abdelrahman Awad) #64525
lib,src:
* (SEMVER-MINOR) improve histogram implementation (James M Snell) #65024
net:
* (SEMVER-MINOR) improve performance of net.BlockList (James M Snell) #64974
perf_hooks:
* (SEMVER-MINOR) add statistical hypothesis testing to histogram (James M Snell) #65416
repl:
* add basic syntax highlighting (Aviv Keller) #64591
sqlite:
* (SEMVER-MINOR) add StatementSync.prototype.close() (Guilherme Araújo) #64232
* (SEMVER-MINOR) add StatementSync.prototype[Symbol.dispose]() (Guilherme Araújo) #64232
util:
* (SEMVER-MINOR) add non-throwing MIMEType.parse (James M Snell) #64965
zlib:
* (SEMVER-MINOR) add ZipEntry, ZipFile, and ZipBuffer (Philipp Dunkel) #64339
PR-URL: #65551
aduh95 pushed a commit that referenced this pull request Aug 27, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 27, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author readyPRs with CI started, the required approvals, and no outstanding review comments.c++Issues and PRs that require attention from people who are familiar with C++.commit-queue-rebasePRs the Commit Queue should land as multiple self-contained commits.needs-ciPRs that need a full CI run.notable-changePRs with changes that should be highlighted in changelogs.semver-minorPRs that contain new features and should be released in the next minor version.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@jasnell@nodejs-github-bot@mcollina@legendecas
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

perf_hooks: add statistical hypothesis testing to histogram - #65416

Closed
jasnell wants to merge 2 commits into
nodejs:mainfrom
jasnell:jasnell/more-histogram-fun
Closed

perf_hooks: add statistical hypothesis testing to histogram#65416
jasnell wants to merge 2 commits into
nodejs:mainfrom
jasnell:jasnell/more-histogram-fun

Conversation

@jasnell

@jasnelljasnell commented Aug 20, 2026

Copy link
Copy Markdown
Member

Continue expanding the capabilities of the Histogram API... and use it to enable a dependency-free benchmark/compare.js option... i.e. no longer requiring R-script to show benchmark analysis... just run benchmark/compare.js with the --analyze option.

The added algorithms cover analysis calculations that would be fairly common in perf analysis (latency, burn rates, detecting regressions, etc). Prior to this, really the only way to get these were to use aging outside dependencies (5+ years old) or R-script.

Just scratching an old itch I've been wanting since I added Histogram back in v11. I've been doing a bunch of benchmarking lately with the QUIC and DTLS impls and just got tired of having to use outside deps for this stuff.

/cc @mcollina

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/performance
  • @nodejs/tsc

@jasnell
jasnell requested review from Qard and mcollinaAugust 20, 2026 06:25
@nodejs-github-botnodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. labels Aug 20, 2026
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from f866877 to bc4d71eCompareAugust 20, 2026 13:24
@codecov

codecovBot commented Aug 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.44262% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.14%. Comparing base (cc57cb7) to head (fedf4f2).
⚠️ Report is 20 commits behind head on main.

Files with missing linesPatch %Lines
src/histogram.cc90.76%4 Missing and 30 partials ⚠️
src/histogram-inl.h86.36%3 Missing and 3 partials ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #65416 +/- ##
==========================================
+ Coverage 90.12% 90.14% +0.02% 
==========================================
Files 751 751 Lines 252536 253137 +601 Branches 47519 47631 +112 ==========================================
+ Hits 227598 228191 +593 + Misses 16250 16202 -48 - Partials 8688 8744 +56 
Files with missing linesCoverage Δ
lib/internal/histogram.js96.09% <100.00%> (+1.32%)⬆️
src/histogram.h68.42% <ø> (ø)
src/histogram-inl.h84.45% <86.36%> (+2.21%)⬆️
src/histogram.cc84.62% <90.76%> (+2.83%)⬆️

... and 35 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mcollinamcollina left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rslgtm

@nodejs-github-bot

This comment was marked as outdated.

@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from bc4d71e to 54c4c56CompareAugust 20, 2026 19:19
@nodejs-github-bot

This comment was marked as outdated.

@jasnell

Copy link
Copy Markdown
MemberAuthor

/cc @nodejs/performance

@jasnelljasnell added the author ready PRs with CI started, the required approvals, and no outstanding review comments. label Aug 20, 2026
Comment threadlib/internal/histogram.js Outdated
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from 54c4c56 to 0793579CompareAugust 20, 2026 20:35
@nodejs-github-bot

This comment was marked as outdated.

@jasnelljasnell added the commit-queue-rebase PRs the Commit Queue should land as multiple self-contained commits. label Aug 20, 2026
@jasnelljasnell added the notable-change PRs with changes that should be highlighted in changelogs. label Aug 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

The notable-changePRs with changes that should be highlighted in changelogs. label has been added by @jasnell.

Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section.

@jasnell

Copy link
Copy Markdown
MemberAuthor

Notes for the notable change:

This PR expands the analytical capabilities of Histogram further by adding EWMA, cliffsD, cohensD, SLO burn rate, mannWhitneyTest, and percentile confidence intervals to the API. Notably, this provides the ability to perform performance benchmark analysis directly within Node.js with no dependencies.

@jasnelljasnell added the semver-minor PRs that contain new features and should be released in the next minor version. label Aug 20, 2026
@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from 0793579 to fedf4f2CompareAugust 22, 2026 23:07
@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

jasnell added a commit that referenced this pull request Aug 23, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
jasnell added a commit that referenced this pull request Aug 23, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
@jasnell

Copy link
Copy Markdown
MemberAuthor

Landed in c28857e...bf67fdc

@jasnelljasnell closed this Aug 23, 2026
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
nodejs-github-bot added a commit that referenced this pull request Aug 25, 2026
Notable changes:
benchmark:
* (SEMVER-MINOR) add --analyze mode to compare.js (James M Snell) #65416
crypto:
* update root certificates to NSS 3.126 (Node.js GitHub Bot) #65495
* (SEMVER-MINOR) enable SIV and GCM-SIV modes in Cipher/Decipher APIs (Filip Skokan) #63411
diagnostics_channel:
* mark TracingChannel as stable (Abdelrahman Awad) #64525
lib,src:
* (SEMVER-MINOR) improve histogram implementation (James M Snell) #65024
net:
* (SEMVER-MINOR) improve performance of net.BlockList (James M Snell) #64974
perf_hooks:
* (SEMVER-MINOR) add statistical hypothesis testing to histogram (James M Snell) #65416
repl:
* add basic syntax highlighting (Aviv Keller) #64591
sqlite:
* (SEMVER-MINOR) add StatementSync.prototype.close() (Guilherme Araújo) #64232
* (SEMVER-MINOR) add StatementSync.prototype[Symbol.dispose]() (Guilherme Araújo) #64232
util:
* (SEMVER-MINOR) add non-throwing MIMEType.parse (James M Snell) #64965
zlib:
* (SEMVER-MINOR) add ZipEntry, ZipFile, and ZipBuffer (Philipp Dunkel) #64339
PR-URL: #65551
aduh95 pushed a commit that referenced this pull request Aug 26, 2026
Notable changes:
benchmark:
* (SEMVER-MINOR) add --analyze mode to compare.js (James M Snell) #65416
crypto:
* update root certificates to NSS 3.126 (Node.js GitHub Bot) #65495
* (SEMVER-MINOR) enable SIV and GCM-SIV modes in Cipher/Decipher APIs (Filip Skokan) #63411
diagnostics_channel:
* mark TracingChannel as stable (Abdelrahman Awad) #64525
lib,src:
* (SEMVER-MINOR) improve histogram implementation (James M Snell) #65024
net:
* (SEMVER-MINOR) improve performance of net.BlockList (James M Snell) #64974
perf_hooks:
* (SEMVER-MINOR) add statistical hypothesis testing to histogram (James M Snell) #65416
repl:
* add basic syntax highlighting (Aviv Keller) #64591
sqlite:
* (SEMVER-MINOR) add StatementSync.prototype.close() (Guilherme Araújo) #64232
* (SEMVER-MINOR) add StatementSync.prototype[Symbol.dispose]() (Guilherme Araújo) #64232
util:
* (SEMVER-MINOR) add non-throwing MIMEType.parse (James M Snell) #64965
zlib:
* (SEMVER-MINOR) add ZipEntry, ZipFile, and ZipBuffer (Philipp Dunkel) #64339
PR-URL: #65551
aduh95 pushed a commit that referenced this pull request Aug 27, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 27, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author readyPRs with CI started, the required approvals, and no outstanding review comments.c++Issues and PRs that require attention from people who are familiar with C++.commit-queue-rebasePRs the Commit Queue should land as multiple self-contained commits.needs-ciPRs that need a full CI run.notable-changePRs with changes that should be highlighted in changelogs.semver-minorPRs that contain new features and should be released in the next minor version.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@jasnell@nodejs-github-bot@mcollina@legendecas
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

perf_hooks: add statistical hypothesis testing to histogram - #65416

Closed
jasnell wants to merge 2 commits into
nodejs:mainfrom
jasnell:jasnell/more-histogram-fun
Closed

perf_hooks: add statistical hypothesis testing to histogram#65416
jasnell wants to merge 2 commits into
nodejs:mainfrom
jasnell:jasnell/more-histogram-fun

Conversation

@jasnell

@jasnelljasnell commented Aug 20, 2026

Copy link
Copy Markdown
Member

Continue expanding the capabilities of the Histogram API... and use it to enable a dependency-free benchmark/compare.js option... i.e. no longer requiring R-script to show benchmark analysis... just run benchmark/compare.js with the --analyze option.

The added algorithms cover analysis calculations that would be fairly common in perf analysis (latency, burn rates, detecting regressions, etc). Prior to this, really the only way to get these were to use aging outside dependencies (5+ years old) or R-script.

Just scratching an old itch I've been wanting since I added Histogram back in v11. I've been doing a bunch of benchmarking lately with the QUIC and DTLS impls and just got tired of having to use outside deps for this stuff.

/cc @mcollina

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/performance
  • @nodejs/tsc

@jasnell
jasnell requested review from Qard and mcollinaAugust 20, 2026 06:25
@nodejs-github-botnodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. labels Aug 20, 2026
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from f866877 to bc4d71eCompareAugust 20, 2026 13:24
@codecov

codecovBot commented Aug 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.44262% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.14%. Comparing base (cc57cb7) to head (fedf4f2).
⚠️ Report is 20 commits behind head on main.

Files with missing linesPatch %Lines
src/histogram.cc90.76%4 Missing and 30 partials ⚠️
src/histogram-inl.h86.36%3 Missing and 3 partials ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #65416 +/- ##
==========================================
+ Coverage 90.12% 90.14% +0.02% 
==========================================
Files 751 751 Lines 252536 253137 +601 Branches 47519 47631 +112 ==========================================
+ Hits 227598 228191 +593 + Misses 16250 16202 -48 - Partials 8688 8744 +56 
Files with missing linesCoverage Δ
lib/internal/histogram.js96.09% <100.00%> (+1.32%)⬆️
src/histogram.h68.42% <ø> (ø)
src/histogram-inl.h84.45% <86.36%> (+2.21%)⬆️
src/histogram.cc84.62% <90.76%> (+2.83%)⬆️

... and 35 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mcollinamcollina left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rslgtm

@nodejs-github-bot

This comment was marked as outdated.

@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from bc4d71e to 54c4c56CompareAugust 20, 2026 19:19
@nodejs-github-bot

This comment was marked as outdated.

@jasnell

Copy link
Copy Markdown
MemberAuthor

/cc @nodejs/performance

@jasnelljasnell added the author ready PRs with CI started, the required approvals, and no outstanding review comments. label Aug 20, 2026
Comment threadlib/internal/histogram.js Outdated
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from 54c4c56 to 0793579CompareAugust 20, 2026 20:35
@nodejs-github-bot

This comment was marked as outdated.

@jasnelljasnell added the commit-queue-rebase PRs the Commit Queue should land as multiple self-contained commits. label Aug 20, 2026
@jasnelljasnell added the notable-change PRs with changes that should be highlighted in changelogs. label Aug 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

The notable-changePRs with changes that should be highlighted in changelogs. label has been added by @jasnell.

Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section.

@jasnell

Copy link
Copy Markdown
MemberAuthor

Notes for the notable change:

This PR expands the analytical capabilities of Histogram further by adding EWMA, cliffsD, cohensD, SLO burn rate, mannWhitneyTest, and percentile confidence intervals to the API. Notably, this provides the ability to perform performance benchmark analysis directly within Node.js with no dependencies.

@jasnelljasnell added the semver-minor PRs that contain new features and should be released in the next minor version. label Aug 20, 2026
@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from 0793579 to fedf4f2CompareAugust 22, 2026 23:07
@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

jasnell added a commit that referenced this pull request Aug 23, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
jasnell added a commit that referenced this pull request Aug 23, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
@jasnell

Copy link
Copy Markdown
MemberAuthor

Landed in c28857e...bf67fdc

@jasnelljasnell closed this Aug 23, 2026
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
nodejs-github-bot added a commit that referenced this pull request Aug 25, 2026
Notable changes:
benchmark:
* (SEMVER-MINOR) add --analyze mode to compare.js (James M Snell) #65416
crypto:
* update root certificates to NSS 3.126 (Node.js GitHub Bot) #65495
* (SEMVER-MINOR) enable SIV and GCM-SIV modes in Cipher/Decipher APIs (Filip Skokan) #63411
diagnostics_channel:
* mark TracingChannel as stable (Abdelrahman Awad) #64525
lib,src:
* (SEMVER-MINOR) improve histogram implementation (James M Snell) #65024
net:
* (SEMVER-MINOR) improve performance of net.BlockList (James M Snell) #64974
perf_hooks:
* (SEMVER-MINOR) add statistical hypothesis testing to histogram (James M Snell) #65416
repl:
* add basic syntax highlighting (Aviv Keller) #64591
sqlite:
* (SEMVER-MINOR) add StatementSync.prototype.close() (Guilherme Araújo) #64232
* (SEMVER-MINOR) add StatementSync.prototype[Symbol.dispose]() (Guilherme Araújo) #64232
util:
* (SEMVER-MINOR) add non-throwing MIMEType.parse (James M Snell) #64965
zlib:
* (SEMVER-MINOR) add ZipEntry, ZipFile, and ZipBuffer (Philipp Dunkel) #64339
PR-URL: #65551
aduh95 pushed a commit that referenced this pull request Aug 26, 2026
Notable changes:
benchmark:
* (SEMVER-MINOR) add --analyze mode to compare.js (James M Snell) #65416
crypto:
* update root certificates to NSS 3.126 (Node.js GitHub Bot) #65495
* (SEMVER-MINOR) enable SIV and GCM-SIV modes in Cipher/Decipher APIs (Filip Skokan) #63411
diagnostics_channel:
* mark TracingChannel as stable (Abdelrahman Awad) #64525
lib,src:
* (SEMVER-MINOR) improve histogram implementation (James M Snell) #65024
net:
* (SEMVER-MINOR) improve performance of net.BlockList (James M Snell) #64974
perf_hooks:
* (SEMVER-MINOR) add statistical hypothesis testing to histogram (James M Snell) #65416
repl:
* add basic syntax highlighting (Aviv Keller) #64591
sqlite:
* (SEMVER-MINOR) add StatementSync.prototype.close() (Guilherme Araújo) #64232
* (SEMVER-MINOR) add StatementSync.prototype[Symbol.dispose]() (Guilherme Araújo) #64232
util:
* (SEMVER-MINOR) add non-throwing MIMEType.parse (James M Snell) #64965
zlib:
* (SEMVER-MINOR) add ZipEntry, ZipFile, and ZipBuffer (Philipp Dunkel) #64339
PR-URL: #65551
aduh95 pushed a commit that referenced this pull request Aug 27, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 27, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author readyPRs with CI started, the required approvals, and no outstanding review comments.c++Issues and PRs that require attention from people who are familiar with C++.commit-queue-rebasePRs the Commit Queue should land as multiple self-contained commits.needs-ciPRs that need a full CI run.notable-changePRs with changes that should be highlighted in changelogs.semver-minorPRs that contain new features and should be released in the next minor version.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@jasnell@nodejs-github-bot@mcollina@legendecas
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

perf_hooks: add statistical hypothesis testing to histogram - #65416

Closed
jasnell wants to merge 2 commits into
nodejs:mainfrom
jasnell:jasnell/more-histogram-fun
Closed

perf_hooks: add statistical hypothesis testing to histogram#65416
jasnell wants to merge 2 commits into
nodejs:mainfrom
jasnell:jasnell/more-histogram-fun

Conversation

@jasnell

@jasnelljasnell commented Aug 20, 2026

Copy link
Copy Markdown
Member

Continue expanding the capabilities of the Histogram API... and use it to enable a dependency-free benchmark/compare.js option... i.e. no longer requiring R-script to show benchmark analysis... just run benchmark/compare.js with the --analyze option.

The added algorithms cover analysis calculations that would be fairly common in perf analysis (latency, burn rates, detecting regressions, etc). Prior to this, really the only way to get these were to use aging outside dependencies (5+ years old) or R-script.

Just scratching an old itch I've been wanting since I added Histogram back in v11. I've been doing a bunch of benchmarking lately with the QUIC and DTLS impls and just got tired of having to use outside deps for this stuff.

/cc @mcollina

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/performance
  • @nodejs/tsc

@jasnell
jasnell requested review from Qard and mcollinaAugust 20, 2026 06:25
@nodejs-github-botnodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. labels Aug 20, 2026
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from f866877 to bc4d71eCompareAugust 20, 2026 13:24
@codecov

codecovBot commented Aug 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.44262% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.14%. Comparing base (cc57cb7) to head (fedf4f2).
⚠️ Report is 20 commits behind head on main.

Files with missing linesPatch %Lines
src/histogram.cc90.76%4 Missing and 30 partials ⚠️
src/histogram-inl.h86.36%3 Missing and 3 partials ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #65416 +/- ##
==========================================
+ Coverage 90.12% 90.14% +0.02% 
==========================================
Files 751 751 Lines 252536 253137 +601 Branches 47519 47631 +112 ==========================================
+ Hits 227598 228191 +593 + Misses 16250 16202 -48 - Partials 8688 8744 +56 
Files with missing linesCoverage Δ
lib/internal/histogram.js96.09% <100.00%> (+1.32%)⬆️
src/histogram.h68.42% <ø> (ø)
src/histogram-inl.h84.45% <86.36%> (+2.21%)⬆️
src/histogram.cc84.62% <90.76%> (+2.83%)⬆️

... and 35 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mcollinamcollina left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rslgtm

@nodejs-github-bot

This comment was marked as outdated.

@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from bc4d71e to 54c4c56CompareAugust 20, 2026 19:19
@nodejs-github-bot

This comment was marked as outdated.

@jasnell

Copy link
Copy Markdown
MemberAuthor

/cc @nodejs/performance

@jasnelljasnell added the author ready PRs with CI started, the required approvals, and no outstanding review comments. label Aug 20, 2026
Comment threadlib/internal/histogram.js Outdated
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from 54c4c56 to 0793579CompareAugust 20, 2026 20:35
@nodejs-github-bot

This comment was marked as outdated.

@jasnelljasnell added the commit-queue-rebase PRs the Commit Queue should land as multiple self-contained commits. label Aug 20, 2026
@jasnelljasnell added the notable-change PRs with changes that should be highlighted in changelogs. label Aug 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

The notable-changePRs with changes that should be highlighted in changelogs. label has been added by @jasnell.

Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section.

@jasnell

Copy link
Copy Markdown
MemberAuthor

Notes for the notable change:

This PR expands the analytical capabilities of Histogram further by adding EWMA, cliffsD, cohensD, SLO burn rate, mannWhitneyTest, and percentile confidence intervals to the API. Notably, this provides the ability to perform performance benchmark analysis directly within Node.js with no dependencies.

@jasnelljasnell added the semver-minor PRs that contain new features and should be released in the next minor version. label Aug 20, 2026
@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from 0793579 to fedf4f2CompareAugust 22, 2026 23:07
@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

jasnell added a commit that referenced this pull request Aug 23, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
jasnell added a commit that referenced this pull request Aug 23, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
@jasnell

Copy link
Copy Markdown
MemberAuthor

Landed in c28857e...bf67fdc

@jasnelljasnell closed this Aug 23, 2026
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
nodejs-github-bot added a commit that referenced this pull request Aug 25, 2026
Notable changes:
benchmark:
* (SEMVER-MINOR) add --analyze mode to compare.js (James M Snell) #65416
crypto:
* update root certificates to NSS 3.126 (Node.js GitHub Bot) #65495
* (SEMVER-MINOR) enable SIV and GCM-SIV modes in Cipher/Decipher APIs (Filip Skokan) #63411
diagnostics_channel:
* mark TracingChannel as stable (Abdelrahman Awad) #64525
lib,src:
* (SEMVER-MINOR) improve histogram implementation (James M Snell) #65024
net:
* (SEMVER-MINOR) improve performance of net.BlockList (James M Snell) #64974
perf_hooks:
* (SEMVER-MINOR) add statistical hypothesis testing to histogram (James M Snell) #65416
repl:
* add basic syntax highlighting (Aviv Keller) #64591
sqlite:
* (SEMVER-MINOR) add StatementSync.prototype.close() (Guilherme Araújo) #64232
* (SEMVER-MINOR) add StatementSync.prototype[Symbol.dispose]() (Guilherme Araújo) #64232
util:
* (SEMVER-MINOR) add non-throwing MIMEType.parse (James M Snell) #64965
zlib:
* (SEMVER-MINOR) add ZipEntry, ZipFile, and ZipBuffer (Philipp Dunkel) #64339
PR-URL: #65551
aduh95 pushed a commit that referenced this pull request Aug 26, 2026
Notable changes:
benchmark:
* (SEMVER-MINOR) add --analyze mode to compare.js (James M Snell) #65416
crypto:
* update root certificates to NSS 3.126 (Node.js GitHub Bot) #65495
* (SEMVER-MINOR) enable SIV and GCM-SIV modes in Cipher/Decipher APIs (Filip Skokan) #63411
diagnostics_channel:
* mark TracingChannel as stable (Abdelrahman Awad) #64525
lib,src:
* (SEMVER-MINOR) improve histogram implementation (James M Snell) #65024
net:
* (SEMVER-MINOR) improve performance of net.BlockList (James M Snell) #64974
perf_hooks:
* (SEMVER-MINOR) add statistical hypothesis testing to histogram (James M Snell) #65416
repl:
* add basic syntax highlighting (Aviv Keller) #64591
sqlite:
* (SEMVER-MINOR) add StatementSync.prototype.close() (Guilherme Araújo) #64232
* (SEMVER-MINOR) add StatementSync.prototype[Symbol.dispose]() (Guilherme Araújo) #64232
util:
* (SEMVER-MINOR) add non-throwing MIMEType.parse (James M Snell) #64965
zlib:
* (SEMVER-MINOR) add ZipEntry, ZipFile, and ZipBuffer (Philipp Dunkel) #64339
PR-URL: #65551
aduh95 pushed a commit that referenced this pull request Aug 27, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 27, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author readyPRs with CI started, the required approvals, and no outstanding review comments.c++Issues and PRs that require attention from people who are familiar with C++.commit-queue-rebasePRs the Commit Queue should land as multiple self-contained commits.needs-ciPRs that need a full CI run.notable-changePRs with changes that should be highlighted in changelogs.semver-minorPRs that contain new features and should be released in the next minor version.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@jasnell@nodejs-github-bot@mcollina@legendecas
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

perf_hooks: add statistical hypothesis testing to histogram - #65416

Closed
jasnell wants to merge 2 commits into
nodejs:mainfrom
jasnell:jasnell/more-histogram-fun
Closed

perf_hooks: add statistical hypothesis testing to histogram#65416
jasnell wants to merge 2 commits into
nodejs:mainfrom
jasnell:jasnell/more-histogram-fun

Conversation

@jasnell

@jasnelljasnell commented Aug 20, 2026

Copy link
Copy Markdown
Member

Continue expanding the capabilities of the Histogram API... and use it to enable a dependency-free benchmark/compare.js option... i.e. no longer requiring R-script to show benchmark analysis... just run benchmark/compare.js with the --analyze option.

The added algorithms cover analysis calculations that would be fairly common in perf analysis (latency, burn rates, detecting regressions, etc). Prior to this, really the only way to get these were to use aging outside dependencies (5+ years old) or R-script.

Just scratching an old itch I've been wanting since I added Histogram back in v11. I've been doing a bunch of benchmarking lately with the QUIC and DTLS impls and just got tired of having to use outside deps for this stuff.

/cc @mcollina

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/performance
  • @nodejs/tsc

@jasnell
jasnell requested review from Qard and mcollinaAugust 20, 2026 06:25
@nodejs-github-botnodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. labels Aug 20, 2026
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from f866877 to bc4d71eCompareAugust 20, 2026 13:24
@codecov

codecovBot commented Aug 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.44262% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.14%. Comparing base (cc57cb7) to head (fedf4f2).
⚠️ Report is 20 commits behind head on main.

Files with missing linesPatch %Lines
src/histogram.cc90.76%4 Missing and 30 partials ⚠️
src/histogram-inl.h86.36%3 Missing and 3 partials ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #65416 +/- ##
==========================================
+ Coverage 90.12% 90.14% +0.02% 
==========================================
Files 751 751 Lines 252536 253137 +601 Branches 47519 47631 +112 ==========================================
+ Hits 227598 228191 +593 + Misses 16250 16202 -48 - Partials 8688 8744 +56 
Files with missing linesCoverage Δ
lib/internal/histogram.js96.09% <100.00%> (+1.32%)⬆️
src/histogram.h68.42% <ø> (ø)
src/histogram-inl.h84.45% <86.36%> (+2.21%)⬆️
src/histogram.cc84.62% <90.76%> (+2.83%)⬆️

... and 35 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mcollinamcollina left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rslgtm

@nodejs-github-bot

This comment was marked as outdated.

@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from bc4d71e to 54c4c56CompareAugust 20, 2026 19:19
@nodejs-github-bot

This comment was marked as outdated.

@jasnell

Copy link
Copy Markdown
MemberAuthor

/cc @nodejs/performance

@jasnelljasnell added the author ready PRs with CI started, the required approvals, and no outstanding review comments. label Aug 20, 2026
Comment threadlib/internal/histogram.js Outdated
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from 54c4c56 to 0793579CompareAugust 20, 2026 20:35
@nodejs-github-bot

This comment was marked as outdated.

@jasnelljasnell added the commit-queue-rebase PRs the Commit Queue should land as multiple self-contained commits. label Aug 20, 2026
@jasnelljasnell added the notable-change PRs with changes that should be highlighted in changelogs. label Aug 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

The notable-changePRs with changes that should be highlighted in changelogs. label has been added by @jasnell.

Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section.

@jasnell

Copy link
Copy Markdown
MemberAuthor

Notes for the notable change:

This PR expands the analytical capabilities of Histogram further by adding EWMA, cliffsD, cohensD, SLO burn rate, mannWhitneyTest, and percentile confidence intervals to the API. Notably, this provides the ability to perform performance benchmark analysis directly within Node.js with no dependencies.

@jasnelljasnell added the semver-minor PRs that contain new features and should be released in the next minor version. label Aug 20, 2026
@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from 0793579 to fedf4f2CompareAugust 22, 2026 23:07
@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

jasnell added a commit that referenced this pull request Aug 23, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
jasnell added a commit that referenced this pull request Aug 23, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
@jasnell

Copy link
Copy Markdown
MemberAuthor

Landed in c28857e...bf67fdc

@jasnelljasnell closed this Aug 23, 2026
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
nodejs-github-bot added a commit that referenced this pull request Aug 25, 2026
Notable changes:
benchmark:
* (SEMVER-MINOR) add --analyze mode to compare.js (James M Snell) #65416
crypto:
* update root certificates to NSS 3.126 (Node.js GitHub Bot) #65495
* (SEMVER-MINOR) enable SIV and GCM-SIV modes in Cipher/Decipher APIs (Filip Skokan) #63411
diagnostics_channel:
* mark TracingChannel as stable (Abdelrahman Awad) #64525
lib,src:
* (SEMVER-MINOR) improve histogram implementation (James M Snell) #65024
net:
* (SEMVER-MINOR) improve performance of net.BlockList (James M Snell) #64974
perf_hooks:
* (SEMVER-MINOR) add statistical hypothesis testing to histogram (James M Snell) #65416
repl:
* add basic syntax highlighting (Aviv Keller) #64591
sqlite:
* (SEMVER-MINOR) add StatementSync.prototype.close() (Guilherme Araújo) #64232
* (SEMVER-MINOR) add StatementSync.prototype[Symbol.dispose]() (Guilherme Araújo) #64232
util:
* (SEMVER-MINOR) add non-throwing MIMEType.parse (James M Snell) #64965
zlib:
* (SEMVER-MINOR) add ZipEntry, ZipFile, and ZipBuffer (Philipp Dunkel) #64339
PR-URL: #65551
aduh95 pushed a commit that referenced this pull request Aug 26, 2026
Notable changes:
benchmark:
* (SEMVER-MINOR) add --analyze mode to compare.js (James M Snell) #65416
crypto:
* update root certificates to NSS 3.126 (Node.js GitHub Bot) #65495
* (SEMVER-MINOR) enable SIV and GCM-SIV modes in Cipher/Decipher APIs (Filip Skokan) #63411
diagnostics_channel:
* mark TracingChannel as stable (Abdelrahman Awad) #64525
lib,src:
* (SEMVER-MINOR) improve histogram implementation (James M Snell) #65024
net:
* (SEMVER-MINOR) improve performance of net.BlockList (James M Snell) #64974
perf_hooks:
* (SEMVER-MINOR) add statistical hypothesis testing to histogram (James M Snell) #65416
repl:
* add basic syntax highlighting (Aviv Keller) #64591
sqlite:
* (SEMVER-MINOR) add StatementSync.prototype.close() (Guilherme Araújo) #64232
* (SEMVER-MINOR) add StatementSync.prototype[Symbol.dispose]() (Guilherme Araújo) #64232
util:
* (SEMVER-MINOR) add non-throwing MIMEType.parse (James M Snell) #64965
zlib:
* (SEMVER-MINOR) add ZipEntry, ZipFile, and ZipBuffer (Philipp Dunkel) #64339
PR-URL: #65551
aduh95 pushed a commit that referenced this pull request Aug 27, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 27, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author readyPRs with CI started, the required approvals, and no outstanding review comments.c++Issues and PRs that require attention from people who are familiar with C++.commit-queue-rebasePRs the Commit Queue should land as multiple self-contained commits.needs-ciPRs that need a full CI run.notable-changePRs with changes that should be highlighted in changelogs.semver-minorPRs that contain new features and should be released in the next minor version.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@jasnell@nodejs-github-bot@mcollina@legendecas
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

perf_hooks: add statistical hypothesis testing to histogram - #65416

Closed
jasnell wants to merge 2 commits into
nodejs:mainfrom
jasnell:jasnell/more-histogram-fun
Closed

perf_hooks: add statistical hypothesis testing to histogram#65416
jasnell wants to merge 2 commits into
nodejs:mainfrom
jasnell:jasnell/more-histogram-fun

Conversation

@jasnell

@jasnelljasnell commented Aug 20, 2026

Copy link
Copy Markdown
Member

Continue expanding the capabilities of the Histogram API... and use it to enable a dependency-free benchmark/compare.js option... i.e. no longer requiring R-script to show benchmark analysis... just run benchmark/compare.js with the --analyze option.

The added algorithms cover analysis calculations that would be fairly common in perf analysis (latency, burn rates, detecting regressions, etc). Prior to this, really the only way to get these were to use aging outside dependencies (5+ years old) or R-script.

Just scratching an old itch I've been wanting since I added Histogram back in v11. I've been doing a bunch of benchmarking lately with the QUIC and DTLS impls and just got tired of having to use outside deps for this stuff.

/cc @mcollina

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/performance
  • @nodejs/tsc

@jasnell
jasnell requested review from Qard and mcollinaAugust 20, 2026 06:25
@nodejs-github-botnodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. labels Aug 20, 2026
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from f866877 to bc4d71eCompareAugust 20, 2026 13:24
@codecov

codecovBot commented Aug 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.44262% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.14%. Comparing base (cc57cb7) to head (fedf4f2).
⚠️ Report is 20 commits behind head on main.

Files with missing linesPatch %Lines
src/histogram.cc90.76%4 Missing and 30 partials ⚠️
src/histogram-inl.h86.36%3 Missing and 3 partials ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #65416 +/- ##
==========================================
+ Coverage 90.12% 90.14% +0.02% 
==========================================
Files 751 751 Lines 252536 253137 +601 Branches 47519 47631 +112 ==========================================
+ Hits 227598 228191 +593 + Misses 16250 16202 -48 - Partials 8688 8744 +56 
Files with missing linesCoverage Δ
lib/internal/histogram.js96.09% <100.00%> (+1.32%)⬆️
src/histogram.h68.42% <ø> (ø)
src/histogram-inl.h84.45% <86.36%> (+2.21%)⬆️
src/histogram.cc84.62% <90.76%> (+2.83%)⬆️

... and 35 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mcollinamcollina left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rslgtm

@nodejs-github-bot

This comment was marked as outdated.

@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from bc4d71e to 54c4c56CompareAugust 20, 2026 19:19
@nodejs-github-bot

This comment was marked as outdated.

@jasnell

Copy link
Copy Markdown
MemberAuthor

/cc @nodejs/performance

@jasnelljasnell added the author ready PRs with CI started, the required approvals, and no outstanding review comments. label Aug 20, 2026
Comment threadlib/internal/histogram.js Outdated
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from 54c4c56 to 0793579CompareAugust 20, 2026 20:35
@nodejs-github-bot

This comment was marked as outdated.

@jasnelljasnell added the commit-queue-rebase PRs the Commit Queue should land as multiple self-contained commits. label Aug 20, 2026
@jasnelljasnell added the notable-change PRs with changes that should be highlighted in changelogs. label Aug 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

The notable-changePRs with changes that should be highlighted in changelogs. label has been added by @jasnell.

Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section.

@jasnell

Copy link
Copy Markdown
MemberAuthor

Notes for the notable change:

This PR expands the analytical capabilities of Histogram further by adding EWMA, cliffsD, cohensD, SLO burn rate, mannWhitneyTest, and percentile confidence intervals to the API. Notably, this provides the ability to perform performance benchmark analysis directly within Node.js with no dependencies.

@jasnelljasnell added the semver-minor PRs that contain new features and should be released in the next minor version. label Aug 20, 2026
@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from 0793579 to fedf4f2CompareAugust 22, 2026 23:07
@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

jasnell added a commit that referenced this pull request Aug 23, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
jasnell added a commit that referenced this pull request Aug 23, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
@jasnell

Copy link
Copy Markdown
MemberAuthor

Landed in c28857e...bf67fdc

@jasnelljasnell closed this Aug 23, 2026
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
nodejs-github-bot added a commit that referenced this pull request Aug 25, 2026
Notable changes:
benchmark:
* (SEMVER-MINOR) add --analyze mode to compare.js (James M Snell) #65416
crypto:
* update root certificates to NSS 3.126 (Node.js GitHub Bot) #65495
* (SEMVER-MINOR) enable SIV and GCM-SIV modes in Cipher/Decipher APIs (Filip Skokan) #63411
diagnostics_channel:
* mark TracingChannel as stable (Abdelrahman Awad) #64525
lib,src:
* (SEMVER-MINOR) improve histogram implementation (James M Snell) #65024
net:
* (SEMVER-MINOR) improve performance of net.BlockList (James M Snell) #64974
perf_hooks:
* (SEMVER-MINOR) add statistical hypothesis testing to histogram (James M Snell) #65416
repl:
* add basic syntax highlighting (Aviv Keller) #64591
sqlite:
* (SEMVER-MINOR) add StatementSync.prototype.close() (Guilherme Araújo) #64232
* (SEMVER-MINOR) add StatementSync.prototype[Symbol.dispose]() (Guilherme Araújo) #64232
util:
* (SEMVER-MINOR) add non-throwing MIMEType.parse (James M Snell) #64965
zlib:
* (SEMVER-MINOR) add ZipEntry, ZipFile, and ZipBuffer (Philipp Dunkel) #64339
PR-URL: #65551
aduh95 pushed a commit that referenced this pull request Aug 26, 2026
Notable changes:
benchmark:
* (SEMVER-MINOR) add --analyze mode to compare.js (James M Snell) #65416
crypto:
* update root certificates to NSS 3.126 (Node.js GitHub Bot) #65495
* (SEMVER-MINOR) enable SIV and GCM-SIV modes in Cipher/Decipher APIs (Filip Skokan) #63411
diagnostics_channel:
* mark TracingChannel as stable (Abdelrahman Awad) #64525
lib,src:
* (SEMVER-MINOR) improve histogram implementation (James M Snell) #65024
net:
* (SEMVER-MINOR) improve performance of net.BlockList (James M Snell) #64974
perf_hooks:
* (SEMVER-MINOR) add statistical hypothesis testing to histogram (James M Snell) #65416
repl:
* add basic syntax highlighting (Aviv Keller) #64591
sqlite:
* (SEMVER-MINOR) add StatementSync.prototype.close() (Guilherme Araújo) #64232
* (SEMVER-MINOR) add StatementSync.prototype[Symbol.dispose]() (Guilherme Araújo) #64232
util:
* (SEMVER-MINOR) add non-throwing MIMEType.parse (James M Snell) #64965
zlib:
* (SEMVER-MINOR) add ZipEntry, ZipFile, and ZipBuffer (Philipp Dunkel) #64339
PR-URL: #65551
aduh95 pushed a commit that referenced this pull request Aug 27, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 27, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author readyPRs with CI started, the required approvals, and no outstanding review comments.c++Issues and PRs that require attention from people who are familiar with C++.commit-queue-rebasePRs the Commit Queue should land as multiple self-contained commits.needs-ciPRs that need a full CI run.notable-changePRs with changes that should be highlighted in changelogs.semver-minorPRs that contain new features and should be released in the next minor version.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@jasnell@nodejs-github-bot@mcollina@legendecas
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

perf_hooks: add statistical hypothesis testing to histogram - #65416

Closed
jasnell wants to merge 2 commits into
nodejs:mainfrom
jasnell:jasnell/more-histogram-fun
Closed

perf_hooks: add statistical hypothesis testing to histogram#65416
jasnell wants to merge 2 commits into
nodejs:mainfrom
jasnell:jasnell/more-histogram-fun

Conversation

@jasnell

@jasnelljasnell commented Aug 20, 2026

Copy link
Copy Markdown
Member

Continue expanding the capabilities of the Histogram API... and use it to enable a dependency-free benchmark/compare.js option... i.e. no longer requiring R-script to show benchmark analysis... just run benchmark/compare.js with the --analyze option.

The added algorithms cover analysis calculations that would be fairly common in perf analysis (latency, burn rates, detecting regressions, etc). Prior to this, really the only way to get these were to use aging outside dependencies (5+ years old) or R-script.

Just scratching an old itch I've been wanting since I added Histogram back in v11. I've been doing a bunch of benchmarking lately with the QUIC and DTLS impls and just got tired of having to use outside deps for this stuff.

/cc @mcollina

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/performance
  • @nodejs/tsc

@jasnell
jasnell requested review from Qard and mcollinaAugust 20, 2026 06:25
@nodejs-github-botnodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. labels Aug 20, 2026
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from f866877 to bc4d71eCompareAugust 20, 2026 13:24
@codecov

codecovBot commented Aug 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.44262% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.14%. Comparing base (cc57cb7) to head (fedf4f2).
⚠️ Report is 20 commits behind head on main.

Files with missing linesPatch %Lines
src/histogram.cc90.76%4 Missing and 30 partials ⚠️
src/histogram-inl.h86.36%3 Missing and 3 partials ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #65416 +/- ##
==========================================
+ Coverage 90.12% 90.14% +0.02% 
==========================================
Files 751 751 Lines 252536 253137 +601 Branches 47519 47631 +112 ==========================================
+ Hits 227598 228191 +593 + Misses 16250 16202 -48 - Partials 8688 8744 +56 
Files with missing linesCoverage Δ
lib/internal/histogram.js96.09% <100.00%> (+1.32%)⬆️
src/histogram.h68.42% <ø> (ø)
src/histogram-inl.h84.45% <86.36%> (+2.21%)⬆️
src/histogram.cc84.62% <90.76%> (+2.83%)⬆️

... and 35 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mcollinamcollina left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rslgtm

@nodejs-github-bot

This comment was marked as outdated.

@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from bc4d71e to 54c4c56CompareAugust 20, 2026 19:19
@nodejs-github-bot

This comment was marked as outdated.

@jasnell

Copy link
Copy Markdown
MemberAuthor

/cc @nodejs/performance

@jasnelljasnell added the author ready PRs with CI started, the required approvals, and no outstanding review comments. label Aug 20, 2026
Comment threadlib/internal/histogram.js Outdated
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from 54c4c56 to 0793579CompareAugust 20, 2026 20:35
@nodejs-github-bot

This comment was marked as outdated.

@jasnelljasnell added the commit-queue-rebase PRs the Commit Queue should land as multiple self-contained commits. label Aug 20, 2026
@jasnelljasnell added the notable-change PRs with changes that should be highlighted in changelogs. label Aug 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

The notable-changePRs with changes that should be highlighted in changelogs. label has been added by @jasnell.

Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section.

@jasnell

Copy link
Copy Markdown
MemberAuthor

Notes for the notable change:

This PR expands the analytical capabilities of Histogram further by adding EWMA, cliffsD, cohensD, SLO burn rate, mannWhitneyTest, and percentile confidence intervals to the API. Notably, this provides the ability to perform performance benchmark analysis directly within Node.js with no dependencies.

@jasnelljasnell added the semver-minor PRs that contain new features and should be released in the next minor version. label Aug 20, 2026
@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from 0793579 to fedf4f2CompareAugust 22, 2026 23:07
@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

jasnell added a commit that referenced this pull request Aug 23, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
jasnell added a commit that referenced this pull request Aug 23, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
@jasnell

Copy link
Copy Markdown
MemberAuthor

Landed in c28857e...bf67fdc

@jasnelljasnell closed this Aug 23, 2026
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
nodejs-github-bot added a commit that referenced this pull request Aug 25, 2026
Notable changes:
benchmark:
* (SEMVER-MINOR) add --analyze mode to compare.js (James M Snell) #65416
crypto:
* update root certificates to NSS 3.126 (Node.js GitHub Bot) #65495
* (SEMVER-MINOR) enable SIV and GCM-SIV modes in Cipher/Decipher APIs (Filip Skokan) #63411
diagnostics_channel:
* mark TracingChannel as stable (Abdelrahman Awad) #64525
lib,src:
* (SEMVER-MINOR) improve histogram implementation (James M Snell) #65024
net:
* (SEMVER-MINOR) improve performance of net.BlockList (James M Snell) #64974
perf_hooks:
* (SEMVER-MINOR) add statistical hypothesis testing to histogram (James M Snell) #65416
repl:
* add basic syntax highlighting (Aviv Keller) #64591
sqlite:
* (SEMVER-MINOR) add StatementSync.prototype.close() (Guilherme Araújo) #64232
* (SEMVER-MINOR) add StatementSync.prototype[Symbol.dispose]() (Guilherme Araújo) #64232
util:
* (SEMVER-MINOR) add non-throwing MIMEType.parse (James M Snell) #64965
zlib:
* (SEMVER-MINOR) add ZipEntry, ZipFile, and ZipBuffer (Philipp Dunkel) #64339
PR-URL: #65551
aduh95 pushed a commit that referenced this pull request Aug 26, 2026
Notable changes:
benchmark:
* (SEMVER-MINOR) add --analyze mode to compare.js (James M Snell) #65416
crypto:
* update root certificates to NSS 3.126 (Node.js GitHub Bot) #65495
* (SEMVER-MINOR) enable SIV and GCM-SIV modes in Cipher/Decipher APIs (Filip Skokan) #63411
diagnostics_channel:
* mark TracingChannel as stable (Abdelrahman Awad) #64525
lib,src:
* (SEMVER-MINOR) improve histogram implementation (James M Snell) #65024
net:
* (SEMVER-MINOR) improve performance of net.BlockList (James M Snell) #64974
perf_hooks:
* (SEMVER-MINOR) add statistical hypothesis testing to histogram (James M Snell) #65416
repl:
* add basic syntax highlighting (Aviv Keller) #64591
sqlite:
* (SEMVER-MINOR) add StatementSync.prototype.close() (Guilherme Araújo) #64232
* (SEMVER-MINOR) add StatementSync.prototype[Symbol.dispose]() (Guilherme Araújo) #64232
util:
* (SEMVER-MINOR) add non-throwing MIMEType.parse (James M Snell) #64965
zlib:
* (SEMVER-MINOR) add ZipEntry, ZipFile, and ZipBuffer (Philipp Dunkel) #64339
PR-URL: #65551
aduh95 pushed a commit that referenced this pull request Aug 27, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 27, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author readyPRs with CI started, the required approvals, and no outstanding review comments.c++Issues and PRs that require attention from people who are familiar with C++.commit-queue-rebasePRs the Commit Queue should land as multiple self-contained commits.needs-ciPRs that need a full CI run.notable-changePRs with changes that should be highlighted in changelogs.semver-minorPRs that contain new features and should be released in the next minor version.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@jasnell@nodejs-github-bot@mcollina@legendecas
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

perf_hooks: add statistical hypothesis testing to histogram - #65416

Closed
jasnell wants to merge 2 commits into
nodejs:mainfrom
jasnell:jasnell/more-histogram-fun
Closed

perf_hooks: add statistical hypothesis testing to histogram#65416
jasnell wants to merge 2 commits into
nodejs:mainfrom
jasnell:jasnell/more-histogram-fun

Conversation

@jasnell

@jasnelljasnell commented Aug 20, 2026

Copy link
Copy Markdown
Member

Continue expanding the capabilities of the Histogram API... and use it to enable a dependency-free benchmark/compare.js option... i.e. no longer requiring R-script to show benchmark analysis... just run benchmark/compare.js with the --analyze option.

The added algorithms cover analysis calculations that would be fairly common in perf analysis (latency, burn rates, detecting regressions, etc). Prior to this, really the only way to get these were to use aging outside dependencies (5+ years old) or R-script.

Just scratching an old itch I've been wanting since I added Histogram back in v11. I've been doing a bunch of benchmarking lately with the QUIC and DTLS impls and just got tired of having to use outside deps for this stuff.

/cc @mcollina

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/performance
  • @nodejs/tsc

@jasnell
jasnell requested review from Qard and mcollinaAugust 20, 2026 06:25
@nodejs-github-botnodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. labels Aug 20, 2026
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from f866877 to bc4d71eCompareAugust 20, 2026 13:24
@codecov

codecovBot commented Aug 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.44262% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.14%. Comparing base (cc57cb7) to head (fedf4f2).
⚠️ Report is 20 commits behind head on main.

Files with missing linesPatch %Lines
src/histogram.cc90.76%4 Missing and 30 partials ⚠️
src/histogram-inl.h86.36%3 Missing and 3 partials ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #65416 +/- ##
==========================================
+ Coverage 90.12% 90.14% +0.02% 
==========================================
Files 751 751 Lines 252536 253137 +601 Branches 47519 47631 +112 ==========================================
+ Hits 227598 228191 +593 + Misses 16250 16202 -48 - Partials 8688 8744 +56 
Files with missing linesCoverage Δ
lib/internal/histogram.js96.09% <100.00%> (+1.32%)⬆️
src/histogram.h68.42% <ø> (ø)
src/histogram-inl.h84.45% <86.36%> (+2.21%)⬆️
src/histogram.cc84.62% <90.76%> (+2.83%)⬆️

... and 35 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mcollinamcollina left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rslgtm

@nodejs-github-bot

This comment was marked as outdated.

@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from bc4d71e to 54c4c56CompareAugust 20, 2026 19:19
@nodejs-github-bot

This comment was marked as outdated.

@jasnell

Copy link
Copy Markdown
MemberAuthor

/cc @nodejs/performance

@jasnelljasnell added the author ready PRs with CI started, the required approvals, and no outstanding review comments. label Aug 20, 2026
Comment threadlib/internal/histogram.js Outdated
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from 54c4c56 to 0793579CompareAugust 20, 2026 20:35
@nodejs-github-bot

This comment was marked as outdated.

@jasnelljasnell added the commit-queue-rebase PRs the Commit Queue should land as multiple self-contained commits. label Aug 20, 2026
@jasnelljasnell added the notable-change PRs with changes that should be highlighted in changelogs. label Aug 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

The notable-changePRs with changes that should be highlighted in changelogs. label has been added by @jasnell.

Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section.

@jasnell

Copy link
Copy Markdown
MemberAuthor

Notes for the notable change:

This PR expands the analytical capabilities of Histogram further by adding EWMA, cliffsD, cohensD, SLO burn rate, mannWhitneyTest, and percentile confidence intervals to the API. Notably, this provides the ability to perform performance benchmark analysis directly within Node.js with no dependencies.

@jasnelljasnell added the semver-minor PRs that contain new features and should be released in the next minor version. label Aug 20, 2026
@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
@jasnell
jasnellforce-pushed the jasnell/more-histogram-fun branch from 0793579 to fedf4f2CompareAugust 22, 2026 23:07
@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

jasnell added a commit that referenced this pull request Aug 23, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
jasnell added a commit that referenced this pull request Aug 23, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
@jasnell

Copy link
Copy Markdown
MemberAuthor

Landed in c28857e...bf67fdc

@jasnelljasnell closed this Aug 23, 2026
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 25, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
nodejs-github-bot added a commit that referenced this pull request Aug 25, 2026
Notable changes:
benchmark:
* (SEMVER-MINOR) add --analyze mode to compare.js (James M Snell) #65416
crypto:
* update root certificates to NSS 3.126 (Node.js GitHub Bot) #65495
* (SEMVER-MINOR) enable SIV and GCM-SIV modes in Cipher/Decipher APIs (Filip Skokan) #63411
diagnostics_channel:
* mark TracingChannel as stable (Abdelrahman Awad) #64525
lib,src:
* (SEMVER-MINOR) improve histogram implementation (James M Snell) #65024
net:
* (SEMVER-MINOR) improve performance of net.BlockList (James M Snell) #64974
perf_hooks:
* (SEMVER-MINOR) add statistical hypothesis testing to histogram (James M Snell) #65416
repl:
* add basic syntax highlighting (Aviv Keller) #64591
sqlite:
* (SEMVER-MINOR) add StatementSync.prototype.close() (Guilherme Araújo) #64232
* (SEMVER-MINOR) add StatementSync.prototype[Symbol.dispose]() (Guilherme Araújo) #64232
util:
* (SEMVER-MINOR) add non-throwing MIMEType.parse (James M Snell) #64965
zlib:
* (SEMVER-MINOR) add ZipEntry, ZipFile, and ZipBuffer (Philipp Dunkel) #64339
PR-URL: #65551
aduh95 pushed a commit that referenced this pull request Aug 26, 2026
Notable changes:
benchmark:
* (SEMVER-MINOR) add --analyze mode to compare.js (James M Snell) #65416
crypto:
* update root certificates to NSS 3.126 (Node.js GitHub Bot) #65495
* (SEMVER-MINOR) enable SIV and GCM-SIV modes in Cipher/Decipher APIs (Filip Skokan) #63411
diagnostics_channel:
* mark TracingChannel as stable (Abdelrahman Awad) #64525
lib,src:
* (SEMVER-MINOR) improve histogram implementation (James M Snell) #65024
net:
* (SEMVER-MINOR) improve performance of net.BlockList (James M Snell) #64974
perf_hooks:
* (SEMVER-MINOR) add statistical hypothesis testing to histogram (James M Snell) #65416
repl:
* add basic syntax highlighting (Aviv Keller) #64591
sqlite:
* (SEMVER-MINOR) add StatementSync.prototype.close() (Guilherme Araújo) #64232
* (SEMVER-MINOR) add StatementSync.prototype[Symbol.dispose]() (Guilherme Araújo) #64232
util:
* (SEMVER-MINOR) add non-throwing MIMEType.parse (James M Snell) #64965
zlib:
* (SEMVER-MINOR) add ZipEntry, ZipFile, and ZipBuffer (Philipp Dunkel) #64339
PR-URL: #65551
aduh95 pushed a commit that referenced this pull request Aug 27, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
aduh95 pushed a commit that referenced this pull request Aug 27, 2026
Add an --analyze flag that performs statistical analysis directly
after benchmarks complete, eliminating the need for R and compare.R.
When --analyze is specified, compare.js collects the rate data during
the run and prints a statistical summary table instead of CSV output.
The table matches the format of compare.R: improvement percentage,
significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence
intervals at three risk levels.
Also adds a --max-regression N option that causes the compare.js to
exit with 1 (error) when the `--new` is N% slower. Useful for CI
use to detect regressions.
Uses the histogram API's welchTest() and cohensD() methods introduced
in the previous commit. Benchmark rates are scaled to integers for
HdrHistogram recording; the --scale option (default 1000) controls
the multiplier for precision.
Usage:
node benchmark/compare.js --old ./node-old --new ./node-new \
--analyze url
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author readyPRs with CI started, the required approvals, and no outstanding review comments.c++Issues and PRs that require attention from people who are familiar with C++.commit-queue-rebasePRs the Commit Queue should land as multiple self-contained commits.needs-ciPRs that need a full CI run.notable-changePRs with changes that should be highlighted in changelogs.semver-minorPRs that contain new features and should be released in the next minor version.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@jasnell@nodejs-github-bot@mcollina@legendecas