Uh oh!
There was an error while loading. Please reload this page.
perf_hooks: add statistical hypothesis testing to histogram - #65416
perf_hooks: add statistical hypothesis testing to histogram#65416jasnell wants to merge 2 commits into
Conversation
nodejs-github-bot
commented
Aug 20, 2026
Review requested:
|
f866877 to
bc4d71eCompareCodecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@## main #65416 +/- ##
==========================================
+ Coverage 90.12% 90.14% +0.02%
==========================================
Files 751 751 Lines 252536 253137 +601 Branches 47519 47631 +112 ==========================================
+ Hits 227598 228191 +593 + Misses 16250 16202 -48 - Partials 8688 8744 +56
🚀 New features to boost your workflow:
|
This comment was marked as outdated.
This comment was marked as outdated.
bc4d71e to
54c4c56Compare
This comment was marked as outdated.
This comment was marked as outdated.
jasnell
commented
Aug 20, 2026
/cc @nodejs/performance |
Uh oh!
There was an error while loading. Please reload this page.
54c4c56 to
0793579Compare
This comment was marked as outdated.
This comment was marked as outdated.
The
notable-change Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section. |
jasnell
commented
Aug 20, 2026
Notes for the notable change: This PR expands the analytical capabilities of |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta, and and handful of others These methods enable in-process benchmark comparison and regression detection without external dependencies. No new dependencies. Tests and docs created by the AI agent. Signed-off-by: James M Snell <jasnell@gmail.com> Assisted-by: Opencode/Opus
Add an --analyze flag that performs statistical analysis directly after benchmarks complete, eliminating the need for R and compare.R. When --analyze is specified, compare.js collects the rate data during the run and prints a statistical summary table instead of CSV output. The table matches the format of compare.R: improvement percentage, significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence intervals at three risk levels. Also adds a --max-regression N option that causes the compare.js to exit with 1 (error) when the `--new` is N% slower. Useful for CI use to detect regressions. Uses the histogram API's welchTest() and cohensD() methods introduced in the previous commit. Benchmark rates are scaled to integers for HdrHistogram recording; the --scale option (default 1000) controls the multiplier for precision. Usage: node benchmark/compare.js --old ./node-old --new ./node-new \ --analyze url Signed-off-by: James M Snell <jasnell@gmail.com> Assisted-by: Opencode/Opus
0793579 to
fedf4f2Compare
This comment was marked as outdated.
This comment was marked as outdated.
nodejs-github-bot
commented
Aug 23, 2026
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta, and and handful of others These methods enable in-process benchmark comparison and regression detection without external dependencies. No new dependencies. Tests and docs created by the AI agent. Signed-off-by: James M Snell <jasnell@gmail.com> Assisted-by: Opencode/Opus PR-URL: #65416 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Add an --analyze flag that performs statistical analysis directly after benchmarks complete, eliminating the need for R and compare.R. When --analyze is specified, compare.js collects the rate data during the run and prints a statistical summary table instead of CSV output. The table matches the format of compare.R: improvement percentage, significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence intervals at three risk levels. Also adds a --max-regression N option that causes the compare.js to exit with 1 (error) when the `--new` is N% slower. Useful for CI use to detect regressions. Uses the histogram API's welchTest() and cohensD() methods introduced in the previous commit. Benchmark rates are scaled to integers for HdrHistogram recording; the --scale option (default 1000) controls the multiplier for precision. Usage: node benchmark/compare.js --old ./node-old --new ./node-new \ --analyze url Signed-off-by: James M Snell <jasnell@gmail.com> Assisted-by: Opencode/Opus PR-URL: #65416 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
jasnell
commented
Aug 23, 2026
Landed in c28857e...bf67fdc |
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta, and and handful of others These methods enable in-process benchmark comparison and regression detection without external dependencies. No new dependencies. Tests and docs created by the AI agent. Signed-off-by: James M Snell <jasnell@gmail.com> Assisted-by: Opencode/Opus PR-URL: #65416 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Add an --analyze flag that performs statistical analysis directly after benchmarks complete, eliminating the need for R and compare.R. When --analyze is specified, compare.js collects the rate data during the run and prints a statistical summary table instead of CSV output. The table matches the format of compare.R: improvement percentage, significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence intervals at three risk levels. Also adds a --max-regression N option that causes the compare.js to exit with 1 (error) when the `--new` is N% slower. Useful for CI use to detect regressions. Uses the histogram API's welchTest() and cohensD() methods introduced in the previous commit. Benchmark rates are scaled to integers for HdrHistogram recording; the --scale option (default 1000) controls the multiplier for precision. Usage: node benchmark/compare.js --old ./node-old --new ./node-new \ --analyze url Signed-off-by: James M Snell <jasnell@gmail.com> Assisted-by: Opencode/Opus PR-URL: #65416 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta, and and handful of others These methods enable in-process benchmark comparison and regression detection without external dependencies. No new dependencies. Tests and docs created by the AI agent. Signed-off-by: James M Snell <jasnell@gmail.com> Assisted-by: Opencode/Opus PR-URL: #65416 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Add an --analyze flag that performs statistical analysis directly after benchmarks complete, eliminating the need for R and compare.R. When --analyze is specified, compare.js collects the rate data during the run and prints a statistical summary table instead of CSV output. The table matches the format of compare.R: improvement percentage, significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence intervals at three risk levels. Also adds a --max-regression N option that causes the compare.js to exit with 1 (error) when the `--new` is N% slower. Useful for CI use to detect regressions. Uses the histogram API's welchTest() and cohensD() methods introduced in the previous commit. Benchmark rates are scaled to integers for HdrHistogram recording; the --scale option (default 1000) controls the multiplier for precision. Usage: node benchmark/compare.js --old ./node-old --new ./node-new \ --analyze url Signed-off-by: James M Snell <jasnell@gmail.com> Assisted-by: Opencode/Opus PR-URL: #65416 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Notable changes: benchmark: * (SEMVER-MINOR) add --analyze mode to compare.js (James M Snell) #65416 crypto: * update root certificates to NSS 3.126 (Node.js GitHub Bot) #65495 * (SEMVER-MINOR) enable SIV and GCM-SIV modes in Cipher/Decipher APIs (Filip Skokan) #63411 diagnostics_channel: * mark TracingChannel as stable (Abdelrahman Awad) #64525 lib,src: * (SEMVER-MINOR) improve histogram implementation (James M Snell) #65024 net: * (SEMVER-MINOR) improve performance of net.BlockList (James M Snell) #64974 perf_hooks: * (SEMVER-MINOR) add statistical hypothesis testing to histogram (James M Snell) #65416 repl: * add basic syntax highlighting (Aviv Keller) #64591 sqlite: * (SEMVER-MINOR) add StatementSync.prototype.close() (Guilherme Araújo) #64232 * (SEMVER-MINOR) add StatementSync.prototype[Symbol.dispose]() (Guilherme Araújo) #64232 util: * (SEMVER-MINOR) add non-throwing MIMEType.parse (James M Snell) #64965 zlib: * (SEMVER-MINOR) add ZipEntry, ZipFile, and ZipBuffer (Philipp Dunkel) #64339 PR-URL: #65551
Notable changes: benchmark: * (SEMVER-MINOR) add --analyze mode to compare.js (James M Snell) #65416 crypto: * update root certificates to NSS 3.126 (Node.js GitHub Bot) #65495 * (SEMVER-MINOR) enable SIV and GCM-SIV modes in Cipher/Decipher APIs (Filip Skokan) #63411 diagnostics_channel: * mark TracingChannel as stable (Abdelrahman Awad) #64525 lib,src: * (SEMVER-MINOR) improve histogram implementation (James M Snell) #65024 net: * (SEMVER-MINOR) improve performance of net.BlockList (James M Snell) #64974 perf_hooks: * (SEMVER-MINOR) add statistical hypothesis testing to histogram (James M Snell) #65416 repl: * add basic syntax highlighting (Aviv Keller) #64591 sqlite: * (SEMVER-MINOR) add StatementSync.prototype.close() (Guilherme Araújo) #64232 * (SEMVER-MINOR) add StatementSync.prototype[Symbol.dispose]() (Guilherme Araújo) #64232 util: * (SEMVER-MINOR) add non-throwing MIMEType.parse (James M Snell) #64965 zlib: * (SEMVER-MINOR) add ZipEntry, ZipFile, and ZipBuffer (Philipp Dunkel) #64339 PR-URL: #65551
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta, and and handful of others These methods enable in-process benchmark comparison and regression detection without external dependencies. No new dependencies. Tests and docs created by the AI agent. Signed-off-by: James M Snell <jasnell@gmail.com> Assisted-by: Opencode/Opus PR-URL: #65416 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Add an --analyze flag that performs statistical analysis directly after benchmarks complete, eliminating the need for R and compare.R. When --analyze is specified, compare.js collects the rate data during the run and prints a statistical summary table instead of CSV output. The table matches the format of compare.R: improvement percentage, significance stars (* p<0.05, ** p<0.01, *** p<0.001), and confidence intervals at three risk levels. Also adds a --max-regression N option that causes the compare.js to exit with 1 (error) when the `--new` is N% slower. Useful for CI use to detect regressions. Uses the histogram API's welchTest() and cohensD() methods introduced in the previous commit. Benchmark rates are scaled to integers for HdrHistogram recording; the --scale option (default 1000) controls the multiplier for precision. Usage: node benchmark/compare.js --old ./node-old --new ./node-new \ --analyze url Signed-off-by: James M Snell <jasnell@gmail.com> Assisted-by: Opencode/Opus PR-URL: #65416 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Continue expanding the capabilities of the
HistogramAPI... and use it to enable a dependency-freebenchmark/compare.jsoption... i.e. no longer requiring R-script to show benchmark analysis... just runbenchmark/compare.jswith the--analyzeoption.The added algorithms cover analysis calculations that would be fairly common in perf analysis (latency, burn rates, detecting regressions, etc). Prior to this, really the only way to get these were to use aging outside dependencies (5+ years old) or R-script.
Just scratching an old itch I've been wanting since I added
Histogramback in v11. I've been doing a bunch of benchmarking lately with the QUIC and DTLS impls and just got tired of having to use outside deps for this stuff./cc @mcollina