Commit 9e8e908

Browse files
jasnelladuh95
jasnell
authored and
aduh95
committed
perf_hooks: add statistical hypothesis testing to histogram
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta, and and handful of others These methods enable in-process benchmark comparison and regression detection without external dependencies. No new dependencies. Tests and docs created by the AI agent. Signed-off-by: James M Snell <jasnell@gmail.com> Assisted-by: Opencode/Opus PR-URL: #65416 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
1 parent dc51c79 commit 9e8e908

6 files changed

Lines changed: 1647 additions & 10 deletions

File tree

‎doc/api/perf_hooks.md‎

Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1625,6 +1625,16 @@ added:
16251625
**Default:**`Number.MAX_SAFE_INTEGER`.
16261626
*`figures` {number} The number of accuracy digits. Must be a number between
16271627
`1` and `5`. **Default:**`3`.
1628+
*`halfLife` {number} The EWMA half-life in number of samples. When set to
1629+
a value greater than 0, the histogram tracks an exponentially weighted
1630+
moving average and standard deviation, accessible via
1631+
`histogram.ewmaMean` and `histogram.ewmaStddev`. After `halfLife`
1632+
recordings, a value's influence has decayed to 50%. **Default:**`0`
1633+
(disabled).
1634+
*`threshold` {number} An SLO threshold value. When set together with
1635+
`halfLife`, the histogram tracks a smoothed error rate for values
1636+
exceeding this threshold, accessible via `histogram.ewmaErrorRate` and
1637+
`histogram.burnRate()`. **Default:**`0` (disabled).
16281638
* Returns: {RecordableHistogram}
16291639

16301640
Returns a {RecordableHistogram}.
@@ -1893,6 +1903,36 @@ value, representing the probability that a recorded value will be less
18931903
than or equal to `value`. This is the inverse operation of
18941904
`histogram.percentile()`.
18951905

1906+
### `histogram.cliffsD(other)`
1907+
1908+
<!-- YAML
1909+
added: REPLACEME
1910+
-->
1911+
1912+
*`other` {Histogram} The histogram to compare against.
1913+
* Returns: {number} A value between -1.0 and 1.0.
1914+
1915+
Computes [Cliff's delta][], a non-parametric effect size measure. Returns
1916+
the probability that a random value from this histogram exceeds a random
1917+
value from `other`, minus the reverse probability. A value of 1 means every
1918+
value in this histogram exceeds every value in `other`; -1 means the
1919+
opposite; 0 means no tendency in either direction.
1920+
1921+
### `histogram.cohensD(other)`
1922+
1923+
<!-- YAML
1924+
added: REPLACEME
1925+
-->
1926+
1927+
*`other` {Histogram} The histogram to compare against.
1928+
* Returns: {number} The effect size.
1929+
1930+
Computes [Cohen's d][] effect size, the standardized difference between the
1931+
means of this histogram and `other`, using the pooled standard deviation.
1932+
Positive values indicate this histogram has a higher mean. By convention,
1933+
|d| < 0.2 is a small effect, 0.5 is medium, and 0.8 or greater is large.
1934+
Both histograms must have at least 2 recorded values; otherwise returns 0.
1935+
18961936
### `histogram.countAt(value)`
18971937

18981938
<!-- YAML
@@ -1929,6 +1969,74 @@ added:
19291969
The number of times the event loop delay exceeded the maximum 1 hour event
19301970
loop delay threshold.
19311971

1972+
### `histogram.ewmaMean`
1973+
1974+
<!-- YAML
1975+
added: REPLACEME
1976+
-->
1977+
1978+
* Type: {number}
1979+
1980+
The exponentially weighted moving average of recorded values. Only active
1981+
when the histogram was created with a `halfLife` option greater than 0.
1982+
Returns `0` when EWMA is disabled or no values have been recorded.
1983+
1984+
### `histogram.ewmaStddev`
1985+
1986+
<!-- YAML
1987+
added: REPLACEME
1988+
-->
1989+
1990+
* Type: {number}
1991+
1992+
The exponentially weighted moving standard deviation. Only active when the
1993+
histogram was created with a `halfLife` option greater than 0. Returns `0`
1994+
when EWMA is disabled or no values have been recorded.
1995+
1996+
### `histogram.ewmaErrorRate`
1997+
1998+
<!-- YAML
1999+
added: REPLACEME
2000+
-->
2001+
2002+
* Type: {number}
2003+
2004+
The EWMA-smoothed probability of a recorded value exceeding the configured
2005+
`threshold`. Only active when the histogram was created with both `halfLife`
2006+
and `threshold` options. Returns `0` when not enabled or no values have been
2007+
recorded.
2008+
2009+
### `histogram.burnRate(sloTarget)`
2010+
2011+
<!-- YAML
2012+
added: REPLACEME
2013+
-->
2014+
2015+
*`sloTarget` {number} The SLO target as a fraction between 0 and 1
2016+
(exclusive). For example, `0.999` for a 99.9% SLO.
2017+
* Returns: {number}
2018+
2019+
Returns the SLO burn rate: `ewmaErrorRate / (1 - sloTarget)`. A burn rate
2020+
of 1 means the error budget will be exactly exhausted over the SLO window.
2021+
A burn rate greater than 1 means it is being consumed faster than allowed.
2022+
Requires the histogram to have been created with both `halfLife` and
2023+
`threshold` options.
2024+
2025+
```js
2026+
const { createHistogram } =require('node:perf_hooks');
2027+
2028+
// Track latency with a 200ms SLO threshold, half-life of 100 samples
2029+
consth=createHistogram({ halfLife:100, threshold:200_000_000 });
2030+
2031+
// ... record latency values ...
2032+
2033+
// Check burn rate against a 99.9% SLO
2034+
constrate=h.burnRate(0.999);
2035+
if (rate >1) {
2036+
console.log(`SLO burn rate: ${rate.toFixed(2)}x — error budget depleting`);
2037+
}
2038+
```
2039+
19322040
### `histogram.ksTest(other)`
19332041

19342042
<!-- YAML
@@ -1982,6 +2090,24 @@ Returns the histogram data rebucketed into logarithmically-spaced
19822090
intervals, where each bucket's width is multiplied by `base`.
19832091
Useful for visualization and export.
19842092

2093+
### `histogram.mannWhitneyTest(other)`
2094+
2095+
<!-- YAML
2096+
added: REPLACEME
2097+
-->
2098+
2099+
*`other` {Histogram} The histogram to compare against.
2100+
* Returns: {Object}
2101+
*`uStatistic` {number} The Mann-Whitney U statistic.
2102+
*`zScore` {number} The z-score (normal approximation).
2103+
*`pValue` {number} Two-tailed p-value.
2104+
2105+
Performs a [Mann-Whitney U test][] comparing whether this histogram tends to
2106+
produce larger or smaller values than `other`. Unlike `welchTest()`, this is a
2107+
non-parametric test that makes no assumptions about the shape of the
2108+
distributions. Uses the normal approximation with tie correction for the
2109+
p-value.
2110+
19852111
### `histogram.max`
19862112

19872113
<!-- YAML
@@ -2060,6 +2186,40 @@ added:
20602186

20612187
Returns the value at the given percentile.
20622188

2189+
### `histogram.percentileCI(percentile[, options])`
2190+
2191+
<!-- YAML
2192+
added: REPLACEME
2193+
-->
2194+
2195+
*`percentile` {number} A percentile value in the range (0, 100].
2196+
*`options` {Object}
2197+
*`confidence` {number} The confidence level for the interval, between
2198+
0 and 1 (exclusive). **Default:**`0.95`.
2199+
* Returns: {Object}
2200+
*`value` {number} The point estimate (same as `histogram.percentile()`).
2201+
*`lower` {number} The lower bound of the confidence interval.
2202+
*`upper` {number} The upper bound of the confidence interval.
2203+
2204+
Returns a confidence interval for the given percentile using the exact
2205+
binomial method. With fewer samples, the interval will be wider, reflecting
2206+
the greater uncertainty in the percentile estimate. Requires at least 2
2207+
recorded values; with fewer than 2, `lower` and `upper` will equal `value`.
2208+
2209+
```js
2210+
const { createHistogram } =require('node:perf_hooks');
2211+
2212+
consth=createHistogram();
2213+
for (let i =0; i <1000; i++) {
2214+
h.record(Math.floor(Math.random() *100));
2215+
}
2216+
2217+
constci=h.percentileCI(99);
2218+
console.log(ci.value); // The p99 point estimate
2219+
console.log(ci.lower); // The lower bound (95% confidence)
2220+
console.log(ci.upper); // The upper bound (95% confidence)
2221+
```
2222+
20632223
### `histogram.percentiles`
20642224

20652225
<!-- YAML
@@ -2127,6 +2287,31 @@ added: v11.10.0
21272287

21282288
The standard deviation of the recorded event loop delays.
21292289

2290+
### `histogram.welchTest(other[, options])`
2291+
2292+
<!-- YAML
2293+
added: REPLACEME
2294+
-->
2295+
2296+
*`other` {Histogram} The histogram to compare against.
2297+
*`options` {Object}
2298+
*`confidence` {number} Confidence level for the interval, between 0 and 1.
2299+
**Default:**`0.95`.
2300+
* Returns: {Object}
2301+
*`tStatistic` {number} The Welch t-statistic.
2302+
*`degreesOfFreedom` {number} Welch-Satterthwaite degrees of freedom.
2303+
*`pValue` {number} Two-tailed p-value.
2304+
*`confidenceInterval` {Object}
2305+
*`lower` {number} Lower bound of the confidence interval on the
2306+
difference of means.
2307+
*`upper` {number} Upper bound.
2308+
2309+
Performs [Welch's t-test][] comparing the means of this histogram and `other`.
2310+
The p-value indicates the probability of observing a difference at least this
2311+
extreme under the null hypothesis that the two distributions have the same
2312+
mean. Both histograms must have at least 2 recorded values; otherwise the
2313+
result has `pValue` 1 and `tStatistic` 0.
2314+
21302315
## Class: `ELDHistogram extends Histogram`
21312316

21322317
A `Histogram` that records event loop delay, returned by
@@ -2288,6 +2473,32 @@ const violating = latency.ccdf(500_000_000);
22882473
console.log(`${(violating *100).toFixed(1)}% of requests violating SLO`);
22892474
```
22902475

2476+
### SLO burn rate monitoring
2477+
2478+
```js
2479+
const { createHistogram } =require('node:perf_hooks');
2480+
2481+
// Track latency with EWMA (half-life 100 samples) and a 200ms SLO threshold
2482+
constlatency=createHistogram({
2483+
halfLife:100,
2484+
threshold:200_000_000, // 200ms in nanoseconds
2485+
});
2486+
2487+
// Record request latencies...
2488+
2489+
// Smoothed error rate: probability of exceeding the threshold
2490+
console.log(`Error rate: ${(latency.ewmaErrorRate*100).toFixed(2)}%`);
2491+
2492+
// Burn rate against a 99.9% SLO
2493+
// >1 means the error budget is depleting faster than allowed
2494+
constrate=latency.burnRate(0.999);
2495+
console.log(`Burn rate: ${rate.toFixed(2)}x`);
2496+
2497+
// EWMA mean and stddev track the smoothed latency
2498+
console.log(`EWMA latency: ${latency.ewmaMean.toFixed(0)}ns`);
2499+
console.log(`EWMA stddev: ${latency.ewmaStddev.toFixed(0)}ns`);
2500+
```
2501+
22912502
### Regression detection with KS test
22922503

22932504
```js
@@ -2339,6 +2550,46 @@ newSnapshot.subtract(snapshot);
23392550
console.log('Recent p99:', newSnapshot.percentile(99));
23402551
```
23412552

2553+
### Benchmark comparison with Welch's t-test
2554+
2555+
```js
2556+
const { createHistogram } =require('node:perf_hooks');
2557+
2558+
constbaseline=createHistogram();
2559+
constcandidate=createHistogram();
2560+
2561+
// Record operation rates from the old and new builds...
2562+
2563+
constresult=baseline.welchTest(candidate);
2564+
constimprovement= ((candidate.mean-baseline.mean) /baseline.mean*100);
2565+
2566+
console.log(`Improvement: ${improvement.toFixed(2)}%`);
2567+
console.log(`p-value: ${result.pValue.toFixed(6)}`);
2568+
console.log(`95% CI: [${result.confidenceInterval.lower.toFixed(2)}, `+
2569+
`${result.confidenceInterval.upper.toFixed(2)}]`);
2570+
2571+
if (result.pValue<0.05) {
2572+
constd=baseline.cohensD(candidate);
2573+
console.log(`Statistically significant (Cohen's d = ${d.toFixed(4)})`);
2574+
}
2575+
```
2576+
2577+
### Effect size with Cliff's delta
2578+
2579+
```js
2580+
const { createHistogram } =require('node:perf_hooks');
2581+
2582+
constbefore=createHistogram();
2583+
constafter=createHistogram();
2584+
2585+
// Record latencies before and after a change...
2586+
2587+
constdelta=before.cliffsD(after);
2588+
// A delta > 0: before tends to produce larger values (improvement)
2589+
// A delta < 0: after tends to produce larger values (regression)
2590+
console.log(`Cliff's delta: ${delta.toFixed(4)}`);
2591+
```
2592+
23422593
## Examples
23432594

23442595
### Measuring the duration of async operations
@@ -2593,13 +2844,17 @@ dns.promises.resolve('localhost');
25932844
```
25942845

25952846
[Async Hooks]: async_hooks.md
2847+
[Cliff's delta]: https://en.wikipedia.org/wiki/Effect_size#Cliff's_delta
2848+
[Cohen's d]: https://en.wikipedia.org/wiki/Effect_size#Cohen's_d
25962849
[Fetch Response Body Info]: https://fetch.spec.whatwg.org/#response-body-info
25972850
[Fetch Timing Info]: https://fetch.spec.whatwg.org/#fetch-timing-info
25982851
[High Resolution Time]: https://www.w3.org/TR/hr-time-2
2852+
[Mann-Whitney U test]: https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test
25992853
[Performance Timeline]: https://w3c.github.io/performance-timeline/
26002854
[Resource Timing]: https://www.w3.org/TR/resource-timing-2/
26012855
[User Timing]: https://www.w3.org/TR/user-timing/
26022856
[Web Performance APIs]: https://w3c.github.io/perf-timing-primer/
2857+
[Welch's t-test]: https://en.wikipedia.org/wiki/Welch%27s_t-test
26032858
[Worker threads]: worker_threads.md#worker-threads
26042859
[`'exit'`]: process.md#event-exit
26052860
[`child_process.spawnSync()`]: child_process.md#child_processspawnsynccommand-args-options

0 commit comments

Comments
 (0)
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Commit 9e8e908
Browse files
jasnelladuh95
jasnell
authored and
aduh95
committed
perf_hooks: add statistical hypothesis testing to histogram
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
1 parent dc51c79 commit 9e8e908
6 files changed
Lines changed: 1647 additions & 10 deletions
File tree
‎doc/api/perf_hooks.md‎
Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1625,6 +1625,16 @@ added:
16251625
**Default:**`Number.MAX_SAFE_INTEGER`.
16261626
*`figures` {number} The number of accuracy digits. Must be a number between
16271627
`1` and `5`. **Default:**`3`.
1628+
*`halfLife` {number} The EWMA half-life in number of samples. When set to
1629+
 a value greater than 0, the histogram tracks an exponentially weighted
1630+
 moving average and standard deviation, accessible via
1631+
`histogram.ewmaMean` and `histogram.ewmaStddev`. After `halfLife`
1632+
 recordings, a value's influence has decayed to 50%. **Default:**`0`
1633+
 (disabled).
1634+
*`threshold` {number} An SLO threshold value. When set together with
1635+
`halfLife`, the histogram tracks a smoothed error rate for values
1636+
 exceeding this threshold, accessible via `histogram.ewmaErrorRate` and
1637+
`histogram.burnRate()`. **Default:**`0` (disabled).
16281638
* Returns: {RecordableHistogram}
16291639

16301640
Returns a {RecordableHistogram}.
@@ -1893,6 +1903,36 @@ value, representing the probability that a recorded value will be less
18931903
than or equal to `value`. This is the inverse operation of
18941904
`histogram.percentile()`.
18951905

1906+
### `histogram.cliffsD(other)`
1907+
1908+
<!-- YAML
1909+
added: REPLACEME
1910+
-->
1911+
1912+
*`other` {Histogram} The histogram to compare against.
1913+
* Returns: {number} A value between -1.0 and 1.0.
1914+
1915+
Computes [Cliff's delta][], a non-parametric effect size measure. Returns
1916+
the probability that a random value from this histogram exceeds a random
1917+
value from `other`, minus the reverse probability. A value of 1 means every
1918+
value in this histogram exceeds every value in `other`; -1 means the
1919+
opposite; 0 means no tendency in either direction.
1920+
1921+
### `histogram.cohensD(other)`
1922+
1923+
<!-- YAML
1924+
added: REPLACEME
1925+
-->
1926+
1927+
*`other` {Histogram} The histogram to compare against.
1928+
* Returns: {number} The effect size.
1929+
1930+
Computes [Cohen's d][] effect size, the standardized difference between the
1931+
means of this histogram and `other`, using the pooled standard deviation.
1932+
Positive values indicate this histogram has a higher mean. By convention,
1933+
|d| < 0.2 is a small effect, 0.5 is medium, and 0.8 or greater is large.
1934+
Both histograms must have at least 2 recorded values; otherwise returns 0.
1935+
18961936
### `histogram.countAt(value)`
18971937

18981938
<!-- YAML
@@ -1929,6 +1969,74 @@ added:
19291969
The number of times the event loop delay exceeded the maximum 1 hour event
19301970
loop delay threshold.
19311971

1972+
### `histogram.ewmaMean`
1973+
1974+
<!-- YAML
1975+
added: REPLACEME
1976+
-->
1977+
1978+
* Type: {number}
1979+
1980+
The exponentially weighted moving average of recorded values. Only active
1981+
when the histogram was created with a `halfLife` option greater than 0.
1982+
Returns `0` when EWMA is disabled or no values have been recorded.
1983+
1984+
### `histogram.ewmaStddev`
1985+
1986+
<!-- YAML
1987+
added: REPLACEME
1988+
-->
1989+
1990+
* Type: {number}
1991+
1992+
The exponentially weighted moving standard deviation. Only active when the
1993+
histogram was created with a `halfLife` option greater than 0. Returns `0`
1994+
when EWMA is disabled or no values have been recorded.
1995+
1996+
### `histogram.ewmaErrorRate`
1997+
1998+
<!-- YAML
1999+
added: REPLACEME
2000+
-->
2001+
2002+
* Type: {number}
2003+
2004+
The EWMA-smoothed probability of a recorded value exceeding the configured
2005+
`threshold`. Only active when the histogram was created with both `halfLife`
2006+
and `threshold` options. Returns `0` when not enabled or no values have been
2007+
recorded.
2008+
2009+
### `histogram.burnRate(sloTarget)`
2010+
2011+
<!-- YAML
2012+
added: REPLACEME
2013+
-->
2014+
2015+
*`sloTarget` {number} The SLO target as a fraction between 0 and 1
2016+
 (exclusive). For example, `0.999` for a 99.9% SLO.
2017+
* Returns: {number}
2018+
2019+
Returns the SLO burn rate: `ewmaErrorRate / (1 - sloTarget)`. A burn rate
2020+
of 1 means the error budget will be exactly exhausted over the SLO window.
2021+
A burn rate greater than 1 means it is being consumed faster than allowed.
2022+
Requires the histogram to have been created with both `halfLife` and
2023+
`threshold` options.
2024+
2025+
```js
2026+
const { createHistogram } =require('node:perf_hooks');
2027+
2028+
// Track latency with a 200ms SLO threshold, half-life of 100 samples
2029+
consth=createHistogram({ halfLife:100, threshold:200_000_000 });
2030+
2031+
// ... record latency values ...
2032+
2033+
// Check burn rate against a 99.9% SLO
2034+
constrate=h.burnRate(0.999);
2035+
if (rate >1) {
2036+
console.log(`SLO burn rate: ${rate.toFixed(2)}x — error budget depleting`);
2037+
}
2038+
```
2039+
19322040
### `histogram.ksTest(other)`
19332041

19342042
<!-- YAML
@@ -1982,6 +2090,24 @@ Returns the histogram data rebucketed into logarithmically-spaced
19822090
intervals, where each bucket's width is multiplied by `base`.
19832091
Useful for visualization and export.
19842092

2093+
### `histogram.mannWhitneyTest(other)`
2094+
2095+
<!-- YAML
2096+
added: REPLACEME
2097+
-->
2098+
2099+
*`other` {Histogram} The histogram to compare against.
2100+
* Returns: {Object}
2101+
*`uStatistic` {number} The Mann-Whitney U statistic.
2102+
*`zScore` {number} The z-score (normal approximation).
2103+
*`pValue` {number} Two-tailed p-value.
2104+
2105+
Performs a [Mann-Whitney U test][] comparing whether this histogram tends to
2106+
produce larger or smaller values than `other`. Unlike `welchTest()`, this is a
2107+
non-parametric test that makes no assumptions about the shape of the
2108+
distributions. Uses the normal approximation with tie correction for the
2109+
p-value.
2110+
19852111
### `histogram.max`
19862112

19872113
<!-- YAML
@@ -2060,6 +2186,40 @@ added:
20602186

20612187
Returns the value at the given percentile.
20622188

2189+
### `histogram.percentileCI(percentile[, options])`
2190+
2191+
<!-- YAML
2192+
added: REPLACEME
2193+
-->
2194+
2195+
*`percentile` {number} A percentile value in the range (0, 100].
2196+
*`options` {Object}
2197+
*`confidence` {number} The confidence level for the interval, between
2198+
 0 and 1 (exclusive). **Default:**`0.95`.
2199+
* Returns: {Object}
2200+
*`value` {number} The point estimate (same as `histogram.percentile()`).
2201+
*`lower` {number} The lower bound of the confidence interval.
2202+
*`upper` {number} The upper bound of the confidence interval.
2203+
2204+
Returns a confidence interval for the given percentile using the exact
2205+
binomial method. With fewer samples, the interval will be wider, reflecting
2206+
the greater uncertainty in the percentile estimate. Requires at least 2
2207+
recorded values; with fewer than 2, `lower` and `upper` will equal `value`.
2208+
2209+
```js
2210+
const { createHistogram } =require('node:perf_hooks');
2211+
2212+
consth=createHistogram();
2213+
for (let i =0; i <1000; i++) {
2214+
h.record(Math.floor(Math.random() *100));
2215+
}
2216+
2217+
constci=h.percentileCI(99);
2218+
console.log(ci.value); // The p99 point estimate
2219+
console.log(ci.lower); // The lower bound (95% confidence)
2220+
console.log(ci.upper); // The upper bound (95% confidence)
2221+
```
2222+
20632223
### `histogram.percentiles`
20642224

20652225
<!-- YAML
@@ -2127,6 +2287,31 @@ added: v11.10.0
21272287

21282288
The standard deviation of the recorded event loop delays.
21292289

2290+
### `histogram.welchTest(other[, options])`
2291+
2292+
<!-- YAML
2293+
added: REPLACEME
2294+
-->
2295+
2296+
*`other` {Histogram} The histogram to compare against.
2297+
*`options` {Object}
2298+
*`confidence` {number} Confidence level for the interval, between 0 and 1.
2299+
**Default:**`0.95`.
2300+
* Returns: {Object}
2301+
*`tStatistic` {number} The Welch t-statistic.
2302+
*`degreesOfFreedom` {number} Welch-Satterthwaite degrees of freedom.
2303+
*`pValue` {number} Two-tailed p-value.
2304+
*`confidenceInterval` {Object}
2305+
*`lower` {number} Lower bound of the confidence interval on the
2306+
 difference of means.
2307+
*`upper` {number} Upper bound.
2308+
2309+
Performs [Welch's t-test][] comparing the means of this histogram and `other`.
2310+
The p-value indicates the probability of observing a difference at least this
2311+
extreme under the null hypothesis that the two distributions have the same
2312+
mean. Both histograms must have at least 2 recorded values; otherwise the
2313+
result has `pValue` 1 and `tStatistic` 0.
2314+
21302315
## Class: `ELDHistogram extends Histogram`
21312316

21322317
A `Histogram` that records event loop delay, returned by
@@ -2288,6 +2473,32 @@ const violating = latency.ccdf(500_000_000);
22882473
console.log(`${(violating *100).toFixed(1)}% of requests violating SLO`);
22892474
```
22902475

2476+
### SLO burn rate monitoring
2477+
2478+
```js
2479+
const { createHistogram } =require('node:perf_hooks');
2480+
2481+
// Track latency with EWMA (half-life 100 samples) and a 200ms SLO threshold
2482+
constlatency=createHistogram({
2483+
 halfLife:100,
2484+
 threshold:200_000_000, // 200ms in nanoseconds
2485+
});
2486+
2487+
// Record request latencies...
2488+
2489+
// Smoothed error rate: probability of exceeding the threshold
2490+
console.log(`Error rate: ${(latency.ewmaErrorRate*100).toFixed(2)}%`);
2491+
2492+
// Burn rate against a 99.9% SLO
2493+
// >1 means the error budget is depleting faster than allowed
2494+
constrate=latency.burnRate(0.999);
2495+
console.log(`Burn rate: ${rate.toFixed(2)}x`);
2496+
2497+
// EWMA mean and stddev track the smoothed latency
2498+
console.log(`EWMA latency: ${latency.ewmaMean.toFixed(0)}ns`);
2499+
console.log(`EWMA stddev: ${latency.ewmaStddev.toFixed(0)}ns`);
2500+
```
2501+
22912502
### Regression detection with KS test
22922503

22932504
```js
@@ -2339,6 +2550,46 @@ newSnapshot.subtract(snapshot);
23392550
console.log('Recent p99:', newSnapshot.percentile(99));
23402551
```
23412552

2553+
### Benchmark comparison with Welch's t-test
2554+
2555+
```js
2556+
const { createHistogram } =require('node:perf_hooks');
2557+
2558+
constbaseline=createHistogram();
2559+
constcandidate=createHistogram();
2560+
2561+
// Record operation rates from the old and new builds...
2562+
2563+
constresult=baseline.welchTest(candidate);
2564+
constimprovement= ((candidate.mean-baseline.mean) /baseline.mean*100);
2565+
2566+
console.log(`Improvement: ${improvement.toFixed(2)}%`);
2567+
console.log(`p-value: ${result.pValue.toFixed(6)}`);
2568+
console.log(`95% CI: [${result.confidenceInterval.lower.toFixed(2)}, `+
2569+
`${result.confidenceInterval.upper.toFixed(2)}]`);
2570+
2571+
if (result.pValue<0.05) {
2572+
constd=baseline.cohensD(candidate);
2573+
console.log(`Statistically significant (Cohen's d = ${d.toFixed(4)})`);
2574+
}
2575+
```
2576+
2577+
### Effect size with Cliff's delta
2578+
2579+
```js
2580+
const { createHistogram } =require('node:perf_hooks');
2581+
2582+
constbefore=createHistogram();
2583+
constafter=createHistogram();
2584+
2585+
// Record latencies before and after a change...
2586+
2587+
constdelta=before.cliffsD(after);
2588+
// A delta > 0: before tends to produce larger values (improvement)
2589+
// A delta < 0: after tends to produce larger values (regression)
2590+
console.log(`Cliff's delta: ${delta.toFixed(4)}`);
2591+
```
2592+
23422593
## Examples
23432594

23442595
### Measuring the duration of async operations
@@ -2593,13 +2844,17 @@ dns.promises.resolve('localhost');
25932844
```
25942845

25952846
[Async Hooks]: async_hooks.md
2847+
[Cliff's delta]: https://en.wikipedia.org/wiki/Effect_size#Cliff's_delta
2848+
[Cohen's d]: https://en.wikipedia.org/wiki/Effect_size#Cohen's_d
25962849
[Fetch Response Body Info]: https://fetch.spec.whatwg.org/#response-body-info
25972850
[Fetch Timing Info]: https://fetch.spec.whatwg.org/#fetch-timing-info
25982851
[High Resolution Time]: https://www.w3.org/TR/hr-time-2
2852+
[Mann-Whitney U test]: https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test
25992853
[Performance Timeline]: https://w3c.github.io/performance-timeline/
26002854
[Resource Timing]: https://www.w3.org/TR/resource-timing-2/
26012855
[User Timing]: https://www.w3.org/TR/user-timing/
26022856
[Web Performance APIs]: https://w3c.github.io/perf-timing-primer/
2857+
[Welch's t-test]: https://en.wikipedia.org/wiki/Welch%27s_t-test
26032858
[Worker threads]: worker_threads.md#worker-threads
26042859
[`'exit'`]: process.md#event-exit
26052860
[`child_process.spawnSync()`]: child_process.md#child_processspawnsynccommand-args-options
0 commit comments
Comments
 (0)

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Commit 9e8e908
Browse files
jasnelladuh95
jasnell
authored and
aduh95
committed
perf_hooks: add statistical hypothesis testing to histogram
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
1 parent dc51c79 commit 9e8e908
6 files changed
Lines changed: 1647 additions & 10 deletions
File tree
‎doc/api/perf_hooks.md‎
Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1625,6 +1625,16 @@ added:
16251625
**Default:**`Number.MAX_SAFE_INTEGER`.
16261626
*`figures` {number} The number of accuracy digits. Must be a number between
16271627
`1` and `5`. **Default:**`3`.
1628+
*`halfLife` {number} The EWMA half-life in number of samples. When set to
1629+
 a value greater than 0, the histogram tracks an exponentially weighted
1630+
 moving average and standard deviation, accessible via
1631+
`histogram.ewmaMean` and `histogram.ewmaStddev`. After `halfLife`
1632+
 recordings, a value's influence has decayed to 50%. **Default:**`0`
1633+
 (disabled).
1634+
*`threshold` {number} An SLO threshold value. When set together with
1635+
`halfLife`, the histogram tracks a smoothed error rate for values
1636+
 exceeding this threshold, accessible via `histogram.ewmaErrorRate` and
1637+
`histogram.burnRate()`. **Default:**`0` (disabled).
16281638
* Returns: {RecordableHistogram}
16291639

16301640
Returns a {RecordableHistogram}.
@@ -1893,6 +1903,36 @@ value, representing the probability that a recorded value will be less
18931903
than or equal to `value`. This is the inverse operation of
18941904
`histogram.percentile()`.
18951905

1906+
### `histogram.cliffsD(other)`
1907+
1908+
<!-- YAML
1909+
added: REPLACEME
1910+
-->
1911+
1912+
*`other` {Histogram} The histogram to compare against.
1913+
* Returns: {number} A value between -1.0 and 1.0.
1914+
1915+
Computes [Cliff's delta][], a non-parametric effect size measure. Returns
1916+
the probability that a random value from this histogram exceeds a random
1917+
value from `other`, minus the reverse probability. A value of 1 means every
1918+
value in this histogram exceeds every value in `other`; -1 means the
1919+
opposite; 0 means no tendency in either direction.
1920+
1921+
### `histogram.cohensD(other)`
1922+
1923+
<!-- YAML
1924+
added: REPLACEME
1925+
-->
1926+
1927+
*`other` {Histogram} The histogram to compare against.
1928+
* Returns: {number} The effect size.
1929+
1930+
Computes [Cohen's d][] effect size, the standardized difference between the
1931+
means of this histogram and `other`, using the pooled standard deviation.
1932+
Positive values indicate this histogram has a higher mean. By convention,
1933+
|d| < 0.2 is a small effect, 0.5 is medium, and 0.8 or greater is large.
1934+
Both histograms must have at least 2 recorded values; otherwise returns 0.
1935+
18961936
### `histogram.countAt(value)`
18971937

18981938
<!-- YAML
@@ -1929,6 +1969,74 @@ added:
19291969
The number of times the event loop delay exceeded the maximum 1 hour event
19301970
loop delay threshold.
19311971

1972+
### `histogram.ewmaMean`
1973+
1974+
<!-- YAML
1975+
added: REPLACEME
1976+
-->
1977+
1978+
* Type: {number}
1979+
1980+
The exponentially weighted moving average of recorded values. Only active
1981+
when the histogram was created with a `halfLife` option greater than 0.
1982+
Returns `0` when EWMA is disabled or no values have been recorded.
1983+
1984+
### `histogram.ewmaStddev`
1985+
1986+
<!-- YAML
1987+
added: REPLACEME
1988+
-->
1989+
1990+
* Type: {number}
1991+
1992+
The exponentially weighted moving standard deviation. Only active when the
1993+
histogram was created with a `halfLife` option greater than 0. Returns `0`
1994+
when EWMA is disabled or no values have been recorded.
1995+
1996+
### `histogram.ewmaErrorRate`
1997+
1998+
<!-- YAML
1999+
added: REPLACEME
2000+
-->
2001+
2002+
* Type: {number}
2003+
2004+
The EWMA-smoothed probability of a recorded value exceeding the configured
2005+
`threshold`. Only active when the histogram was created with both `halfLife`
2006+
and `threshold` options. Returns `0` when not enabled or no values have been
2007+
recorded.
2008+
2009+
### `histogram.burnRate(sloTarget)`
2010+
2011+
<!-- YAML
2012+
added: REPLACEME
2013+
-->
2014+
2015+
*`sloTarget` {number} The SLO target as a fraction between 0 and 1
2016+
 (exclusive). For example, `0.999` for a 99.9% SLO.
2017+
* Returns: {number}
2018+
2019+
Returns the SLO burn rate: `ewmaErrorRate / (1 - sloTarget)`. A burn rate
2020+
of 1 means the error budget will be exactly exhausted over the SLO window.
2021+
A burn rate greater than 1 means it is being consumed faster than allowed.
2022+
Requires the histogram to have been created with both `halfLife` and
2023+
`threshold` options.
2024+
2025+
```js
2026+
const { createHistogram } =require('node:perf_hooks');
2027+
2028+
// Track latency with a 200ms SLO threshold, half-life of 100 samples
2029+
consth=createHistogram({ halfLife:100, threshold:200_000_000 });
2030+
2031+
// ... record latency values ...
2032+
2033+
// Check burn rate against a 99.9% SLO
2034+
constrate=h.burnRate(0.999);
2035+
if (rate >1) {
2036+
console.log(`SLO burn rate: ${rate.toFixed(2)}x — error budget depleting`);
2037+
}
2038+
```
2039+
19322040
### `histogram.ksTest(other)`
19332041

19342042
<!-- YAML
@@ -1982,6 +2090,24 @@ Returns the histogram data rebucketed into logarithmically-spaced
19822090
intervals, where each bucket's width is multiplied by `base`.
19832091
Useful for visualization and export.
19842092

2093+
### `histogram.mannWhitneyTest(other)`
2094+
2095+
<!-- YAML
2096+
added: REPLACEME
2097+
-->
2098+
2099+
*`other` {Histogram} The histogram to compare against.
2100+
* Returns: {Object}
2101+
*`uStatistic` {number} The Mann-Whitney U statistic.
2102+
*`zScore` {number} The z-score (normal approximation).
2103+
*`pValue` {number} Two-tailed p-value.
2104+
2105+
Performs a [Mann-Whitney U test][] comparing whether this histogram tends to
2106+
produce larger or smaller values than `other`. Unlike `welchTest()`, this is a
2107+
non-parametric test that makes no assumptions about the shape of the
2108+
distributions. Uses the normal approximation with tie correction for the
2109+
p-value.
2110+
19852111
### `histogram.max`
19862112

19872113
<!-- YAML
@@ -2060,6 +2186,40 @@ added:
20602186

20612187
Returns the value at the given percentile.
20622188

2189+
### `histogram.percentileCI(percentile[, options])`
2190+
2191+
<!-- YAML
2192+
added: REPLACEME
2193+
-->
2194+
2195+
*`percentile` {number} A percentile value in the range (0, 100].
2196+
*`options` {Object}
2197+
*`confidence` {number} The confidence level for the interval, between
2198+
 0 and 1 (exclusive). **Default:**`0.95`.
2199+
* Returns: {Object}
2200+
*`value` {number} The point estimate (same as `histogram.percentile()`).
2201+
*`lower` {number} The lower bound of the confidence interval.
2202+
*`upper` {number} The upper bound of the confidence interval.
2203+
2204+
Returns a confidence interval for the given percentile using the exact
2205+
binomial method. With fewer samples, the interval will be wider, reflecting
2206+
the greater uncertainty in the percentile estimate. Requires at least 2
2207+
recorded values; with fewer than 2, `lower` and `upper` will equal `value`.
2208+
2209+
```js
2210+
const { createHistogram } =require('node:perf_hooks');
2211+
2212+
consth=createHistogram();
2213+
for (let i =0; i <1000; i++) {
2214+
h.record(Math.floor(Math.random() *100));
2215+
}
2216+
2217+
constci=h.percentileCI(99);
2218+
console.log(ci.value); // The p99 point estimate
2219+
console.log(ci.lower); // The lower bound (95% confidence)
2220+
console.log(ci.upper); // The upper bound (95% confidence)
2221+
```
2222+
20632223
### `histogram.percentiles`
20642224

20652225
<!-- YAML
@@ -2127,6 +2287,31 @@ added: v11.10.0
21272287

21282288
The standard deviation of the recorded event loop delays.
21292289

2290+
### `histogram.welchTest(other[, options])`
2291+
2292+
<!-- YAML
2293+
added: REPLACEME
2294+
-->
2295+
2296+
*`other` {Histogram} The histogram to compare against.
2297+
*`options` {Object}
2298+
*`confidence` {number} Confidence level for the interval, between 0 and 1.
2299+
**Default:**`0.95`.
2300+
* Returns: {Object}
2301+
*`tStatistic` {number} The Welch t-statistic.
2302+
*`degreesOfFreedom` {number} Welch-Satterthwaite degrees of freedom.
2303+
*`pValue` {number} Two-tailed p-value.
2304+
*`confidenceInterval` {Object}
2305+
*`lower` {number} Lower bound of the confidence interval on the
2306+
 difference of means.
2307+
*`upper` {number} Upper bound.
2308+
2309+
Performs [Welch's t-test][] comparing the means of this histogram and `other`.
2310+
The p-value indicates the probability of observing a difference at least this
2311+
extreme under the null hypothesis that the two distributions have the same
2312+
mean. Both histograms must have at least 2 recorded values; otherwise the
2313+
result has `pValue` 1 and `tStatistic` 0.
2314+
21302315
## Class: `ELDHistogram extends Histogram`
21312316

21322317
A `Histogram` that records event loop delay, returned by
@@ -2288,6 +2473,32 @@ const violating = latency.ccdf(500_000_000);
22882473
console.log(`${(violating *100).toFixed(1)}% of requests violating SLO`);
22892474
```
22902475

2476+
### SLO burn rate monitoring
2477+
2478+
```js
2479+
const { createHistogram } =require('node:perf_hooks');
2480+
2481+
// Track latency with EWMA (half-life 100 samples) and a 200ms SLO threshold
2482+
constlatency=createHistogram({
2483+
 halfLife:100,
2484+
 threshold:200_000_000, // 200ms in nanoseconds
2485+
});
2486+
2487+
// Record request latencies...
2488+
2489+
// Smoothed error rate: probability of exceeding the threshold
2490+
console.log(`Error rate: ${(latency.ewmaErrorRate*100).toFixed(2)}%`);
2491+
2492+
// Burn rate against a 99.9% SLO
2493+
// >1 means the error budget is depleting faster than allowed
2494+
constrate=latency.burnRate(0.999);
2495+
console.log(`Burn rate: ${rate.toFixed(2)}x`);
2496+
2497+
// EWMA mean and stddev track the smoothed latency
2498+
console.log(`EWMA latency: ${latency.ewmaMean.toFixed(0)}ns`);
2499+
console.log(`EWMA stddev: ${latency.ewmaStddev.toFixed(0)}ns`);
2500+
```
2501+
22912502
### Regression detection with KS test
22922503

22932504
```js
@@ -2339,6 +2550,46 @@ newSnapshot.subtract(snapshot);
23392550
console.log('Recent p99:', newSnapshot.percentile(99));
23402551
```
23412552

2553+
### Benchmark comparison with Welch's t-test
2554+
2555+
```js
2556+
const { createHistogram } =require('node:perf_hooks');
2557+
2558+
constbaseline=createHistogram();
2559+
constcandidate=createHistogram();
2560+
2561+
// Record operation rates from the old and new builds...
2562+
2563+
constresult=baseline.welchTest(candidate);
2564+
constimprovement= ((candidate.mean-baseline.mean) /baseline.mean*100);
2565+
2566+
console.log(`Improvement: ${improvement.toFixed(2)}%`);
2567+
console.log(`p-value: ${result.pValue.toFixed(6)}`);
2568+
console.log(`95% CI: [${result.confidenceInterval.lower.toFixed(2)}, `+
2569+
`${result.confidenceInterval.upper.toFixed(2)}]`);
2570+
2571+
if (result.pValue<0.05) {
2572+
constd=baseline.cohensD(candidate);
2573+
console.log(`Statistically significant (Cohen's d = ${d.toFixed(4)})`);
2574+
}
2575+
```
2576+
2577+
### Effect size with Cliff's delta
2578+
2579+
```js
2580+
const { createHistogram } =require('node:perf_hooks');
2581+
2582+
constbefore=createHistogram();
2583+
constafter=createHistogram();
2584+
2585+
// Record latencies before and after a change...
2586+
2587+
constdelta=before.cliffsD(after);
2588+
// A delta > 0: before tends to produce larger values (improvement)
2589+
// A delta < 0: after tends to produce larger values (regression)
2590+
console.log(`Cliff's delta: ${delta.toFixed(4)}`);
2591+
```
2592+
23422593
## Examples
23432594

23442595
### Measuring the duration of async operations
@@ -2593,13 +2844,17 @@ dns.promises.resolve('localhost');
25932844
```
25942845

25952846
[Async Hooks]: async_hooks.md
2847+
[Cliff's delta]: https://en.wikipedia.org/wiki/Effect_size#Cliff's_delta
2848+
[Cohen's d]: https://en.wikipedia.org/wiki/Effect_size#Cohen's_d
25962849
[Fetch Response Body Info]: https://fetch.spec.whatwg.org/#response-body-info
25972850
[Fetch Timing Info]: https://fetch.spec.whatwg.org/#fetch-timing-info
25982851
[High Resolution Time]: https://www.w3.org/TR/hr-time-2
2852+
[Mann-Whitney U test]: https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test
25992853
[Performance Timeline]: https://w3c.github.io/performance-timeline/
26002854
[Resource Timing]: https://www.w3.org/TR/resource-timing-2/
26012855
[User Timing]: https://www.w3.org/TR/user-timing/
26022856
[Web Performance APIs]: https://w3c.github.io/perf-timing-primer/
2857+
[Welch's t-test]: https://en.wikipedia.org/wiki/Welch%27s_t-test
26032858
[Worker threads]: worker_threads.md#worker-threads
26042859
[`'exit'`]: process.md#event-exit
26052860
[`child_process.spawnSync()`]: child_process.md#child_processspawnsynccommand-args-options
0 commit comments
Comments
 (0)

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Commit 9e8e908
Browse files
jasnelladuh95
jasnell
authored and
aduh95
committed
perf_hooks: add statistical hypothesis testing to histogram
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
1 parent dc51c79 commit 9e8e908
6 files changed
Lines changed: 1647 additions & 10 deletions
File tree
‎doc/api/perf_hooks.md‎
Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1625,6 +1625,16 @@ added:
16251625
**Default:**`Number.MAX_SAFE_INTEGER`.
16261626
*`figures` {number} The number of accuracy digits. Must be a number between
16271627
`1` and `5`. **Default:**`3`.
1628+
*`halfLife` {number} The EWMA half-life in number of samples. When set to
1629+
 a value greater than 0, the histogram tracks an exponentially weighted
1630+
 moving average and standard deviation, accessible via
1631+
`histogram.ewmaMean` and `histogram.ewmaStddev`. After `halfLife`
1632+
 recordings, a value's influence has decayed to 50%. **Default:**`0`
1633+
 (disabled).
1634+
*`threshold` {number} An SLO threshold value. When set together with
1635+
`halfLife`, the histogram tracks a smoothed error rate for values
1636+
 exceeding this threshold, accessible via `histogram.ewmaErrorRate` and
1637+
`histogram.burnRate()`. **Default:**`0` (disabled).
16281638
* Returns: {RecordableHistogram}
16291639

16301640
Returns a {RecordableHistogram}.
@@ -1893,6 +1903,36 @@ value, representing the probability that a recorded value will be less
18931903
than or equal to `value`. This is the inverse operation of
18941904
`histogram.percentile()`.
18951905

1906+
### `histogram.cliffsD(other)`
1907+
1908+
<!-- YAML
1909+
added: REPLACEME
1910+
-->
1911+
1912+
*`other` {Histogram} The histogram to compare against.
1913+
* Returns: {number} A value between -1.0 and 1.0.
1914+
1915+
Computes [Cliff's delta][], a non-parametric effect size measure. Returns
1916+
the probability that a random value from this histogram exceeds a random
1917+
value from `other`, minus the reverse probability. A value of 1 means every
1918+
value in this histogram exceeds every value in `other`; -1 means the
1919+
opposite; 0 means no tendency in either direction.
1920+
1921+
### `histogram.cohensD(other)`
1922+
1923+
<!-- YAML
1924+
added: REPLACEME
1925+
-->
1926+
1927+
*`other` {Histogram} The histogram to compare against.
1928+
* Returns: {number} The effect size.
1929+
1930+
Computes [Cohen's d][] effect size, the standardized difference between the
1931+
means of this histogram and `other`, using the pooled standard deviation.
1932+
Positive values indicate this histogram has a higher mean. By convention,
1933+
|d| < 0.2 is a small effect, 0.5 is medium, and 0.8 or greater is large.
1934+
Both histograms must have at least 2 recorded values; otherwise returns 0.
1935+
18961936
### `histogram.countAt(value)`
18971937

18981938
<!-- YAML
@@ -1929,6 +1969,74 @@ added:
19291969
The number of times the event loop delay exceeded the maximum 1 hour event
19301970
loop delay threshold.
19311971

1972+
### `histogram.ewmaMean`
1973+
1974+
<!-- YAML
1975+
added: REPLACEME
1976+
-->
1977+
1978+
* Type: {number}
1979+
1980+
The exponentially weighted moving average of recorded values. Only active
1981+
when the histogram was created with a `halfLife` option greater than 0.
1982+
Returns `0` when EWMA is disabled or no values have been recorded.
1983+
1984+
### `histogram.ewmaStddev`
1985+
1986+
<!-- YAML
1987+
added: REPLACEME
1988+
-->
1989+
1990+
* Type: {number}
1991+
1992+
The exponentially weighted moving standard deviation. Only active when the
1993+
histogram was created with a `halfLife` option greater than 0. Returns `0`
1994+
when EWMA is disabled or no values have been recorded.
1995+
1996+
### `histogram.ewmaErrorRate`
1997+
1998+
<!-- YAML
1999+
added: REPLACEME
2000+
-->
2001+
2002+
* Type: {number}
2003+
2004+
The EWMA-smoothed probability of a recorded value exceeding the configured
2005+
`threshold`. Only active when the histogram was created with both `halfLife`
2006+
and `threshold` options. Returns `0` when not enabled or no values have been
2007+
recorded.
2008+
2009+
### `histogram.burnRate(sloTarget)`
2010+
2011+
<!-- YAML
2012+
added: REPLACEME
2013+
-->
2014+
2015+
*`sloTarget` {number} The SLO target as a fraction between 0 and 1
2016+
 (exclusive). For example, `0.999` for a 99.9% SLO.
2017+
* Returns: {number}
2018+
2019+
Returns the SLO burn rate: `ewmaErrorRate / (1 - sloTarget)`. A burn rate
2020+
of 1 means the error budget will be exactly exhausted over the SLO window.
2021+
A burn rate greater than 1 means it is being consumed faster than allowed.
2022+
Requires the histogram to have been created with both `halfLife` and
2023+
`threshold` options.
2024+
2025+
```js
2026+
const { createHistogram } =require('node:perf_hooks');
2027+
2028+
// Track latency with a 200ms SLO threshold, half-life of 100 samples
2029+
consth=createHistogram({ halfLife:100, threshold:200_000_000 });
2030+
2031+
// ... record latency values ...
2032+
2033+
// Check burn rate against a 99.9% SLO
2034+
constrate=h.burnRate(0.999);
2035+
if (rate >1) {
2036+
console.log(`SLO burn rate: ${rate.toFixed(2)}x — error budget depleting`);
2037+
}
2038+
```
2039+
19322040
### `histogram.ksTest(other)`
19332041

19342042
<!-- YAML
@@ -1982,6 +2090,24 @@ Returns the histogram data rebucketed into logarithmically-spaced
19822090
intervals, where each bucket's width is multiplied by `base`.
19832091
Useful for visualization and export.
19842092

2093+
### `histogram.mannWhitneyTest(other)`
2094+
2095+
<!-- YAML
2096+
added: REPLACEME
2097+
-->
2098+
2099+
*`other` {Histogram} The histogram to compare against.
2100+
* Returns: {Object}
2101+
*`uStatistic` {number} The Mann-Whitney U statistic.
2102+
*`zScore` {number} The z-score (normal approximation).
2103+
*`pValue` {number} Two-tailed p-value.
2104+
2105+
Performs a [Mann-Whitney U test][] comparing whether this histogram tends to
2106+
produce larger or smaller values than `other`. Unlike `welchTest()`, this is a
2107+
non-parametric test that makes no assumptions about the shape of the
2108+
distributions. Uses the normal approximation with tie correction for the
2109+
p-value.
2110+
19852111
### `histogram.max`
19862112

19872113
<!-- YAML
@@ -2060,6 +2186,40 @@ added:
20602186

20612187
Returns the value at the given percentile.
20622188

2189+
### `histogram.percentileCI(percentile[, options])`
2190+
2191+
<!-- YAML
2192+
added: REPLACEME
2193+
-->
2194+
2195+
*`percentile` {number} A percentile value in the range (0, 100].
2196+
*`options` {Object}
2197+
*`confidence` {number} The confidence level for the interval, between
2198+
 0 and 1 (exclusive). **Default:**`0.95`.
2199+
* Returns: {Object}
2200+
*`value` {number} The point estimate (same as `histogram.percentile()`).
2201+
*`lower` {number} The lower bound of the confidence interval.
2202+
*`upper` {number} The upper bound of the confidence interval.
2203+
2204+
Returns a confidence interval for the given percentile using the exact
2205+
binomial method. With fewer samples, the interval will be wider, reflecting
2206+
the greater uncertainty in the percentile estimate. Requires at least 2
2207+
recorded values; with fewer than 2, `lower` and `upper` will equal `value`.
2208+
2209+
```js
2210+
const { createHistogram } =require('node:perf_hooks');
2211+
2212+
consth=createHistogram();
2213+
for (let i =0; i <1000; i++) {
2214+
h.record(Math.floor(Math.random() *100));
2215+
}
2216+
2217+
constci=h.percentileCI(99);
2218+
console.log(ci.value); // The p99 point estimate
2219+
console.log(ci.lower); // The lower bound (95% confidence)
2220+
console.log(ci.upper); // The upper bound (95% confidence)
2221+
```
2222+
20632223
### `histogram.percentiles`
20642224

20652225
<!-- YAML
@@ -2127,6 +2287,31 @@ added: v11.10.0
21272287

21282288
The standard deviation of the recorded event loop delays.
21292289

2290+
### `histogram.welchTest(other[, options])`
2291+
2292+
<!-- YAML
2293+
added: REPLACEME
2294+
-->
2295+
2296+
*`other` {Histogram} The histogram to compare against.
2297+
*`options` {Object}
2298+
*`confidence` {number} Confidence level for the interval, between 0 and 1.
2299+
**Default:**`0.95`.
2300+
* Returns: {Object}
2301+
*`tStatistic` {number} The Welch t-statistic.
2302+
*`degreesOfFreedom` {number} Welch-Satterthwaite degrees of freedom.
2303+
*`pValue` {number} Two-tailed p-value.
2304+
*`confidenceInterval` {Object}
2305+
*`lower` {number} Lower bound of the confidence interval on the
2306+
 difference of means.
2307+
*`upper` {number} Upper bound.
2308+
2309+
Performs [Welch's t-test][] comparing the means of this histogram and `other`.
2310+
The p-value indicates the probability of observing a difference at least this
2311+
extreme under the null hypothesis that the two distributions have the same
2312+
mean. Both histograms must have at least 2 recorded values; otherwise the
2313+
result has `pValue` 1 and `tStatistic` 0.
2314+
21302315
## Class: `ELDHistogram extends Histogram`
21312316

21322317
A `Histogram` that records event loop delay, returned by
@@ -2288,6 +2473,32 @@ const violating = latency.ccdf(500_000_000);
22882473
console.log(`${(violating *100).toFixed(1)}% of requests violating SLO`);
22892474
```
22902475

2476+
### SLO burn rate monitoring
2477+
2478+
```js
2479+
const { createHistogram } =require('node:perf_hooks');
2480+
2481+
// Track latency with EWMA (half-life 100 samples) and a 200ms SLO threshold
2482+
constlatency=createHistogram({
2483+
 halfLife:100,
2484+
 threshold:200_000_000, // 200ms in nanoseconds
2485+
});
2486+
2487+
// Record request latencies...
2488+
2489+
// Smoothed error rate: probability of exceeding the threshold
2490+
console.log(`Error rate: ${(latency.ewmaErrorRate*100).toFixed(2)}%`);
2491+
2492+
// Burn rate against a 99.9% SLO
2493+
// >1 means the error budget is depleting faster than allowed
2494+
constrate=latency.burnRate(0.999);
2495+
console.log(`Burn rate: ${rate.toFixed(2)}x`);
2496+
2497+
// EWMA mean and stddev track the smoothed latency
2498+
console.log(`EWMA latency: ${latency.ewmaMean.toFixed(0)}ns`);
2499+
console.log(`EWMA stddev: ${latency.ewmaStddev.toFixed(0)}ns`);
2500+
```
2501+
22912502
### Regression detection with KS test
22922503

22932504
```js
@@ -2339,6 +2550,46 @@ newSnapshot.subtract(snapshot);
23392550
console.log('Recent p99:', newSnapshot.percentile(99));
23402551
```
23412552

2553+
### Benchmark comparison with Welch's t-test
2554+
2555+
```js
2556+
const { createHistogram } =require('node:perf_hooks');
2557+
2558+
constbaseline=createHistogram();
2559+
constcandidate=createHistogram();
2560+
2561+
// Record operation rates from the old and new builds...
2562+
2563+
constresult=baseline.welchTest(candidate);
2564+
constimprovement= ((candidate.mean-baseline.mean) /baseline.mean*100);
2565+
2566+
console.log(`Improvement: ${improvement.toFixed(2)}%`);
2567+
console.log(`p-value: ${result.pValue.toFixed(6)}`);
2568+
console.log(`95% CI: [${result.confidenceInterval.lower.toFixed(2)}, `+
2569+
`${result.confidenceInterval.upper.toFixed(2)}]`);
2570+
2571+
if (result.pValue<0.05) {
2572+
constd=baseline.cohensD(candidate);
2573+
console.log(`Statistically significant (Cohen's d = ${d.toFixed(4)})`);
2574+
}
2575+
```
2576+
2577+
### Effect size with Cliff's delta
2578+
2579+
```js
2580+
const { createHistogram } =require('node:perf_hooks');
2581+
2582+
constbefore=createHistogram();
2583+
constafter=createHistogram();
2584+
2585+
// Record latencies before and after a change...
2586+
2587+
constdelta=before.cliffsD(after);
2588+
// A delta > 0: before tends to produce larger values (improvement)
2589+
// A delta < 0: after tends to produce larger values (regression)
2590+
console.log(`Cliff's delta: ${delta.toFixed(4)}`);
2591+
```
2592+
23422593
## Examples
23432594

23442595
### Measuring the duration of async operations
@@ -2593,13 +2844,17 @@ dns.promises.resolve('localhost');
25932844
```
25942845

25952846
[Async Hooks]: async_hooks.md
2847+
[Cliff's delta]: https://en.wikipedia.org/wiki/Effect_size#Cliff's_delta
2848+
[Cohen's d]: https://en.wikipedia.org/wiki/Effect_size#Cohen's_d
25962849
[Fetch Response Body Info]: https://fetch.spec.whatwg.org/#response-body-info
25972850
[Fetch Timing Info]: https://fetch.spec.whatwg.org/#fetch-timing-info
25982851
[High Resolution Time]: https://www.w3.org/TR/hr-time-2
2852+
[Mann-Whitney U test]: https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test
25992853
[Performance Timeline]: https://w3c.github.io/performance-timeline/
26002854
[Resource Timing]: https://www.w3.org/TR/resource-timing-2/
26012855
[User Timing]: https://www.w3.org/TR/user-timing/
26022856
[Web Performance APIs]: https://w3c.github.io/perf-timing-primer/
2857+
[Welch's t-test]: https://en.wikipedia.org/wiki/Welch%27s_t-test
26032858
[Worker threads]: worker_threads.md#worker-threads
26042859
[`'exit'`]: process.md#event-exit
26052860
[`child_process.spawnSync()`]: child_process.md#child_processspawnsynccommand-args-options
0 commit comments
Comments
 (0)

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Commit 9e8e908
Browse files
jasnelladuh95
jasnell
authored and
aduh95
committed
perf_hooks: add statistical hypothesis testing to histogram
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
1 parent dc51c79 commit 9e8e908
6 files changed
Lines changed: 1647 additions & 10 deletions
File tree
‎doc/api/perf_hooks.md‎
Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1625,6 +1625,16 @@ added:
16251625
**Default:**`Number.MAX_SAFE_INTEGER`.
16261626
*`figures` {number} The number of accuracy digits. Must be a number between
16271627
`1` and `5`. **Default:**`3`.
1628+
*`halfLife` {number} The EWMA half-life in number of samples. When set to
1629+
 a value greater than 0, the histogram tracks an exponentially weighted
1630+
 moving average and standard deviation, accessible via
1631+
`histogram.ewmaMean` and `histogram.ewmaStddev`. After `halfLife`
1632+
 recordings, a value's influence has decayed to 50%. **Default:**`0`
1633+
 (disabled).
1634+
*`threshold` {number} An SLO threshold value. When set together with
1635+
`halfLife`, the histogram tracks a smoothed error rate for values
1636+
 exceeding this threshold, accessible via `histogram.ewmaErrorRate` and
1637+
`histogram.burnRate()`. **Default:**`0` (disabled).
16281638
* Returns: {RecordableHistogram}
16291639

16301640
Returns a {RecordableHistogram}.
@@ -1893,6 +1903,36 @@ value, representing the probability that a recorded value will be less
18931903
than or equal to `value`. This is the inverse operation of
18941904
`histogram.percentile()`.
18951905

1906+
### `histogram.cliffsD(other)`
1907+
1908+
<!-- YAML
1909+
added: REPLACEME
1910+
-->
1911+
1912+
*`other` {Histogram} The histogram to compare against.
1913+
* Returns: {number} A value between -1.0 and 1.0.
1914+
1915+
Computes [Cliff's delta][], a non-parametric effect size measure. Returns
1916+
the probability that a random value from this histogram exceeds a random
1917+
value from `other`, minus the reverse probability. A value of 1 means every
1918+
value in this histogram exceeds every value in `other`; -1 means the
1919+
opposite; 0 means no tendency in either direction.
1920+
1921+
### `histogram.cohensD(other)`
1922+
1923+
<!-- YAML
1924+
added: REPLACEME
1925+
-->
1926+
1927+
*`other` {Histogram} The histogram to compare against.
1928+
* Returns: {number} The effect size.
1929+
1930+
Computes [Cohen's d][] effect size, the standardized difference between the
1931+
means of this histogram and `other`, using the pooled standard deviation.
1932+
Positive values indicate this histogram has a higher mean. By convention,
1933+
|d| < 0.2 is a small effect, 0.5 is medium, and 0.8 or greater is large.
1934+
Both histograms must have at least 2 recorded values; otherwise returns 0.
1935+
18961936
### `histogram.countAt(value)`
18971937

18981938
<!-- YAML
@@ -1929,6 +1969,74 @@ added:
19291969
The number of times the event loop delay exceeded the maximum 1 hour event
19301970
loop delay threshold.
19311971

1972+
### `histogram.ewmaMean`
1973+
1974+
<!-- YAML
1975+
added: REPLACEME
1976+
-->
1977+
1978+
* Type: {number}
1979+
1980+
The exponentially weighted moving average of recorded values. Only active
1981+
when the histogram was created with a `halfLife` option greater than 0.
1982+
Returns `0` when EWMA is disabled or no values have been recorded.
1983+
1984+
### `histogram.ewmaStddev`
1985+
1986+
<!-- YAML
1987+
added: REPLACEME
1988+
-->
1989+
1990+
* Type: {number}
1991+
1992+
The exponentially weighted moving standard deviation. Only active when the
1993+
histogram was created with a `halfLife` option greater than 0. Returns `0`
1994+
when EWMA is disabled or no values have been recorded.
1995+
1996+
### `histogram.ewmaErrorRate`
1997+
1998+
<!-- YAML
1999+
added: REPLACEME
2000+
-->
2001+
2002+
* Type: {number}
2003+
2004+
The EWMA-smoothed probability of a recorded value exceeding the configured
2005+
`threshold`. Only active when the histogram was created with both `halfLife`
2006+
and `threshold` options. Returns `0` when not enabled or no values have been
2007+
recorded.
2008+
2009+
### `histogram.burnRate(sloTarget)`
2010+
2011+
<!-- YAML
2012+
added: REPLACEME
2013+
-->
2014+
2015+
*`sloTarget` {number} The SLO target as a fraction between 0 and 1
2016+
 (exclusive). For example, `0.999` for a 99.9% SLO.
2017+
* Returns: {number}
2018+
2019+
Returns the SLO burn rate: `ewmaErrorRate / (1 - sloTarget)`. A burn rate
2020+
of 1 means the error budget will be exactly exhausted over the SLO window.
2021+
A burn rate greater than 1 means it is being consumed faster than allowed.
2022+
Requires the histogram to have been created with both `halfLife` and
2023+
`threshold` options.
2024+
2025+
```js
2026+
const { createHistogram } =require('node:perf_hooks');
2027+
2028+
// Track latency with a 200ms SLO threshold, half-life of 100 samples
2029+
consth=createHistogram({ halfLife:100, threshold:200_000_000 });
2030+
2031+
// ... record latency values ...
2032+
2033+
// Check burn rate against a 99.9% SLO
2034+
constrate=h.burnRate(0.999);
2035+
if (rate >1) {
2036+
console.log(`SLO burn rate: ${rate.toFixed(2)}x — error budget depleting`);
2037+
}
2038+
```
2039+
19322040
### `histogram.ksTest(other)`
19332041

19342042
<!-- YAML
@@ -1982,6 +2090,24 @@ Returns the histogram data rebucketed into logarithmically-spaced
19822090
intervals, where each bucket's width is multiplied by `base`.
19832091
Useful for visualization and export.
19842092

2093+
### `histogram.mannWhitneyTest(other)`
2094+
2095+
<!-- YAML
2096+
added: REPLACEME
2097+
-->
2098+
2099+
*`other` {Histogram} The histogram to compare against.
2100+
* Returns: {Object}
2101+
*`uStatistic` {number} The Mann-Whitney U statistic.
2102+
*`zScore` {number} The z-score (normal approximation).
2103+
*`pValue` {number} Two-tailed p-value.
2104+
2105+
Performs a [Mann-Whitney U test][] comparing whether this histogram tends to
2106+
produce larger or smaller values than `other`. Unlike `welchTest()`, this is a
2107+
non-parametric test that makes no assumptions about the shape of the
2108+
distributions. Uses the normal approximation with tie correction for the
2109+
p-value.
2110+
19852111
### `histogram.max`
19862112

19872113
<!-- YAML
@@ -2060,6 +2186,40 @@ added:
20602186

20612187
Returns the value at the given percentile.
20622188

2189+
### `histogram.percentileCI(percentile[, options])`
2190+
2191+
<!-- YAML
2192+
added: REPLACEME
2193+
-->
2194+
2195+
*`percentile` {number} A percentile value in the range (0, 100].
2196+
*`options` {Object}
2197+
*`confidence` {number} The confidence level for the interval, between
2198+
 0 and 1 (exclusive). **Default:**`0.95`.
2199+
* Returns: {Object}
2200+
*`value` {number} The point estimate (same as `histogram.percentile()`).
2201+
*`lower` {number} The lower bound of the confidence interval.
2202+
*`upper` {number} The upper bound of the confidence interval.
2203+
2204+
Returns a confidence interval for the given percentile using the exact
2205+
binomial method. With fewer samples, the interval will be wider, reflecting
2206+
the greater uncertainty in the percentile estimate. Requires at least 2
2207+
recorded values; with fewer than 2, `lower` and `upper` will equal `value`.
2208+
2209+
```js
2210+
const { createHistogram } =require('node:perf_hooks');
2211+
2212+
consth=createHistogram();
2213+
for (let i =0; i <1000; i++) {
2214+
h.record(Math.floor(Math.random() *100));
2215+
}
2216+
2217+
constci=h.percentileCI(99);
2218+
console.log(ci.value); // The p99 point estimate
2219+
console.log(ci.lower); // The lower bound (95% confidence)
2220+
console.log(ci.upper); // The upper bound (95% confidence)
2221+
```
2222+
20632223
### `histogram.percentiles`
20642224

20652225
<!-- YAML
@@ -2127,6 +2287,31 @@ added: v11.10.0
21272287

21282288
The standard deviation of the recorded event loop delays.
21292289

2290+
### `histogram.welchTest(other[, options])`
2291+
2292+
<!-- YAML
2293+
added: REPLACEME
2294+
-->
2295+
2296+
*`other` {Histogram} The histogram to compare against.
2297+
*`options` {Object}
2298+
*`confidence` {number} Confidence level for the interval, between 0 and 1.
2299+
**Default:**`0.95`.
2300+
* Returns: {Object}
2301+
*`tStatistic` {number} The Welch t-statistic.
2302+
*`degreesOfFreedom` {number} Welch-Satterthwaite degrees of freedom.
2303+
*`pValue` {number} Two-tailed p-value.
2304+
*`confidenceInterval` {Object}
2305+
*`lower` {number} Lower bound of the confidence interval on the
2306+
 difference of means.
2307+
*`upper` {number} Upper bound.
2308+
2309+
Performs [Welch's t-test][] comparing the means of this histogram and `other`.
2310+
The p-value indicates the probability of observing a difference at least this
2311+
extreme under the null hypothesis that the two distributions have the same
2312+
mean. Both histograms must have at least 2 recorded values; otherwise the
2313+
result has `pValue` 1 and `tStatistic` 0.
2314+
21302315
## Class: `ELDHistogram extends Histogram`
21312316

21322317
A `Histogram` that records event loop delay, returned by
@@ -2288,6 +2473,32 @@ const violating = latency.ccdf(500_000_000);
22882473
console.log(`${(violating *100).toFixed(1)}% of requests violating SLO`);
22892474
```
22902475

2476+
### SLO burn rate monitoring
2477+
2478+
```js
2479+
const { createHistogram } =require('node:perf_hooks');
2480+
2481+
// Track latency with EWMA (half-life 100 samples) and a 200ms SLO threshold
2482+
constlatency=createHistogram({
2483+
 halfLife:100,
2484+
 threshold:200_000_000, // 200ms in nanoseconds
2485+
});
2486+
2487+
// Record request latencies...
2488+
2489+
// Smoothed error rate: probability of exceeding the threshold
2490+
console.log(`Error rate: ${(latency.ewmaErrorRate*100).toFixed(2)}%`);
2491+
2492+
// Burn rate against a 99.9% SLO
2493+
// >1 means the error budget is depleting faster than allowed
2494+
constrate=latency.burnRate(0.999);
2495+
console.log(`Burn rate: ${rate.toFixed(2)}x`);
2496+
2497+
// EWMA mean and stddev track the smoothed latency
2498+
console.log(`EWMA latency: ${latency.ewmaMean.toFixed(0)}ns`);
2499+
console.log(`EWMA stddev: ${latency.ewmaStddev.toFixed(0)}ns`);
2500+
```
2501+
22912502
### Regression detection with KS test
22922503

22932504
```js
@@ -2339,6 +2550,46 @@ newSnapshot.subtract(snapshot);
23392550
console.log('Recent p99:', newSnapshot.percentile(99));
23402551
```
23412552

2553+
### Benchmark comparison with Welch's t-test
2554+
2555+
```js
2556+
const { createHistogram } =require('node:perf_hooks');
2557+
2558+
constbaseline=createHistogram();
2559+
constcandidate=createHistogram();
2560+
2561+
// Record operation rates from the old and new builds...
2562+
2563+
constresult=baseline.welchTest(candidate);
2564+
constimprovement= ((candidate.mean-baseline.mean) /baseline.mean*100);
2565+
2566+
console.log(`Improvement: ${improvement.toFixed(2)}%`);
2567+
console.log(`p-value: ${result.pValue.toFixed(6)}`);
2568+
console.log(`95% CI: [${result.confidenceInterval.lower.toFixed(2)}, `+
2569+
`${result.confidenceInterval.upper.toFixed(2)}]`);
2570+
2571+
if (result.pValue<0.05) {
2572+
constd=baseline.cohensD(candidate);
2573+
console.log(`Statistically significant (Cohen's d = ${d.toFixed(4)})`);
2574+
}
2575+
```
2576+
2577+
### Effect size with Cliff's delta
2578+
2579+
```js
2580+
const { createHistogram } =require('node:perf_hooks');
2581+
2582+
constbefore=createHistogram();
2583+
constafter=createHistogram();
2584+
2585+
// Record latencies before and after a change...
2586+
2587+
constdelta=before.cliffsD(after);
2588+
// A delta > 0: before tends to produce larger values (improvement)
2589+
// A delta < 0: after tends to produce larger values (regression)
2590+
console.log(`Cliff's delta: ${delta.toFixed(4)}`);
2591+
```
2592+
23422593
## Examples
23432594

23442595
### Measuring the duration of async operations
@@ -2593,13 +2844,17 @@ dns.promises.resolve('localhost');
25932844
```
25942845

25952846
[Async Hooks]: async_hooks.md
2847+
[Cliff's delta]: https://en.wikipedia.org/wiki/Effect_size#Cliff's_delta
2848+
[Cohen's d]: https://en.wikipedia.org/wiki/Effect_size#Cohen's_d
25962849
[Fetch Response Body Info]: https://fetch.spec.whatwg.org/#response-body-info
25972850
[Fetch Timing Info]: https://fetch.spec.whatwg.org/#fetch-timing-info
25982851
[High Resolution Time]: https://www.w3.org/TR/hr-time-2
2852+
[Mann-Whitney U test]: https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test
25992853
[Performance Timeline]: https://w3c.github.io/performance-timeline/
26002854
[Resource Timing]: https://www.w3.org/TR/resource-timing-2/
26012855
[User Timing]: https://www.w3.org/TR/user-timing/
26022856
[Web Performance APIs]: https://w3c.github.io/perf-timing-primer/
2857+
[Welch's t-test]: https://en.wikipedia.org/wiki/Welch%27s_t-test
26032858
[Worker threads]: worker_threads.md#worker-threads
26042859
[`'exit'`]: process.md#event-exit
26052860
[`child_process.spawnSync()`]: child_process.md#child_processspawnsynccommand-args-options
0 commit comments
Comments
 (0)

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Commit 9e8e908
Browse files
jasnelladuh95
jasnell
authored and
aduh95
committed
perf_hooks: add statistical hypothesis testing to histogram
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
1 parent dc51c79 commit 9e8e908
6 files changed
Lines changed: 1647 additions & 10 deletions
File tree
‎doc/api/perf_hooks.md‎
Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1625,6 +1625,16 @@ added:
16251625
**Default:**`Number.MAX_SAFE_INTEGER`.
16261626
*`figures` {number} The number of accuracy digits. Must be a number between
16271627
`1` and `5`. **Default:**`3`.
1628+
*`halfLife` {number} The EWMA half-life in number of samples. When set to
1629+
 a value greater than 0, the histogram tracks an exponentially weighted
1630+
 moving average and standard deviation, accessible via
1631+
`histogram.ewmaMean` and `histogram.ewmaStddev`. After `halfLife`
1632+
 recordings, a value's influence has decayed to 50%. **Default:**`0`
1633+
 (disabled).
1634+
*`threshold` {number} An SLO threshold value. When set together with
1635+
`halfLife`, the histogram tracks a smoothed error rate for values
1636+
 exceeding this threshold, accessible via `histogram.ewmaErrorRate` and
1637+
`histogram.burnRate()`. **Default:**`0` (disabled).
16281638
* Returns: {RecordableHistogram}
16291639

16301640
Returns a {RecordableHistogram}.
@@ -1893,6 +1903,36 @@ value, representing the probability that a recorded value will be less
18931903
than or equal to `value`. This is the inverse operation of
18941904
`histogram.percentile()`.
18951905

1906+
### `histogram.cliffsD(other)`
1907+
1908+
<!-- YAML
1909+
added: REPLACEME
1910+
-->
1911+
1912+
*`other` {Histogram} The histogram to compare against.
1913+
* Returns: {number} A value between -1.0 and 1.0.
1914+
1915+
Computes [Cliff's delta][], a non-parametric effect size measure. Returns
1916+
the probability that a random value from this histogram exceeds a random
1917+
value from `other`, minus the reverse probability. A value of 1 means every
1918+
value in this histogram exceeds every value in `other`; -1 means the
1919+
opposite; 0 means no tendency in either direction.
1920+
1921+
### `histogram.cohensD(other)`
1922+
1923+
<!-- YAML
1924+
added: REPLACEME
1925+
-->
1926+
1927+
*`other` {Histogram} The histogram to compare against.
1928+
* Returns: {number} The effect size.
1929+
1930+
Computes [Cohen's d][] effect size, the standardized difference between the
1931+
means of this histogram and `other`, using the pooled standard deviation.
1932+
Positive values indicate this histogram has a higher mean. By convention,
1933+
|d| < 0.2 is a small effect, 0.5 is medium, and 0.8 or greater is large.
1934+
Both histograms must have at least 2 recorded values; otherwise returns 0.
1935+
18961936
### `histogram.countAt(value)`
18971937

18981938
<!-- YAML
@@ -1929,6 +1969,74 @@ added:
19291969
The number of times the event loop delay exceeded the maximum 1 hour event
19301970
loop delay threshold.
19311971

1972+
### `histogram.ewmaMean`
1973+
1974+
<!-- YAML
1975+
added: REPLACEME
1976+
-->
1977+
1978+
* Type: {number}
1979+
1980+
The exponentially weighted moving average of recorded values. Only active
1981+
when the histogram was created with a `halfLife` option greater than 0.
1982+
Returns `0` when EWMA is disabled or no values have been recorded.
1983+
1984+
### `histogram.ewmaStddev`
1985+
1986+
<!-- YAML
1987+
added: REPLACEME
1988+
-->
1989+
1990+
* Type: {number}
1991+
1992+
The exponentially weighted moving standard deviation. Only active when the
1993+
histogram was created with a `halfLife` option greater than 0. Returns `0`
1994+
when EWMA is disabled or no values have been recorded.
1995+
1996+
### `histogram.ewmaErrorRate`
1997+
1998+
<!-- YAML
1999+
added: REPLACEME
2000+
-->
2001+
2002+
* Type: {number}
2003+
2004+
The EWMA-smoothed probability of a recorded value exceeding the configured
2005+
`threshold`. Only active when the histogram was created with both `halfLife`
2006+
and `threshold` options. Returns `0` when not enabled or no values have been
2007+
recorded.
2008+
2009+
### `histogram.burnRate(sloTarget)`
2010+
2011+
<!-- YAML
2012+
added: REPLACEME
2013+
-->
2014+
2015+
*`sloTarget` {number} The SLO target as a fraction between 0 and 1
2016+
 (exclusive). For example, `0.999` for a 99.9% SLO.
2017+
* Returns: {number}
2018+
2019+
Returns the SLO burn rate: `ewmaErrorRate / (1 - sloTarget)`. A burn rate
2020+
of 1 means the error budget will be exactly exhausted over the SLO window.
2021+
A burn rate greater than 1 means it is being consumed faster than allowed.
2022+
Requires the histogram to have been created with both `halfLife` and
2023+
`threshold` options.
2024+
2025+
```js
2026+
const { createHistogram } =require('node:perf_hooks');
2027+
2028+
// Track latency with a 200ms SLO threshold, half-life of 100 samples
2029+
consth=createHistogram({ halfLife:100, threshold:200_000_000 });
2030+
2031+
// ... record latency values ...
2032+
2033+
// Check burn rate against a 99.9% SLO
2034+
constrate=h.burnRate(0.999);
2035+
if (rate >1) {
2036+
console.log(`SLO burn rate: ${rate.toFixed(2)}x — error budget depleting`);
2037+
}
2038+
```
2039+
19322040
### `histogram.ksTest(other)`
19332041

19342042
<!-- YAML
@@ -1982,6 +2090,24 @@ Returns the histogram data rebucketed into logarithmically-spaced
19822090
intervals, where each bucket's width is multiplied by `base`.
19832091
Useful for visualization and export.
19842092

2093+
### `histogram.mannWhitneyTest(other)`
2094+
2095+
<!-- YAML
2096+
added: REPLACEME
2097+
-->
2098+
2099+
*`other` {Histogram} The histogram to compare against.
2100+
* Returns: {Object}
2101+
*`uStatistic` {number} The Mann-Whitney U statistic.
2102+
*`zScore` {number} The z-score (normal approximation).
2103+
*`pValue` {number} Two-tailed p-value.
2104+
2105+
Performs a [Mann-Whitney U test][] comparing whether this histogram tends to
2106+
produce larger or smaller values than `other`. Unlike `welchTest()`, this is a
2107+
non-parametric test that makes no assumptions about the shape of the
2108+
distributions. Uses the normal approximation with tie correction for the
2109+
p-value.
2110+
19852111
### `histogram.max`
19862112

19872113
<!-- YAML
@@ -2060,6 +2186,40 @@ added:
20602186

20612187
Returns the value at the given percentile.
20622188

2189+
### `histogram.percentileCI(percentile[, options])`
2190+
2191+
<!-- YAML
2192+
added: REPLACEME
2193+
-->
2194+
2195+
*`percentile` {number} A percentile value in the range (0, 100].
2196+
*`options` {Object}
2197+
*`confidence` {number} The confidence level for the interval, between
2198+
 0 and 1 (exclusive). **Default:**`0.95`.
2199+
* Returns: {Object}
2200+
*`value` {number} The point estimate (same as `histogram.percentile()`).
2201+
*`lower` {number} The lower bound of the confidence interval.
2202+
*`upper` {number} The upper bound of the confidence interval.
2203+
2204+
Returns a confidence interval for the given percentile using the exact
2205+
binomial method. With fewer samples, the interval will be wider, reflecting
2206+
the greater uncertainty in the percentile estimate. Requires at least 2
2207+
recorded values; with fewer than 2, `lower` and `upper` will equal `value`.
2208+
2209+
```js
2210+
const { createHistogram } =require('node:perf_hooks');
2211+
2212+
consth=createHistogram();
2213+
for (let i =0; i <1000; i++) {
2214+
h.record(Math.floor(Math.random() *100));
2215+
}
2216+
2217+
constci=h.percentileCI(99);
2218+
console.log(ci.value); // The p99 point estimate
2219+
console.log(ci.lower); // The lower bound (95% confidence)
2220+
console.log(ci.upper); // The upper bound (95% confidence)
2221+
```
2222+
20632223
### `histogram.percentiles`
20642224

20652225
<!-- YAML
@@ -2127,6 +2287,31 @@ added: v11.10.0
21272287

21282288
The standard deviation of the recorded event loop delays.
21292289

2290+
### `histogram.welchTest(other[, options])`
2291+
2292+
<!-- YAML
2293+
added: REPLACEME
2294+
-->
2295+
2296+
*`other` {Histogram} The histogram to compare against.
2297+
*`options` {Object}
2298+
*`confidence` {number} Confidence level for the interval, between 0 and 1.
2299+
**Default:**`0.95`.
2300+
* Returns: {Object}
2301+
*`tStatistic` {number} The Welch t-statistic.
2302+
*`degreesOfFreedom` {number} Welch-Satterthwaite degrees of freedom.
2303+
*`pValue` {number} Two-tailed p-value.
2304+
*`confidenceInterval` {Object}
2305+
*`lower` {number} Lower bound of the confidence interval on the
2306+
 difference of means.
2307+
*`upper` {number} Upper bound.
2308+
2309+
Performs [Welch's t-test][] comparing the means of this histogram and `other`.
2310+
The p-value indicates the probability of observing a difference at least this
2311+
extreme under the null hypothesis that the two distributions have the same
2312+
mean. Both histograms must have at least 2 recorded values; otherwise the
2313+
result has `pValue` 1 and `tStatistic` 0.
2314+
21302315
## Class: `ELDHistogram extends Histogram`
21312316

21322317
A `Histogram` that records event loop delay, returned by
@@ -2288,6 +2473,32 @@ const violating = latency.ccdf(500_000_000);
22882473
console.log(`${(violating *100).toFixed(1)}% of requests violating SLO`);
22892474
```
22902475

2476+
### SLO burn rate monitoring
2477+
2478+
```js
2479+
const { createHistogram } =require('node:perf_hooks');
2480+
2481+
// Track latency with EWMA (half-life 100 samples) and a 200ms SLO threshold
2482+
constlatency=createHistogram({
2483+
 halfLife:100,
2484+
 threshold:200_000_000, // 200ms in nanoseconds
2485+
});
2486+
2487+
// Record request latencies...
2488+
2489+
// Smoothed error rate: probability of exceeding the threshold
2490+
console.log(`Error rate: ${(latency.ewmaErrorRate*100).toFixed(2)}%`);
2491+
2492+
// Burn rate against a 99.9% SLO
2493+
// >1 means the error budget is depleting faster than allowed
2494+
constrate=latency.burnRate(0.999);
2495+
console.log(`Burn rate: ${rate.toFixed(2)}x`);
2496+
2497+
// EWMA mean and stddev track the smoothed latency
2498+
console.log(`EWMA latency: ${latency.ewmaMean.toFixed(0)}ns`);
2499+
console.log(`EWMA stddev: ${latency.ewmaStddev.toFixed(0)}ns`);
2500+
```
2501+
22912502
### Regression detection with KS test
22922503

22932504
```js
@@ -2339,6 +2550,46 @@ newSnapshot.subtract(snapshot);
23392550
console.log('Recent p99:', newSnapshot.percentile(99));
23402551
```
23412552

2553+
### Benchmark comparison with Welch's t-test
2554+
2555+
```js
2556+
const { createHistogram } =require('node:perf_hooks');
2557+
2558+
constbaseline=createHistogram();
2559+
constcandidate=createHistogram();
2560+
2561+
// Record operation rates from the old and new builds...
2562+
2563+
constresult=baseline.welchTest(candidate);
2564+
constimprovement= ((candidate.mean-baseline.mean) /baseline.mean*100);
2565+
2566+
console.log(`Improvement: ${improvement.toFixed(2)}%`);
2567+
console.log(`p-value: ${result.pValue.toFixed(6)}`);
2568+
console.log(`95% CI: [${result.confidenceInterval.lower.toFixed(2)}, `+
2569+
`${result.confidenceInterval.upper.toFixed(2)}]`);
2570+
2571+
if (result.pValue<0.05) {
2572+
constd=baseline.cohensD(candidate);
2573+
console.log(`Statistically significant (Cohen's d = ${d.toFixed(4)})`);
2574+
}
2575+
```
2576+
2577+
### Effect size with Cliff's delta
2578+
2579+
```js
2580+
const { createHistogram } =require('node:perf_hooks');
2581+
2582+
constbefore=createHistogram();
2583+
constafter=createHistogram();
2584+
2585+
// Record latencies before and after a change...
2586+
2587+
constdelta=before.cliffsD(after);
2588+
// A delta > 0: before tends to produce larger values (improvement)
2589+
// A delta < 0: after tends to produce larger values (regression)
2590+
console.log(`Cliff's delta: ${delta.toFixed(4)}`);
2591+
```
2592+
23422593
## Examples
23432594

23442595
### Measuring the duration of async operations
@@ -2593,13 +2844,17 @@ dns.promises.resolve('localhost');
25932844
```
25942845

25952846
[Async Hooks]: async_hooks.md
2847+
[Cliff's delta]: https://en.wikipedia.org/wiki/Effect_size#Cliff's_delta
2848+
[Cohen's d]: https://en.wikipedia.org/wiki/Effect_size#Cohen's_d
25962849
[Fetch Response Body Info]: https://fetch.spec.whatwg.org/#response-body-info
25972850
[Fetch Timing Info]: https://fetch.spec.whatwg.org/#fetch-timing-info
25982851
[High Resolution Time]: https://www.w3.org/TR/hr-time-2
2852+
[Mann-Whitney U test]: https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test
25992853
[Performance Timeline]: https://w3c.github.io/performance-timeline/
26002854
[Resource Timing]: https://www.w3.org/TR/resource-timing-2/
26012855
[User Timing]: https://www.w3.org/TR/user-timing/
26022856
[Web Performance APIs]: https://w3c.github.io/perf-timing-primer/
2857+
[Welch's t-test]: https://en.wikipedia.org/wiki/Welch%27s_t-test
26032858
[Worker threads]: worker_threads.md#worker-threads
26042859
[`'exit'`]: process.md#event-exit
26052860
[`child_process.spawnSync()`]: child_process.md#child_processspawnsynccommand-args-options
0 commit comments
Comments
 (0)

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Commit 9e8e908
Browse files
jasnelladuh95
jasnell
authored and
aduh95
committed
perf_hooks: add statistical hypothesis testing to histogram
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
1 parent dc51c79 commit 9e8e908
6 files changed
Lines changed: 1647 additions & 10 deletions
File tree
‎doc/api/perf_hooks.md‎
Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1625,6 +1625,16 @@ added:
16251625
**Default:**`Number.MAX_SAFE_INTEGER`.
16261626
*`figures` {number} The number of accuracy digits. Must be a number between
16271627
`1` and `5`. **Default:**`3`.
1628+
*`halfLife` {number} The EWMA half-life in number of samples. When set to
1629+
 a value greater than 0, the histogram tracks an exponentially weighted
1630+
 moving average and standard deviation, accessible via
1631+
`histogram.ewmaMean` and `histogram.ewmaStddev`. After `halfLife`
1632+
 recordings, a value's influence has decayed to 50%. **Default:**`0`
1633+
 (disabled).
1634+
*`threshold` {number} An SLO threshold value. When set together with
1635+
`halfLife`, the histogram tracks a smoothed error rate for values
1636+
 exceeding this threshold, accessible via `histogram.ewmaErrorRate` and
1637+
`histogram.burnRate()`. **Default:**`0` (disabled).
16281638
* Returns: {RecordableHistogram}
16291639

16301640
Returns a {RecordableHistogram}.
@@ -1893,6 +1903,36 @@ value, representing the probability that a recorded value will be less
18931903
than or equal to `value`. This is the inverse operation of
18941904
`histogram.percentile()`.
18951905

1906+
### `histogram.cliffsD(other)`
1907+
1908+
<!-- YAML
1909+
added: REPLACEME
1910+
-->
1911+
1912+
*`other` {Histogram} The histogram to compare against.
1913+
* Returns: {number} A value between -1.0 and 1.0.
1914+
1915+
Computes [Cliff's delta][], a non-parametric effect size measure. Returns
1916+
the probability that a random value from this histogram exceeds a random
1917+
value from `other`, minus the reverse probability. A value of 1 means every
1918+
value in this histogram exceeds every value in `other`; -1 means the
1919+
opposite; 0 means no tendency in either direction.
1920+
1921+
### `histogram.cohensD(other)`
1922+
1923+
<!-- YAML
1924+
added: REPLACEME
1925+
-->
1926+
1927+
*`other` {Histogram} The histogram to compare against.
1928+
* Returns: {number} The effect size.
1929+
1930+
Computes [Cohen's d][] effect size, the standardized difference between the
1931+
means of this histogram and `other`, using the pooled standard deviation.
1932+
Positive values indicate this histogram has a higher mean. By convention,
1933+
|d| < 0.2 is a small effect, 0.5 is medium, and 0.8 or greater is large.
1934+
Both histograms must have at least 2 recorded values; otherwise returns 0.
1935+
18961936
### `histogram.countAt(value)`
18971937

18981938
<!-- YAML
@@ -1929,6 +1969,74 @@ added:
19291969
The number of times the event loop delay exceeded the maximum 1 hour event
19301970
loop delay threshold.
19311971

1972+
### `histogram.ewmaMean`
1973+
1974+
<!-- YAML
1975+
added: REPLACEME
1976+
-->
1977+
1978+
* Type: {number}
1979+
1980+
The exponentially weighted moving average of recorded values. Only active
1981+
when the histogram was created with a `halfLife` option greater than 0.
1982+
Returns `0` when EWMA is disabled or no values have been recorded.
1983+
1984+
### `histogram.ewmaStddev`
1985+
1986+
<!-- YAML
1987+
added: REPLACEME
1988+
-->
1989+
1990+
* Type: {number}
1991+
1992+
The exponentially weighted moving standard deviation. Only active when the
1993+
histogram was created with a `halfLife` option greater than 0. Returns `0`
1994+
when EWMA is disabled or no values have been recorded.
1995+
1996+
### `histogram.ewmaErrorRate`
1997+
1998+
<!-- YAML
1999+
added: REPLACEME
2000+
-->
2001+
2002+
* Type: {number}
2003+
2004+
The EWMA-smoothed probability of a recorded value exceeding the configured
2005+
`threshold`. Only active when the histogram was created with both `halfLife`
2006+
and `threshold` options. Returns `0` when not enabled or no values have been
2007+
recorded.
2008+
2009+
### `histogram.burnRate(sloTarget)`
2010+
2011+
<!-- YAML
2012+
added: REPLACEME
2013+
-->
2014+
2015+
*`sloTarget` {number} The SLO target as a fraction between 0 and 1
2016+
 (exclusive). For example, `0.999` for a 99.9% SLO.
2017+
* Returns: {number}
2018+
2019+
Returns the SLO burn rate: `ewmaErrorRate / (1 - sloTarget)`. A burn rate
2020+
of 1 means the error budget will be exactly exhausted over the SLO window.
2021+
A burn rate greater than 1 means it is being consumed faster than allowed.
2022+
Requires the histogram to have been created with both `halfLife` and
2023+
`threshold` options.
2024+
2025+
```js
2026+
const { createHistogram } =require('node:perf_hooks');
2027+
2028+
// Track latency with a 200ms SLO threshold, half-life of 100 samples
2029+
consth=createHistogram({ halfLife:100, threshold:200_000_000 });
2030+
2031+
// ... record latency values ...
2032+
2033+
// Check burn rate against a 99.9% SLO
2034+
constrate=h.burnRate(0.999);
2035+
if (rate >1) {
2036+
console.log(`SLO burn rate: ${rate.toFixed(2)}x — error budget depleting`);
2037+
}
2038+
```
2039+
19322040
### `histogram.ksTest(other)`
19332041

19342042
<!-- YAML
@@ -1982,6 +2090,24 @@ Returns the histogram data rebucketed into logarithmically-spaced
19822090
intervals, where each bucket's width is multiplied by `base`.
19832091
Useful for visualization and export.
19842092

2093+
### `histogram.mannWhitneyTest(other)`
2094+
2095+
<!-- YAML
2096+
added: REPLACEME
2097+
-->
2098+
2099+
*`other` {Histogram} The histogram to compare against.
2100+
* Returns: {Object}
2101+
*`uStatistic` {number} The Mann-Whitney U statistic.
2102+
*`zScore` {number} The z-score (normal approximation).
2103+
*`pValue` {number} Two-tailed p-value.
2104+
2105+
Performs a [Mann-Whitney U test][] comparing whether this histogram tends to
2106+
produce larger or smaller values than `other`. Unlike `welchTest()`, this is a
2107+
non-parametric test that makes no assumptions about the shape of the
2108+
distributions. Uses the normal approximation with tie correction for the
2109+
p-value.
2110+
19852111
### `histogram.max`
19862112

19872113
<!-- YAML
@@ -2060,6 +2186,40 @@ added:
20602186

20612187
Returns the value at the given percentile.
20622188

2189+
### `histogram.percentileCI(percentile[, options])`
2190+
2191+
<!-- YAML
2192+
added: REPLACEME
2193+
-->
2194+
2195+
*`percentile` {number} A percentile value in the range (0, 100].
2196+
*`options` {Object}
2197+
*`confidence` {number} The confidence level for the interval, between
2198+
 0 and 1 (exclusive). **Default:**`0.95`.
2199+
* Returns: {Object}
2200+
*`value` {number} The point estimate (same as `histogram.percentile()`).
2201+
*`lower` {number} The lower bound of the confidence interval.
2202+
*`upper` {number} The upper bound of the confidence interval.
2203+
2204+
Returns a confidence interval for the given percentile using the exact
2205+
binomial method. With fewer samples, the interval will be wider, reflecting
2206+
the greater uncertainty in the percentile estimate. Requires at least 2
2207+
recorded values; with fewer than 2, `lower` and `upper` will equal `value`.
2208+
2209+
```js
2210+
const { createHistogram } =require('node:perf_hooks');
2211+
2212+
consth=createHistogram();
2213+
for (let i =0; i <1000; i++) {
2214+
h.record(Math.floor(Math.random() *100));
2215+
}
2216+
2217+
constci=h.percentileCI(99);
2218+
console.log(ci.value); // The p99 point estimate
2219+
console.log(ci.lower); // The lower bound (95% confidence)
2220+
console.log(ci.upper); // The upper bound (95% confidence)
2221+
```
2222+
20632223
### `histogram.percentiles`
20642224

20652225
<!-- YAML
@@ -2127,6 +2287,31 @@ added: v11.10.0
21272287

21282288
The standard deviation of the recorded event loop delays.
21292289

2290+
### `histogram.welchTest(other[, options])`
2291+
2292+
<!-- YAML
2293+
added: REPLACEME
2294+
-->
2295+
2296+
*`other` {Histogram} The histogram to compare against.
2297+
*`options` {Object}
2298+
*`confidence` {number} Confidence level for the interval, between 0 and 1.
2299+
**Default:**`0.95`.
2300+
* Returns: {Object}
2301+
*`tStatistic` {number} The Welch t-statistic.
2302+
*`degreesOfFreedom` {number} Welch-Satterthwaite degrees of freedom.
2303+
*`pValue` {number} Two-tailed p-value.
2304+
*`confidenceInterval` {Object}
2305+
*`lower` {number} Lower bound of the confidence interval on the
2306+
 difference of means.
2307+
*`upper` {number} Upper bound.
2308+
2309+
Performs [Welch's t-test][] comparing the means of this histogram and `other`.
2310+
The p-value indicates the probability of observing a difference at least this
2311+
extreme under the null hypothesis that the two distributions have the same
2312+
mean. Both histograms must have at least 2 recorded values; otherwise the
2313+
result has `pValue` 1 and `tStatistic` 0.
2314+
21302315
## Class: `ELDHistogram extends Histogram`
21312316

21322317
A `Histogram` that records event loop delay, returned by
@@ -2288,6 +2473,32 @@ const violating = latency.ccdf(500_000_000);
22882473
console.log(`${(violating *100).toFixed(1)}% of requests violating SLO`);
22892474
```
22902475

2476+
### SLO burn rate monitoring
2477+
2478+
```js
2479+
const { createHistogram } =require('node:perf_hooks');
2480+
2481+
// Track latency with EWMA (half-life 100 samples) and a 200ms SLO threshold
2482+
constlatency=createHistogram({
2483+
 halfLife:100,
2484+
 threshold:200_000_000, // 200ms in nanoseconds
2485+
});
2486+
2487+
// Record request latencies...
2488+
2489+
// Smoothed error rate: probability of exceeding the threshold
2490+
console.log(`Error rate: ${(latency.ewmaErrorRate*100).toFixed(2)}%`);
2491+
2492+
// Burn rate against a 99.9% SLO
2493+
// >1 means the error budget is depleting faster than allowed
2494+
constrate=latency.burnRate(0.999);
2495+
console.log(`Burn rate: ${rate.toFixed(2)}x`);
2496+
2497+
// EWMA mean and stddev track the smoothed latency
2498+
console.log(`EWMA latency: ${latency.ewmaMean.toFixed(0)}ns`);
2499+
console.log(`EWMA stddev: ${latency.ewmaStddev.toFixed(0)}ns`);
2500+
```
2501+
22912502
### Regression detection with KS test
22922503

22932504
```js
@@ -2339,6 +2550,46 @@ newSnapshot.subtract(snapshot);
23392550
console.log('Recent p99:', newSnapshot.percentile(99));
23402551
```
23412552

2553+
### Benchmark comparison with Welch's t-test
2554+
2555+
```js
2556+
const { createHistogram } =require('node:perf_hooks');
2557+
2558+
constbaseline=createHistogram();
2559+
constcandidate=createHistogram();
2560+
2561+
// Record operation rates from the old and new builds...
2562+
2563+
constresult=baseline.welchTest(candidate);
2564+
constimprovement= ((candidate.mean-baseline.mean) /baseline.mean*100);
2565+
2566+
console.log(`Improvement: ${improvement.toFixed(2)}%`);
2567+
console.log(`p-value: ${result.pValue.toFixed(6)}`);
2568+
console.log(`95% CI: [${result.confidenceInterval.lower.toFixed(2)}, `+
2569+
`${result.confidenceInterval.upper.toFixed(2)}]`);
2570+
2571+
if (result.pValue<0.05) {
2572+
constd=baseline.cohensD(candidate);
2573+
console.log(`Statistically significant (Cohen's d = ${d.toFixed(4)})`);
2574+
}
2575+
```
2576+
2577+
### Effect size with Cliff's delta
2578+
2579+
```js
2580+
const { createHistogram } =require('node:perf_hooks');
2581+
2582+
constbefore=createHistogram();
2583+
constafter=createHistogram();
2584+
2585+
// Record latencies before and after a change...
2586+
2587+
constdelta=before.cliffsD(after);
2588+
// A delta > 0: before tends to produce larger values (improvement)
2589+
// A delta < 0: after tends to produce larger values (regression)
2590+
console.log(`Cliff's delta: ${delta.toFixed(4)}`);
2591+
```
2592+
23422593
## Examples
23432594

23442595
### Measuring the duration of async operations
@@ -2593,13 +2844,17 @@ dns.promises.resolve('localhost');
25932844
```
25942845

25952846
[Async Hooks]: async_hooks.md
2847+
[Cliff's delta]: https://en.wikipedia.org/wiki/Effect_size#Cliff's_delta
2848+
[Cohen's d]: https://en.wikipedia.org/wiki/Effect_size#Cohen's_d
25962849
[Fetch Response Body Info]: https://fetch.spec.whatwg.org/#response-body-info
25972850
[Fetch Timing Info]: https://fetch.spec.whatwg.org/#fetch-timing-info
25982851
[High Resolution Time]: https://www.w3.org/TR/hr-time-2
2852+
[Mann-Whitney U test]: https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test
25992853
[Performance Timeline]: https://w3c.github.io/performance-timeline/
26002854
[Resource Timing]: https://www.w3.org/TR/resource-timing-2/
26012855
[User Timing]: https://www.w3.org/TR/user-timing/
26022856
[Web Performance APIs]: https://w3c.github.io/perf-timing-primer/
2857+
[Welch's t-test]: https://en.wikipedia.org/wiki/Welch%27s_t-test
26032858
[Worker threads]: worker_threads.md#worker-threads
26042859
[`'exit'`]: process.md#event-exit
26052860
[`child_process.spawnSync()`]: child_process.md#child_processspawnsynccommand-args-options
0 commit comments
Comments
 (0)

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Commit 9e8e908
Browse files
jasnelladuh95
jasnell
authored and
aduh95
committed
perf_hooks: add statistical hypothesis testing to histogram
Welch's t-test, Mann-Whitney U test, Cohen's d, and Cliff's delta,
and and handful of others
These methods enable in-process benchmark comparison and regression
detection without external dependencies.
No new dependencies.
Tests and docs created by the AI agent.
Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode/Opus
PR-URL: #65416
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
1 parent dc51c79 commit 9e8e908
6 files changed
Lines changed: 1647 additions & 10 deletions
File tree
‎doc/api/perf_hooks.md‎
Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1625,6 +1625,16 @@ added:
16251625
**Default:**`Number.MAX_SAFE_INTEGER`.
16261626
*`figures` {number} The number of accuracy digits. Must be a number between
16271627
`1` and `5`. **Default:**`3`.
1628+
*`halfLife` {number} The EWMA half-life in number of samples. When set to
1629+
 a value greater than 0, the histogram tracks an exponentially weighted
1630+
 moving average and standard deviation, accessible via
1631+
`histogram.ewmaMean` and `histogram.ewmaStddev`. After `halfLife`
1632+
 recordings, a value's influence has decayed to 50%. **Default:**`0`
1633+
 (disabled).
1634+
*`threshold` {number} An SLO threshold value. When set together with
1635+
`halfLife`, the histogram tracks a smoothed error rate for values
1636+
 exceeding this threshold, accessible via `histogram.ewmaErrorRate` and
1637+
`histogram.burnRate()`. **Default:**`0` (disabled).
16281638
* Returns: {RecordableHistogram}
16291639

16301640
Returns a {RecordableHistogram}.
@@ -1893,6 +1903,36 @@ value, representing the probability that a recorded value will be less
18931903
than or equal to `value`. This is the inverse operation of
18941904
`histogram.percentile()`.
18951905

1906+
### `histogram.cliffsD(other)`
1907+
1908+
<!-- YAML
1909+
added: REPLACEME
1910+
-->
1911+
1912+
*`other` {Histogram} The histogram to compare against.
1913+
* Returns: {number} A value between -1.0 and 1.0.
1914+
1915+
Computes [Cliff's delta][], a non-parametric effect size measure. Returns
1916+
the probability that a random value from this histogram exceeds a random
1917+
value from `other`, minus the reverse probability. A value of 1 means every
1918+
value in this histogram exceeds every value in `other`; -1 means the
1919+
opposite; 0 means no tendency in either direction.
1920+
1921+
### `histogram.cohensD(other)`
1922+
1923+
<!-- YAML
1924+
added: REPLACEME
1925+
-->
1926+
1927+
*`other` {Histogram} The histogram to compare against.
1928+
* Returns: {number} The effect size.
1929+
1930+
Computes [Cohen's d][] effect size, the standardized difference between the
1931+
means of this histogram and `other`, using the pooled standard deviation.
1932+
Positive values indicate this histogram has a higher mean. By convention,
1933+
|d| < 0.2 is a small effect, 0.5 is medium, and 0.8 or greater is large.
1934+
Both histograms must have at least 2 recorded values; otherwise returns 0.
1935+
18961936
### `histogram.countAt(value)`
18971937

18981938
<!-- YAML
@@ -1929,6 +1969,74 @@ added:
19291969
The number of times the event loop delay exceeded the maximum 1 hour event
19301970
loop delay threshold.
19311971

1972+
### `histogram.ewmaMean`
1973+
1974+
<!-- YAML
1975+
added: REPLACEME
1976+
-->
1977+
1978+
* Type: {number}
1979+
1980+
The exponentially weighted moving average of recorded values. Only active
1981+
when the histogram was created with a `halfLife` option greater than 0.
1982+
Returns `0` when EWMA is disabled or no values have been recorded.
1983+
1984+
### `histogram.ewmaStddev`
1985+
1986+
<!-- YAML
1987+
added: REPLACEME
1988+
-->
1989+
1990+
* Type: {number}
1991+
1992+
The exponentially weighted moving standard deviation. Only active when the
1993+
histogram was created with a `halfLife` option greater than 0. Returns `0`
1994+
when EWMA is disabled or no values have been recorded.
1995+
1996+
### `histogram.ewmaErrorRate`
1997+
1998+
<!-- YAML
1999+
added: REPLACEME
2000+
-->
2001+
2002+
* Type: {number}
2003+
2004+
The EWMA-smoothed probability of a recorded value exceeding the configured
2005+
`threshold`. Only active when the histogram was created with both `halfLife`
2006+
and `threshold` options. Returns `0` when not enabled or no values have been
2007+
recorded.
2008+
2009+
### `histogram.burnRate(sloTarget)`
2010+
2011+
<!-- YAML
2012+
added: REPLACEME
2013+
-->
2014+
2015+
*`sloTarget` {number} The SLO target as a fraction between 0 and 1
2016+
 (exclusive). For example, `0.999` for a 99.9% SLO.
2017+
* Returns: {number}
2018+
2019+
Returns the SLO burn rate: `ewmaErrorRate / (1 - sloTarget)`. A burn rate
2020+
of 1 means the error budget will be exactly exhausted over the SLO window.
2021+
A burn rate greater than 1 means it is being consumed faster than allowed.
2022+
Requires the histogram to have been created with both `halfLife` and
2023+
`threshold` options.
2024+
2025+
```js
2026+
const { createHistogram } =require('node:perf_hooks');
2027+
2028+
// Track latency with a 200ms SLO threshold, half-life of 100 samples
2029+
consth=createHistogram({ halfLife:100, threshold:200_000_000 });
2030+
2031+
// ... record latency values ...
2032+
2033+
// Check burn rate against a 99.9% SLO
2034+
constrate=h.burnRate(0.999);
2035+
if (rate >1) {
2036+
console.log(`SLO burn rate: ${rate.toFixed(2)}x — error budget depleting`);
2037+
}
2038+
```
2039+
19322040
### `histogram.ksTest(other)`
19332041

19342042
<!-- YAML
@@ -1982,6 +2090,24 @@ Returns the histogram data rebucketed into logarithmically-spaced
19822090
intervals, where each bucket's width is multiplied by `base`.
19832091
Useful for visualization and export.
19842092

2093+
### `histogram.mannWhitneyTest(other)`
2094+
2095+
<!-- YAML
2096+
added: REPLACEME
2097+
-->
2098+
2099+
*`other` {Histogram} The histogram to compare against.
2100+
* Returns: {Object}
2101+
*`uStatistic` {number} The Mann-Whitney U statistic.
2102+
*`zScore` {number} The z-score (normal approximation).
2103+
*`pValue` {number} Two-tailed p-value.
2104+
2105+
Performs a [Mann-Whitney U test][] comparing whether this histogram tends to
2106+
produce larger or smaller values than `other`. Unlike `welchTest()`, this is a
2107+
non-parametric test that makes no assumptions about the shape of the
2108+
distributions. Uses the normal approximation with tie correction for the
2109+
p-value.
2110+
19852111
### `histogram.max`
19862112

19872113
<!-- YAML
@@ -2060,6 +2186,40 @@ added:
20602186

20612187
Returns the value at the given percentile.
20622188

2189+
### `histogram.percentileCI(percentile[, options])`
2190+
2191+
<!-- YAML
2192+
added: REPLACEME
2193+
-->
2194+
2195+
*`percentile` {number} A percentile value in the range (0, 100].
2196+
*`options` {Object}
2197+
*`confidence` {number} The confidence level for the interval, between
2198+
 0 and 1 (exclusive). **Default:**`0.95`.
2199+
* Returns: {Object}
2200+
*`value` {number} The point estimate (same as `histogram.percentile()`).
2201+
*`lower` {number} The lower bound of the confidence interval.
2202+
*`upper` {number} The upper bound of the confidence interval.
2203+
2204+
Returns a confidence interval for the given percentile using the exact
2205+
binomial method. With fewer samples, the interval will be wider, reflecting
2206+
the greater uncertainty in the percentile estimate. Requires at least 2
2207+
recorded values; with fewer than 2, `lower` and `upper` will equal `value`.
2208+
2209+
```js
2210+
const { createHistogram } =require('node:perf_hooks');
2211+
2212+
consth=createHistogram();
2213+
for (let i =0; i <1000; i++) {
2214+
h.record(Math.floor(Math.random() *100));
2215+
}
2216+
2217+
constci=h.percentileCI(99);
2218+
console.log(ci.value); // The p99 point estimate
2219+
console.log(ci.lower); // The lower bound (95% confidence)
2220+
console.log(ci.upper); // The upper bound (95% confidence)
2221+
```
2222+
20632223
### `histogram.percentiles`
20642224

20652225
<!-- YAML
@@ -2127,6 +2287,31 @@ added: v11.10.0
21272287

21282288
The standard deviation of the recorded event loop delays.
21292289

2290+
### `histogram.welchTest(other[, options])`
2291+
2292+
<!-- YAML
2293+
added: REPLACEME
2294+
-->
2295+
2296+
*`other` {Histogram} The histogram to compare against.
2297+
*`options` {Object}
2298+
*`confidence` {number} Confidence level for the interval, between 0 and 1.
2299+
**Default:**`0.95`.
2300+
* Returns: {Object}
2301+
*`tStatistic` {number} The Welch t-statistic.
2302+
*`degreesOfFreedom` {number} Welch-Satterthwaite degrees of freedom.
2303+
*`pValue` {number} Two-tailed p-value.
2304+
*`confidenceInterval` {Object}
2305+
*`lower` {number} Lower bound of the confidence interval on the
2306+
 difference of means.
2307+
*`upper` {number} Upper bound.
2308+
2309+
Performs [Welch's t-test][] comparing the means of this histogram and `other`.
2310+
The p-value indicates the probability of observing a difference at least this
2311+
extreme under the null hypothesis that the two distributions have the same
2312+
mean. Both histograms must have at least 2 recorded values; otherwise the
2313+
result has `pValue` 1 and `tStatistic` 0.
2314+
21302315
## Class: `ELDHistogram extends Histogram`
21312316

21322317
A `Histogram` that records event loop delay, returned by
@@ -2288,6 +2473,32 @@ const violating = latency.ccdf(500_000_000);
22882473
console.log(`${(violating *100).toFixed(1)}% of requests violating SLO`);
22892474
```
22902475

2476+
### SLO burn rate monitoring
2477+
2478+
```js
2479+
const { createHistogram } =require('node:perf_hooks');
2480+
2481+
// Track latency with EWMA (half-life 100 samples) and a 200ms SLO threshold
2482+
constlatency=createHistogram({
2483+
 halfLife:100,
2484+
 threshold:200_000_000, // 200ms in nanoseconds
2485+
});
2486+
2487+
// Record request latencies...
2488+
2489+
// Smoothed error rate: probability of exceeding the threshold
2490+
console.log(`Error rate: ${(latency.ewmaErrorRate*100).toFixed(2)}%`);
2491+
2492+
// Burn rate against a 99.9% SLO
2493+
// >1 means the error budget is depleting faster than allowed
2494+
constrate=latency.burnRate(0.999);
2495+
console.log(`Burn rate: ${rate.toFixed(2)}x`);
2496+
2497+
// EWMA mean and stddev track the smoothed latency
2498+
console.log(`EWMA latency: ${latency.ewmaMean.toFixed(0)}ns`);
2499+
console.log(`EWMA stddev: ${latency.ewmaStddev.toFixed(0)}ns`);
2500+
```
2501+
22912502
### Regression detection with KS test
22922503

22932504
```js
@@ -2339,6 +2550,46 @@ newSnapshot.subtract(snapshot);
23392550
console.log('Recent p99:', newSnapshot.percentile(99));
23402551
```
23412552

2553+
### Benchmark comparison with Welch's t-test
2554+
2555+
```js
2556+
const { createHistogram } =require('node:perf_hooks');
2557+
2558+
constbaseline=createHistogram();
2559+
constcandidate=createHistogram();
2560+
2561+
// Record operation rates from the old and new builds...
2562+
2563+
constresult=baseline.welchTest(candidate);
2564+
constimprovement= ((candidate.mean-baseline.mean) /baseline.mean*100);
2565+
2566+
console.log(`Improvement: ${improvement.toFixed(2)}%`);
2567+
console.log(`p-value: ${result.pValue.toFixed(6)}`);
2568+
console.log(`95% CI: [${result.confidenceInterval.lower.toFixed(2)}, `+
2569+
`${result.confidenceInterval.upper.toFixed(2)}]`);
2570+
2571+
if (result.pValue<0.05) {
2572+
constd=baseline.cohensD(candidate);
2573+
console.log(`Statistically significant (Cohen's d = ${d.toFixed(4)})`);
2574+
}
2575+
```
2576+
2577+
### Effect size with Cliff's delta
2578+
2579+
```js
2580+
const { createHistogram } =require('node:perf_hooks');
2581+
2582+
constbefore=createHistogram();
2583+
constafter=createHistogram();
2584+
2585+
// Record latencies before and after a change...
2586+
2587+
constdelta=before.cliffsD(after);
2588+
// A delta > 0: before tends to produce larger values (improvement)
2589+
// A delta < 0: after tends to produce larger values (regression)
2590+
console.log(`Cliff's delta: ${delta.toFixed(4)}`);
2591+
```
2592+
23422593
## Examples
23432594

23442595
### Measuring the duration of async operations
@@ -2593,13 +2844,17 @@ dns.promises.resolve('localhost');
25932844
```
25942845

25952846
[Async Hooks]: async_hooks.md
2847+
[Cliff's delta]: https://en.wikipedia.org/wiki/Effect_size#Cliff's_delta
2848+
[Cohen's d]: https://en.wikipedia.org/wiki/Effect_size#Cohen's_d
25962849
[Fetch Response Body Info]: https://fetch.spec.whatwg.org/#response-body-info
25972850
[Fetch Timing Info]: https://fetch.spec.whatwg.org/#fetch-timing-info
25982851
[High Resolution Time]: https://www.w3.org/TR/hr-time-2
2852+
[Mann-Whitney U test]: https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test
25992853
[Performance Timeline]: https://w3c.github.io/performance-timeline/
26002854
[Resource Timing]: https://www.w3.org/TR/resource-timing-2/
26012855
[User Timing]: https://www.w3.org/TR/user-timing/
26022856
[Web Performance APIs]: https://w3c.github.io/perf-timing-primer/
2857+
[Welch's t-test]: https://en.wikipedia.org/wiki/Welch%27s_t-test
26032858
[Worker threads]: worker_threads.md#worker-threads
26042859
[`'exit'`]: process.md#event-exit
26052860
[`child_process.spawnSync()`]: child_process.md#child_processspawnsynccommand-args-options
0 commit comments
Comments
 (0)