fix: exclude PR-modified tests from the flaky count

[zeexzeex]

What

A failure in a test that the pull request itself modified is counted as an independent flaky occurrence, which overstates the number of affected pull requests.

Taking the example from the issue, ffi/test-ffi-fast-buffer is listed with two failed PRs in the 2026-08-15 report:

PR	Files changed
#65233	src/ffi/fast.cc, test/ffi/test-ffi-fast-buffer.js
#65113	lib/fs.js, unrelated fs tests

Only #65113 is a genuine occurrence.

How

Compare the failing test against the files the pull request touched, and drop the occurrence when they match. formatAsMarkdown already skips entries with fewer than two PRs, so such an entry leaves the report on its own.

When the file list cannot be fetched the occurrence is kept, following lib/benchmark.js: incomplete information should never remove a genuine failure.

This makes aggregate() async. Both call sites in bin/ncu-ci.js were already async methods. The lazy fallbacks inside formatAsMarkdown() and display() now throw instead, since a pending promise there would fail silently.

Tests

Four tests cover the exclusion, the untouched case, the fetch failure, and non-test failures. Removing the exclusion makes only the first fail, while the other three keep passing.

Fixes: #1164

[zeexzeex]

What

A failure in a test that the pull request itself modified is counted as an independent flaky occurrence, which overstates the number of affected pull requests.

Taking the example from the issue, ffi/test-ffi-fast-buffer is listed with two failed PRs in the 2026-08-15 report:

PR	Files changed
#65233	src/ffi/fast.cc, test/ffi/test-ffi-fast-buffer.js
#65113	lib/fs.js, unrelated fs tests

Only #65113 is a genuine occurrence.

How

Compare the failing test against the files the pull request touched, and drop the occurrence when they match. formatAsMarkdown already skips entries with fewer than two PRs, so such an entry leaves the report on its own.

When the file list cannot be fetched the occurrence is kept, following lib/benchmark.js: incomplete information should never remove a genuine failure.

This makes aggregate() async. Both call sites in bin/ncu-ci.js were already async methods. The lazy fallbacks inside formatAsMarkdown() and display() now throw instead, since a pending promise there would fail silently.

Tests

Four tests cover the exclusion, the untouched case, the fetch failure, and non-test failures. Removing the exclusion makes only the first fail, while the other three keep passing.

Fixes: #1164

[zeexzeex]

What

A failure in a test that the pull request itself modified is counted as an independent flaky occurrence, which overstates the number of affected pull requests.

Taking the example from the issue, ffi/test-ffi-fast-buffer is listed with two failed PRs in the 2026-08-15 report:

PR	Files changed
#65233	src/ffi/fast.cc, test/ffi/test-ffi-fast-buffer.js
#65113	lib/fs.js, unrelated fs tests

Only #65113 is a genuine occurrence.

How

Compare the failing test against the files the pull request touched, and drop the occurrence when they match. formatAsMarkdown already skips entries with fewer than two PRs, so such an entry leaves the report on its own.

When the file list cannot be fetched the occurrence is kept, following lib/benchmark.js: incomplete information should never remove a genuine failure.

This makes aggregate() async. Both call sites in bin/ncu-ci.js were already async methods. The lazy fallbacks inside formatAsMarkdown() and display() now throw instead, since a pending promise there would fail silently.

Tests

Four tests cover the exclusion, the untouched case, the fetch failure, and non-test failures. Removing the exclusion makes only the first fail, while the other three keep passing.

Fixes: #1164

[zeexzeex]

What

A failure in a test that the pull request itself modified is counted as an independent flaky occurrence, which overstates the number of affected pull requests.

Taking the example from the issue, ffi/test-ffi-fast-buffer is listed with two failed PRs in the 2026-08-15 report:

PR	Files changed
#65233	src/ffi/fast.cc, test/ffi/test-ffi-fast-buffer.js
#65113	lib/fs.js, unrelated fs tests

Only #65113 is a genuine occurrence.

How

Compare the failing test against the files the pull request touched, and drop the occurrence when they match. formatAsMarkdown already skips entries with fewer than two PRs, so such an entry leaves the report on its own.

When the file list cannot be fetched the occurrence is kept, following lib/benchmark.js: incomplete information should never remove a genuine failure.

This makes aggregate() async. Both call sites in bin/ncu-ci.js were already async methods. The lazy fallbacks inside formatAsMarkdown() and display() now throw instead, since a pending promise there would fail silently.

Tests

Four tests cover the exclusion, the untouched case, the fetch failure, and non-test failures. Removing the exclusion makes only the first fail, while the other three keep passing.

Fixes: #1164

[zeexzeex]

What

A failure in a test that the pull request itself modified is counted as an independent flaky occurrence, which overstates the number of affected pull requests.

Taking the example from the issue, ffi/test-ffi-fast-buffer is listed with two failed PRs in the 2026-08-15 report:

PR	Files changed
#65233	src/ffi/fast.cc, test/ffi/test-ffi-fast-buffer.js
#65113	lib/fs.js, unrelated fs tests

Only #65113 is a genuine occurrence.

How

Compare the failing test against the files the pull request touched, and drop the occurrence when they match. formatAsMarkdown already skips entries with fewer than two PRs, so such an entry leaves the report on its own.

When the file list cannot be fetched the occurrence is kept, following lib/benchmark.js: incomplete information should never remove a genuine failure.

This makes aggregate() async. Both call sites in bin/ncu-ci.js were already async methods. The lazy fallbacks inside formatAsMarkdown() and display() now throw instead, since a pending promise there would fail silently.

Tests

Four tests cover the exclusion, the untouched case, the fetch failure, and non-test failures. Removing the exclusion makes only the first fail, while the other three keep passing.

Fixes: #1164

[zeexzeex]

What

A failure in a test that the pull request itself modified is counted as an independent flaky occurrence, which overstates the number of affected pull requests.

Taking the example from the issue, ffi/test-ffi-fast-buffer is listed with two failed PRs in the 2026-08-15 report:

PR	Files changed
#65233	src/ffi/fast.cc, test/ffi/test-ffi-fast-buffer.js
#65113	lib/fs.js, unrelated fs tests

Only #65113 is a genuine occurrence.

How

Compare the failing test against the files the pull request touched, and drop the occurrence when they match. formatAsMarkdown already skips entries with fewer than two PRs, so such an entry leaves the report on its own.

When the file list cannot be fetched the occurrence is kept, following lib/benchmark.js: incomplete information should never remove a genuine failure.

This makes aggregate() async. Both call sites in bin/ncu-ci.js were already async methods. The lazy fallbacks inside formatAsMarkdown() and display() now throw instead, since a pending promise there would fail silently.

Tests

Four tests cover the exclusion, the untouched case, the fetch failure, and non-test failures. Removing the exclusion makes only the first fail, while the other three keep passing.

Fixes: #1164

[zeexzeex]

What

A failure in a test that the pull request itself modified is counted as an independent flaky occurrence, which overstates the number of affected pull requests.

Taking the example from the issue, ffi/test-ffi-fast-buffer is listed with two failed PRs in the 2026-08-15 report:

PR	Files changed
#65233	src/ffi/fast.cc, test/ffi/test-ffi-fast-buffer.js
#65113	lib/fs.js, unrelated fs tests

Only #65113 is a genuine occurrence.

How

Compare the failing test against the files the pull request touched, and drop the occurrence when they match. formatAsMarkdown already skips entries with fewer than two PRs, so such an entry leaves the report on its own.

When the file list cannot be fetched the occurrence is kept, following lib/benchmark.js: incomplete information should never remove a genuine failure.

This makes aggregate() async. Both call sites in bin/ncu-ci.js were already async methods. The lazy fallbacks inside formatAsMarkdown() and display() now throw instead, since a pending promise there would fail silently.

Tests

Four tests cover the exclusion, the untouched case, the fetch failure, and non-test failures. Removing the exclusion makes only the first fail, while the other three keep passing.

Fixes: #1164

[zeexzeex]

What

A failure in a test that the pull request itself modified is counted as an independent flaky occurrence, which overstates the number of affected pull requests.

Taking the example from the issue, ffi/test-ffi-fast-buffer is listed with two failed PRs in the 2026-08-15 report:

PR	Files changed
#65233	src/ffi/fast.cc, test/ffi/test-ffi-fast-buffer.js
#65113	lib/fs.js, unrelated fs tests

Only #65113 is a genuine occurrence.

How

Compare the failing test against the files the pull request touched, and drop the occurrence when they match. formatAsMarkdown already skips entries with fewer than two PRs, so such an entry leaves the report on its own.

When the file list cannot be fetched the occurrence is kept, following lib/benchmark.js: incomplete information should never remove a genuine failure.

This makes aggregate() async. Both call sites in bin/ncu-ci.js were already async methods. The lazy fallbacks inside formatAsMarkdown() and display() now throw instead, since a pending promise there would fail silently.

Tests

Four tests cover the exclusion, the untouched case, the fetch failure, and non-test failures. Removing the exclusion makes only the first fail, while the other three keep passing.

Fixes: #1164