✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two) - #311

Closed
taras wants to merge 1 commit into
mainfrom
feat/issue-280-affected-tests
Closed

✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two)#311
taras wants to merge 1 commit into
mainfrom
feat/issue-280-affected-tests

Conversation

@taras

@tarastaras commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Why

Agents run the whole corpus under three runtimes after every change, and CI spends
306–316 s doing the same. #280 rules that agents run only the tests their branch and
worktree affect, decided by Deno's native selection, while CI keeps running everything.

This is PR A of two: the mechanism, with no policy change. AGENTS.md's default and
the exhaustive CI jobs are untouched, so the command ships alongside the existing battery
and can be exercised before anything depends on it. PR B records CI-measured weights,
installs the sharded matrix and its aggregate, updates the docs, and closes#280.

What changes

Before: the only local test commands run the complete corpus — deno task test,
pnpm test:node, bun run test:bun.

After: deno task test:affected runs the tests a change affects, under all three
runtimes, from one selected set.

$ deno task test:affected --base HEAD # one changed file in packages/acp/src
changed paths (1)
M packages/acp/src/session-key.ts
selected 9 of 174 test files
d n b packages/acp/tests/acceptance.test.ts
…
results
deno passed in 47s
node passed in 56.2s
bun passed in 40.9s
this command runs tests only — still applicable:
deno task lint / check / check:jsr / compiled smoke / xmd test packages/core/src --raw
CI runs the complete corpus under every runtime. This run is not equivalent to it.

66 s against roughly ten minutes for the three full suites.

How it works

git change set → classification → reachability proof → Deno's selection → 3 runtimes
  1. Change set (scripts/lib/git-changes.ts) — merge-base diff, staged, unstaged, and
    untracked, parsed NUL-safely and status-aware with explicit -M -C. A rename
    contributes both paths, the old one as a deletion; a copy contributes only its new
    path, because the source still exists.
  2. Classification (scripts/lib/change-classes.ts) — every path lands in exactly one
    class, and anything unmatched is unknown, which runs everything. Deletions,
    configuration, dependency state, the selector itself, the shared harness, runtime-read
    documents and fixtures, and bundle inputs each run the full corpus.
  3. Reachability proof (--related) — selection says nothing about a file in no test's
    graph, so any changed TypeScript with no static dependent runs the whole corpus rather
    than reporting a narrow green.
  4. Selection (--changed + --no-run, one candidate per probe) — Deno's answer,
    captured before anything runs.
  5. Three runtimes — the same set, minus each runtime's recorded exclusions, run
    concurrently under allSettled.

Review guide

Start with:scripts/lib/affected.ts — the module doc records why every other seam
was rejected.

Then review:

  1. scripts/lib/change-classes.ts — the trigger table and its unknown fallback
  2. scripts/lib/git-changes.ts — rename versus copy, and the statuses it refuses
  3. scripts/affected-tests.ts — failure semantics and the report
  4. scripts/lib/shard.ts, scripts/measure-weights.ts — weighted partitioning (used by PR B)

Look carefully at:

  • The two different failures in scripts/affected-tests.ts: an unreadable change set
    stops with remediation; a probe that fails afterwards widens to the full corpus.
    Narrowing is never the response to an error.

What must stay true

  • Type-checking stays on in the probe — under --no-check the sweep silently loses
    every test that reaches a change only through import type. Enforced by the shared
    FLAGS and checked by "selects a test that reaches the change only through
    import type"
    .
  • Escalation is one-way — checked by the deletion, rename, fixture, document, unknown,
    unreachable, and probe-failure cases.
  • One discovery — CI will hand explicit lists to all three runners, so
    scripts/lib/test-files.ts must find everything Deno's glob would. Checked by
    scripts/tests/test-file-discovery.test.ts.
  • Every file, exactly once, in the partition — checked by scripts/tests/shard.test.ts.

How to verify it

Every correctness case runs against a scratch git repository the test builds, so no
fixture depends on the state of this repository.

  • "selects the tests that import a changed module, and nothing else" fails if selection
    widens or narrows.
  • "selects a test that reaches the change only through import type" fails if the probe
    ever adopts --no-check.
  • "runs everything when a changed module is in no test's graph" fails if a
    dynamically-loaded or subprocess-loaded file is trusted to select honestly.
  • "runs everything for a deletion, which selects nothing on its own" fails if a deleted
    path is treated as an ordinary change — the false green Deno produces natively.
  • "reports both sides of a committed rename" and "takes only the new path of a copy"
    fail if rename and copy are conflated.
  • "fails rather than reporting no changes when the base cannot be resolved" fails if an
    unreadable git state degrades to "nothing changed".
  • "runs everything and says which probe failed" fails if an error narrows the run.
  • "still covers the corpus when nothing has been measured" fails if unweighted files
    pile into one shard — it caught exactly that defect during development.

Manual: deno task test:affected --base HEAD after touching one source file.

Scope

Included

  • deno task test:affected and its libraries, with the full regression suite
  • deno task weights:measure and the workflow_dispatch job that runs it on ubuntu-latest

Intentionally unchanged

  • AGENTS.md still requires the full battery; affected mode is not yet the default
  • ci.yml still runs the unsharded corpus; the sharded matrix and runtime-tests
    aggregate are PR B
  • test-weights.json is not committed here — weights are recorded from a CI run in PR B
  • Deno --parallel stays out of Shard exhaustive CI runtime suites under five minutes #280: it is 3.7× faster locally and currently exits 1

New abstractions

  • Probe (scripts/lib/affected.ts) exists so selection can be driven by the real
    deno test and by a stub that fails on demand.
  • captured() (scripts/lib/captured.ts) exists because exec(...).join() forwards child
    output as it arrives, and three concurrent suites would interleave into one terminal.

Risks and limitations

Scope confirmation

  • Every changed file supports the purpose described above.
  • Unrelated cleanup and formatting changes are excluded.
  • Generated or mechanical changes are clearly identified.
  • The description matches the final diff and test results.

Part of #280

Deno decides which tests a change affects; this adds the machinery around
that answer and nothing that depends on it yet.
`deno task test:affected` reads the change set with status intact, widens the
run for every input the module graph cannot see, asks Deno which tests the
surviving TypeScript affects, and runs that one set under all three runtimes
minus their recorded exclusions.
The seam is measured, not assumed. Deno 2.9.1 exposes selection but not the
selected set: reporters lose module identity under `--parallel`, JUnit names
the module that registered each test — every suite here is attributed to
`packages/test-support/bdd.ts` — and coverage needs the run to finish. Explicit
file arguments intersect with `--changed`, so `--no-run` decides one candidate
without executing anything, and sweeping the corpus that way reproduced Deno's
own discovery-driven selection exactly. Type-checking stays on: under
`--no-check` the same sweep lost every test that reaches a change through
`import type`.
Selection cannot speak for a file in no test's graph, so `--related` proves
reachability first. A path with no static dependent — a CLI entry point tests
spawn, a component loaded through `import(file://…)`, a file nothing imports
yet — runs the whole corpus rather than reporting a narrow green.
Sharding is weighted rather than counted: `deno test --shard` splits the sorted
list into consecutive groups, which lands this corpus's heavyweights together.
`weights:measure` records what each file costs on the runner CI uses.
Part of #280
@github-actions

Copy link
Copy Markdown

PR #311: ✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two)

17 files, +1915 / -0

Scope

🔴 PR has 1915 lines changed. Split into focused PRs.

🟡 1915 lines changed. PRs under 400 receive more thorough review.

🟡 PR mixes config and source changes.

Structural

🟡 18 console statements.

Slop

✅ Slop indicators look low.

Static Analysis

✅ Oxlint found no issues.

Correctness

No extraneous code patterns detected.

@taras

taras commented Aug 3, 2026

Copy link
Copy Markdown
OwnerAuthor

Closing without merging: the prerequisite is unmet. #279 Stage 2 has not merged and #279
is still open, and the #280 ruling puts PR A after both. The branch
feat/issue-280-affected-tests is preserved at 490de0b; this is a sequencing close, not
an abandonment.

Recorded for the rebase, once #279 closes — revalidate against its final interfaces first
(the inert flag set probe and run must adopt together, the task-entry lock protection,
and the scripts/lib/ helpers), then repair:

1. Code rules. Swept the additions; exactly four violations, no others:

  • scripts/affected-tests.ts:298 and scripts/measure-weights.ts:98 — top-level await main(...)
  • scripts/affected-tests.ts:46as const on RUNTIMES
  • scripts/tests/affected-selection.test.ts:23 — the fixture's explicit Promise<unknown> annotation

2. Complete the approved automated matrix. Currently missing: cross-workspace
transitivity; the --no-check negative case as its own assertion rather than only the
positive type-only case; a real copy record and a real staged deletion (git rm) driven
through git rather than the parser alone; renames into and out of a triggering directory;
both dynamic-import cases (analyzable template and computed target); every classification
exercised through scratch selection rather than the pure table; and ignored output versus
tracked generator inputs.

3. Command-level tests. None exist today — scripts/affected-tests.ts is covered only
end to end by hand. Needed: remediation on an unreadable base, full-corpus fallback after a
probe failure, allSettled behaviour when one runtime fails, complete first-failure output,
cancellation terminating every child process, and clean state after success, failure, and
cancellation.

4. AGENTS.md stays unchanged in PR A — it already does here, and it stays that way.
Agents do not learn to run affected tests until PR B lands it beside exhaustive sharded CI.

5. Re-verification. Complete pinned battery on the rebased head, tracked-tree
cleanliness proven, fully green CI and zero unresolved threads before re-review.

Then stop after PR A. PR B changes the agent default and closes #280.

@tarastaras closed this Aug 3, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Shard exhaustive CI runtime suites under five minutes

1 participant

@taras
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all \u003cpre\u003e\u003ccode\u003e blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks"); } } catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); } })(); (function(){ try { var __m = "github.com"; var __re = new RegExp('^' + "github\\.com" + '
Skip to content

✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two) - #311

Closed
taras wants to merge 1 commit into
mainfrom
feat/issue-280-affected-tests
Closed

✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two)#311
taras wants to merge 1 commit into
mainfrom
feat/issue-280-affected-tests

Conversation

@taras

@tarastaras commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Why

Agents run the whole corpus under three runtimes after every change, and CI spends
306–316 s doing the same. #280 rules that agents run only the tests their branch and
worktree affect, decided by Deno's native selection, while CI keeps running everything.

This is PR A of two: the mechanism, with no policy change. AGENTS.md's default and
the exhaustive CI jobs are untouched, so the command ships alongside the existing battery
and can be exercised before anything depends on it. PR B records CI-measured weights,
installs the sharded matrix and its aggregate, updates the docs, and closes#280.

What changes

Before: the only local test commands run the complete corpus — deno task test,
pnpm test:node, bun run test:bun.

After: deno task test:affected runs the tests a change affects, under all three
runtimes, from one selected set.

$ deno task test:affected --base HEAD # one changed file in packages/acp/src
changed paths (1)
M packages/acp/src/session-key.ts
selected 9 of 174 test files
d n b packages/acp/tests/acceptance.test.ts
…
results
deno passed in 47s
node passed in 56.2s
bun passed in 40.9s
this command runs tests only — still applicable:
deno task lint / check / check:jsr / compiled smoke / xmd test packages/core/src --raw
CI runs the complete corpus under every runtime. This run is not equivalent to it.

66 s against roughly ten minutes for the three full suites.

How it works

git change set → classification → reachability proof → Deno's selection → 3 runtimes
  1. Change set (scripts/lib/git-changes.ts) — merge-base diff, staged, unstaged, and
    untracked, parsed NUL-safely and status-aware with explicit -M -C. A rename
    contributes both paths, the old one as a deletion; a copy contributes only its new
    path, because the source still exists.
  2. Classification (scripts/lib/change-classes.ts) — every path lands in exactly one
    class, and anything unmatched is unknown, which runs everything. Deletions,
    configuration, dependency state, the selector itself, the shared harness, runtime-read
    documents and fixtures, and bundle inputs each run the full corpus.
  3. Reachability proof (--related) — selection says nothing about a file in no test's
    graph, so any changed TypeScript with no static dependent runs the whole corpus rather
    than reporting a narrow green.
  4. Selection (--changed + --no-run, one candidate per probe) — Deno's answer,
    captured before anything runs.
  5. Three runtimes — the same set, minus each runtime's recorded exclusions, run
    concurrently under allSettled.

Review guide

Start with:scripts/lib/affected.ts — the module doc records why every other seam
was rejected.

Then review:

  1. scripts/lib/change-classes.ts — the trigger table and its unknown fallback
  2. scripts/lib/git-changes.ts — rename versus copy, and the statuses it refuses
  3. scripts/affected-tests.ts — failure semantics and the report
  4. scripts/lib/shard.ts, scripts/measure-weights.ts — weighted partitioning (used by PR B)

Look carefully at:

  • The two different failures in scripts/affected-tests.ts: an unreadable change set
    stops with remediation; a probe that fails afterwards widens to the full corpus.
    Narrowing is never the response to an error.

What must stay true

  • Type-checking stays on in the probe — under --no-check the sweep silently loses
    every test that reaches a change only through import type. Enforced by the shared
    FLAGS and checked by "selects a test that reaches the change only through
    import type"
    .
  • Escalation is one-way — checked by the deletion, rename, fixture, document, unknown,
    unreachable, and probe-failure cases.
  • One discovery — CI will hand explicit lists to all three runners, so
    scripts/lib/test-files.ts must find everything Deno's glob would. Checked by
    scripts/tests/test-file-discovery.test.ts.
  • Every file, exactly once, in the partition — checked by scripts/tests/shard.test.ts.

How to verify it

Every correctness case runs against a scratch git repository the test builds, so no
fixture depends on the state of this repository.

  • "selects the tests that import a changed module, and nothing else" fails if selection
    widens or narrows.
  • "selects a test that reaches the change only through import type" fails if the probe
    ever adopts --no-check.
  • "runs everything when a changed module is in no test's graph" fails if a
    dynamically-loaded or subprocess-loaded file is trusted to select honestly.
  • "runs everything for a deletion, which selects nothing on its own" fails if a deleted
    path is treated as an ordinary change — the false green Deno produces natively.
  • "reports both sides of a committed rename" and "takes only the new path of a copy"
    fail if rename and copy are conflated.
  • "fails rather than reporting no changes when the base cannot be resolved" fails if an
    unreadable git state degrades to "nothing changed".
  • "runs everything and says which probe failed" fails if an error narrows the run.
  • "still covers the corpus when nothing has been measured" fails if unweighted files
    pile into one shard — it caught exactly that defect during development.

Manual: deno task test:affected --base HEAD after touching one source file.

Scope

Included

  • deno task test:affected and its libraries, with the full regression suite
  • deno task weights:measure and the workflow_dispatch job that runs it on ubuntu-latest

Intentionally unchanged

  • AGENTS.md still requires the full battery; affected mode is not yet the default
  • ci.yml still runs the unsharded corpus; the sharded matrix and runtime-tests
    aggregate are PR B
  • test-weights.json is not committed here — weights are recorded from a CI run in PR B
  • Deno --parallel stays out of Shard exhaustive CI runtime suites under five minutes #280: it is 3.7× faster locally and currently exits 1

New abstractions

  • Probe (scripts/lib/affected.ts) exists so selection can be driven by the real
    deno test and by a stub that fails on demand.
  • captured() (scripts/lib/captured.ts) exists because exec(...).join() forwards child
    output as it arrives, and three concurrent suites would interleave into one terminal.

Risks and limitations

Scope confirmation

  • Every changed file supports the purpose described above.
  • Unrelated cleanup and formatting changes are excluded.
  • Generated or mechanical changes are clearly identified.
  • The description matches the final diff and test results.

Part of #280

Deno decides which tests a change affects; this adds the machinery around
that answer and nothing that depends on it yet.
`deno task test:affected` reads the change set with status intact, widens the
run for every input the module graph cannot see, asks Deno which tests the
surviving TypeScript affects, and runs that one set under all three runtimes
minus their recorded exclusions.
The seam is measured, not assumed. Deno 2.9.1 exposes selection but not the
selected set: reporters lose module identity under `--parallel`, JUnit names
the module that registered each test — every suite here is attributed to
`packages/test-support/bdd.ts` — and coverage needs the run to finish. Explicit
file arguments intersect with `--changed`, so `--no-run` decides one candidate
without executing anything, and sweeping the corpus that way reproduced Deno's
own discovery-driven selection exactly. Type-checking stays on: under
`--no-check` the same sweep lost every test that reaches a change through
`import type`.
Selection cannot speak for a file in no test's graph, so `--related` proves
reachability first. A path with no static dependent — a CLI entry point tests
spawn, a component loaded through `import(file://…)`, a file nothing imports
yet — runs the whole corpus rather than reporting a narrow green.
Sharding is weighted rather than counted: `deno test --shard` splits the sorted
list into consecutive groups, which lands this corpus's heavyweights together.
`weights:measure` records what each file costs on the runner CI uses.
Part of #280
@github-actions

Copy link
Copy Markdown

PR #311: ✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two)

17 files, +1915 / -0

Scope

🔴 PR has 1915 lines changed. Split into focused PRs.

🟡 1915 lines changed. PRs under 400 receive more thorough review.

🟡 PR mixes config and source changes.

Structural

🟡 18 console statements.

Slop

✅ Slop indicators look low.

Static Analysis

✅ Oxlint found no issues.

Correctness

No extraneous code patterns detected.

@taras

taras commented Aug 3, 2026

Copy link
Copy Markdown
OwnerAuthor

Closing without merging: the prerequisite is unmet. #279 Stage 2 has not merged and #279
is still open, and the #280 ruling puts PR A after both. The branch
feat/issue-280-affected-tests is preserved at 490de0b; this is a sequencing close, not
an abandonment.

Recorded for the rebase, once #279 closes — revalidate against its final interfaces first
(the inert flag set probe and run must adopt together, the task-entry lock protection,
and the scripts/lib/ helpers), then repair:

1. Code rules. Swept the additions; exactly four violations, no others:

  • scripts/affected-tests.ts:298 and scripts/measure-weights.ts:98 — top-level await main(...)
  • scripts/affected-tests.ts:46as const on RUNTIMES
  • scripts/tests/affected-selection.test.ts:23 — the fixture's explicit Promise<unknown> annotation

2. Complete the approved automated matrix. Currently missing: cross-workspace
transitivity; the --no-check negative case as its own assertion rather than only the
positive type-only case; a real copy record and a real staged deletion (git rm) driven
through git rather than the parser alone; renames into and out of a triggering directory;
both dynamic-import cases (analyzable template and computed target); every classification
exercised through scratch selection rather than the pure table; and ignored output versus
tracked generator inputs.

3. Command-level tests. None exist today — scripts/affected-tests.ts is covered only
end to end by hand. Needed: remediation on an unreadable base, full-corpus fallback after a
probe failure, allSettled behaviour when one runtime fails, complete first-failure output,
cancellation terminating every child process, and clean state after success, failure, and
cancellation.

4. AGENTS.md stays unchanged in PR A — it already does here, and it stays that way.
Agents do not learn to run affected tests until PR B lands it beside exhaustive sharded CI.

5. Re-verification. Complete pinned battery on the rebased head, tracked-tree
cleanliness proven, fully green CI and zero unresolved threads before re-review.

Then stop after PR A. PR B changes the agent default and closes #280.

@tarastaras closed this Aug 3, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Shard exhaustive CI runtime suites under five minutes

1 participant

@taras
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two) - #311

Closed
taras wants to merge 1 commit into
mainfrom
feat/issue-280-affected-tests
Closed

✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two)#311
taras wants to merge 1 commit into
mainfrom
feat/issue-280-affected-tests

Conversation

@taras

@tarastaras commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Why

Agents run the whole corpus under three runtimes after every change, and CI spends
306–316 s doing the same. #280 rules that agents run only the tests their branch and
worktree affect, decided by Deno's native selection, while CI keeps running everything.

This is PR A of two: the mechanism, with no policy change. AGENTS.md's default and
the exhaustive CI jobs are untouched, so the command ships alongside the existing battery
and can be exercised before anything depends on it. PR B records CI-measured weights,
installs the sharded matrix and its aggregate, updates the docs, and closes#280.

What changes

Before: the only local test commands run the complete corpus — deno task test,
pnpm test:node, bun run test:bun.

After: deno task test:affected runs the tests a change affects, under all three
runtimes, from one selected set.

$ deno task test:affected --base HEAD # one changed file in packages/acp/src
changed paths (1)
M packages/acp/src/session-key.ts
selected 9 of 174 test files
d n b packages/acp/tests/acceptance.test.ts
…
results
deno passed in 47s
node passed in 56.2s
bun passed in 40.9s
this command runs tests only — still applicable:
deno task lint / check / check:jsr / compiled smoke / xmd test packages/core/src --raw
CI runs the complete corpus under every runtime. This run is not equivalent to it.

66 s against roughly ten minutes for the three full suites.

How it works

git change set → classification → reachability proof → Deno's selection → 3 runtimes
  1. Change set (scripts/lib/git-changes.ts) — merge-base diff, staged, unstaged, and
    untracked, parsed NUL-safely and status-aware with explicit -M -C. A rename
    contributes both paths, the old one as a deletion; a copy contributes only its new
    path, because the source still exists.
  2. Classification (scripts/lib/change-classes.ts) — every path lands in exactly one
    class, and anything unmatched is unknown, which runs everything. Deletions,
    configuration, dependency state, the selector itself, the shared harness, runtime-read
    documents and fixtures, and bundle inputs each run the full corpus.
  3. Reachability proof (--related) — selection says nothing about a file in no test's
    graph, so any changed TypeScript with no static dependent runs the whole corpus rather
    than reporting a narrow green.
  4. Selection (--changed + --no-run, one candidate per probe) — Deno's answer,
    captured before anything runs.
  5. Three runtimes — the same set, minus each runtime's recorded exclusions, run
    concurrently under allSettled.

Review guide

Start with:scripts/lib/affected.ts — the module doc records why every other seam
was rejected.

Then review:

  1. scripts/lib/change-classes.ts — the trigger table and its unknown fallback
  2. scripts/lib/git-changes.ts — rename versus copy, and the statuses it refuses
  3. scripts/affected-tests.ts — failure semantics and the report
  4. scripts/lib/shard.ts, scripts/measure-weights.ts — weighted partitioning (used by PR B)

Look carefully at:

  • The two different failures in scripts/affected-tests.ts: an unreadable change set
    stops with remediation; a probe that fails afterwards widens to the full corpus.
    Narrowing is never the response to an error.

What must stay true

  • Type-checking stays on in the probe — under --no-check the sweep silently loses
    every test that reaches a change only through import type. Enforced by the shared
    FLAGS and checked by "selects a test that reaches the change only through
    import type"
    .
  • Escalation is one-way — checked by the deletion, rename, fixture, document, unknown,
    unreachable, and probe-failure cases.
  • One discovery — CI will hand explicit lists to all three runners, so
    scripts/lib/test-files.ts must find everything Deno's glob would. Checked by
    scripts/tests/test-file-discovery.test.ts.
  • Every file, exactly once, in the partition — checked by scripts/tests/shard.test.ts.

How to verify it

Every correctness case runs against a scratch git repository the test builds, so no
fixture depends on the state of this repository.

  • "selects the tests that import a changed module, and nothing else" fails if selection
    widens or narrows.
  • "selects a test that reaches the change only through import type" fails if the probe
    ever adopts --no-check.
  • "runs everything when a changed module is in no test's graph" fails if a
    dynamically-loaded or subprocess-loaded file is trusted to select honestly.
  • "runs everything for a deletion, which selects nothing on its own" fails if a deleted
    path is treated as an ordinary change — the false green Deno produces natively.
  • "reports both sides of a committed rename" and "takes only the new path of a copy"
    fail if rename and copy are conflated.
  • "fails rather than reporting no changes when the base cannot be resolved" fails if an
    unreadable git state degrades to "nothing changed".
  • "runs everything and says which probe failed" fails if an error narrows the run.
  • "still covers the corpus when nothing has been measured" fails if unweighted files
    pile into one shard — it caught exactly that defect during development.

Manual: deno task test:affected --base HEAD after touching one source file.

Scope

Included

  • deno task test:affected and its libraries, with the full regression suite
  • deno task weights:measure and the workflow_dispatch job that runs it on ubuntu-latest

Intentionally unchanged

  • AGENTS.md still requires the full battery; affected mode is not yet the default
  • ci.yml still runs the unsharded corpus; the sharded matrix and runtime-tests
    aggregate are PR B
  • test-weights.json is not committed here — weights are recorded from a CI run in PR B
  • Deno --parallel stays out of Shard exhaustive CI runtime suites under five minutes #280: it is 3.7× faster locally and currently exits 1

New abstractions

  • Probe (scripts/lib/affected.ts) exists so selection can be driven by the real
    deno test and by a stub that fails on demand.
  • captured() (scripts/lib/captured.ts) exists because exec(...).join() forwards child
    output as it arrives, and three concurrent suites would interleave into one terminal.

Risks and limitations

Scope confirmation

  • Every changed file supports the purpose described above.
  • Unrelated cleanup and formatting changes are excluded.
  • Generated or mechanical changes are clearly identified.
  • The description matches the final diff and test results.

Part of #280

Deno decides which tests a change affects; this adds the machinery around
that answer and nothing that depends on it yet.
`deno task test:affected` reads the change set with status intact, widens the
run for every input the module graph cannot see, asks Deno which tests the
surviving TypeScript affects, and runs that one set under all three runtimes
minus their recorded exclusions.
The seam is measured, not assumed. Deno 2.9.1 exposes selection but not the
selected set: reporters lose module identity under `--parallel`, JUnit names
the module that registered each test — every suite here is attributed to
`packages/test-support/bdd.ts` — and coverage needs the run to finish. Explicit
file arguments intersect with `--changed`, so `--no-run` decides one candidate
without executing anything, and sweeping the corpus that way reproduced Deno's
own discovery-driven selection exactly. Type-checking stays on: under
`--no-check` the same sweep lost every test that reaches a change through
`import type`.
Selection cannot speak for a file in no test's graph, so `--related` proves
reachability first. A path with no static dependent — a CLI entry point tests
spawn, a component loaded through `import(file://…)`, a file nothing imports
yet — runs the whole corpus rather than reporting a narrow green.
Sharding is weighted rather than counted: `deno test --shard` splits the sorted
list into consecutive groups, which lands this corpus's heavyweights together.
`weights:measure` records what each file costs on the runner CI uses.
Part of #280
@github-actions

Copy link
Copy Markdown

PR #311: ✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two)

17 files, +1915 / -0

Scope

🔴 PR has 1915 lines changed. Split into focused PRs.

🟡 1915 lines changed. PRs under 400 receive more thorough review.

🟡 PR mixes config and source changes.

Structural

🟡 18 console statements.

Slop

✅ Slop indicators look low.

Static Analysis

✅ Oxlint found no issues.

Correctness

No extraneous code patterns detected.

@taras

taras commented Aug 3, 2026

Copy link
Copy Markdown
OwnerAuthor

Closing without merging: the prerequisite is unmet. #279 Stage 2 has not merged and #279
is still open, and the #280 ruling puts PR A after both. The branch
feat/issue-280-affected-tests is preserved at 490de0b; this is a sequencing close, not
an abandonment.

Recorded for the rebase, once #279 closes — revalidate against its final interfaces first
(the inert flag set probe and run must adopt together, the task-entry lock protection,
and the scripts/lib/ helpers), then repair:

1. Code rules. Swept the additions; exactly four violations, no others:

  • scripts/affected-tests.ts:298 and scripts/measure-weights.ts:98 — top-level await main(...)
  • scripts/affected-tests.ts:46as const on RUNTIMES
  • scripts/tests/affected-selection.test.ts:23 — the fixture's explicit Promise<unknown> annotation

2. Complete the approved automated matrix. Currently missing: cross-workspace
transitivity; the --no-check negative case as its own assertion rather than only the
positive type-only case; a real copy record and a real staged deletion (git rm) driven
through git rather than the parser alone; renames into and out of a triggering directory;
both dynamic-import cases (analyzable template and computed target); every classification
exercised through scratch selection rather than the pure table; and ignored output versus
tracked generator inputs.

3. Command-level tests. None exist today — scripts/affected-tests.ts is covered only
end to end by hand. Needed: remediation on an unreadable base, full-corpus fallback after a
probe failure, allSettled behaviour when one runtime fails, complete first-failure output,
cancellation terminating every child process, and clean state after success, failure, and
cancellation.

4. AGENTS.md stays unchanged in PR A — it already does here, and it stays that way.
Agents do not learn to run affected tests until PR B lands it beside exhaustive sharded CI.

5. Re-verification. Complete pinned battery on the rebased head, tracked-tree
cleanliness proven, fully green CI and zero unresolved threads before re-review.

Then stop after PR A. PR B changes the agent default and closes #280.

@tarastaras closed this Aug 3, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Shard exhaustive CI runtime suites under five minutes

1 participant

@taras
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length \u003e 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two) - #311

Closed
taras wants to merge 1 commit into
mainfrom
feat/issue-280-affected-tests
Closed

✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two)#311
taras wants to merge 1 commit into
mainfrom
feat/issue-280-affected-tests

Conversation

@taras

@tarastaras commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Why

Agents run the whole corpus under three runtimes after every change, and CI spends
306–316 s doing the same. #280 rules that agents run only the tests their branch and
worktree affect, decided by Deno's native selection, while CI keeps running everything.

This is PR A of two: the mechanism, with no policy change. AGENTS.md's default and
the exhaustive CI jobs are untouched, so the command ships alongside the existing battery
and can be exercised before anything depends on it. PR B records CI-measured weights,
installs the sharded matrix and its aggregate, updates the docs, and closes#280.

What changes

Before: the only local test commands run the complete corpus — deno task test,
pnpm test:node, bun run test:bun.

After: deno task test:affected runs the tests a change affects, under all three
runtimes, from one selected set.

$ deno task test:affected --base HEAD # one changed file in packages/acp/src
changed paths (1)
M packages/acp/src/session-key.ts
selected 9 of 174 test files
d n b packages/acp/tests/acceptance.test.ts
…
results
deno passed in 47s
node passed in 56.2s
bun passed in 40.9s
this command runs tests only — still applicable:
deno task lint / check / check:jsr / compiled smoke / xmd test packages/core/src --raw
CI runs the complete corpus under every runtime. This run is not equivalent to it.

66 s against roughly ten minutes for the three full suites.

How it works

git change set → classification → reachability proof → Deno's selection → 3 runtimes
  1. Change set (scripts/lib/git-changes.ts) — merge-base diff, staged, unstaged, and
    untracked, parsed NUL-safely and status-aware with explicit -M -C. A rename
    contributes both paths, the old one as a deletion; a copy contributes only its new
    path, because the source still exists.
  2. Classification (scripts/lib/change-classes.ts) — every path lands in exactly one
    class, and anything unmatched is unknown, which runs everything. Deletions,
    configuration, dependency state, the selector itself, the shared harness, runtime-read
    documents and fixtures, and bundle inputs each run the full corpus.
  3. Reachability proof (--related) — selection says nothing about a file in no test's
    graph, so any changed TypeScript with no static dependent runs the whole corpus rather
    than reporting a narrow green.
  4. Selection (--changed + --no-run, one candidate per probe) — Deno's answer,
    captured before anything runs.
  5. Three runtimes — the same set, minus each runtime's recorded exclusions, run
    concurrently under allSettled.

Review guide

Start with:scripts/lib/affected.ts — the module doc records why every other seam
was rejected.

Then review:

  1. scripts/lib/change-classes.ts — the trigger table and its unknown fallback
  2. scripts/lib/git-changes.ts — rename versus copy, and the statuses it refuses
  3. scripts/affected-tests.ts — failure semantics and the report
  4. scripts/lib/shard.ts, scripts/measure-weights.ts — weighted partitioning (used by PR B)

Look carefully at:

  • The two different failures in scripts/affected-tests.ts: an unreadable change set
    stops with remediation; a probe that fails afterwards widens to the full corpus.
    Narrowing is never the response to an error.

What must stay true

  • Type-checking stays on in the probe — under --no-check the sweep silently loses
    every test that reaches a change only through import type. Enforced by the shared
    FLAGS and checked by "selects a test that reaches the change only through
    import type"
    .
  • Escalation is one-way — checked by the deletion, rename, fixture, document, unknown,
    unreachable, and probe-failure cases.
  • One discovery — CI will hand explicit lists to all three runners, so
    scripts/lib/test-files.ts must find everything Deno's glob would. Checked by
    scripts/tests/test-file-discovery.test.ts.
  • Every file, exactly once, in the partition — checked by scripts/tests/shard.test.ts.

How to verify it

Every correctness case runs against a scratch git repository the test builds, so no
fixture depends on the state of this repository.

  • "selects the tests that import a changed module, and nothing else" fails if selection
    widens or narrows.
  • "selects a test that reaches the change only through import type" fails if the probe
    ever adopts --no-check.
  • "runs everything when a changed module is in no test's graph" fails if a
    dynamically-loaded or subprocess-loaded file is trusted to select honestly.
  • "runs everything for a deletion, which selects nothing on its own" fails if a deleted
    path is treated as an ordinary change — the false green Deno produces natively.
  • "reports both sides of a committed rename" and "takes only the new path of a copy"
    fail if rename and copy are conflated.
  • "fails rather than reporting no changes when the base cannot be resolved" fails if an
    unreadable git state degrades to "nothing changed".
  • "runs everything and says which probe failed" fails if an error narrows the run.
  • "still covers the corpus when nothing has been measured" fails if unweighted files
    pile into one shard — it caught exactly that defect during development.

Manual: deno task test:affected --base HEAD after touching one source file.

Scope

Included

  • deno task test:affected and its libraries, with the full regression suite
  • deno task weights:measure and the workflow_dispatch job that runs it on ubuntu-latest

Intentionally unchanged

  • AGENTS.md still requires the full battery; affected mode is not yet the default
  • ci.yml still runs the unsharded corpus; the sharded matrix and runtime-tests
    aggregate are PR B
  • test-weights.json is not committed here — weights are recorded from a CI run in PR B
  • Deno --parallel stays out of Shard exhaustive CI runtime suites under five minutes #280: it is 3.7× faster locally and currently exits 1

New abstractions

  • Probe (scripts/lib/affected.ts) exists so selection can be driven by the real
    deno test and by a stub that fails on demand.
  • captured() (scripts/lib/captured.ts) exists because exec(...).join() forwards child
    output as it arrives, and three concurrent suites would interleave into one terminal.

Risks and limitations

Scope confirmation

  • Every changed file supports the purpose described above.
  • Unrelated cleanup and formatting changes are excluded.
  • Generated or mechanical changes are clearly identified.
  • The description matches the final diff and test results.

Part of #280

Deno decides which tests a change affects; this adds the machinery around
that answer and nothing that depends on it yet.
`deno task test:affected` reads the change set with status intact, widens the
run for every input the module graph cannot see, asks Deno which tests the
surviving TypeScript affects, and runs that one set under all three runtimes
minus their recorded exclusions.
The seam is measured, not assumed. Deno 2.9.1 exposes selection but not the
selected set: reporters lose module identity under `--parallel`, JUnit names
the module that registered each test — every suite here is attributed to
`packages/test-support/bdd.ts` — and coverage needs the run to finish. Explicit
file arguments intersect with `--changed`, so `--no-run` decides one candidate
without executing anything, and sweeping the corpus that way reproduced Deno's
own discovery-driven selection exactly. Type-checking stays on: under
`--no-check` the same sweep lost every test that reaches a change through
`import type`.
Selection cannot speak for a file in no test's graph, so `--related` proves
reachability first. A path with no static dependent — a CLI entry point tests
spawn, a component loaded through `import(file://…)`, a file nothing imports
yet — runs the whole corpus rather than reporting a narrow green.
Sharding is weighted rather than counted: `deno test --shard` splits the sorted
list into consecutive groups, which lands this corpus's heavyweights together.
`weights:measure` records what each file costs on the runner CI uses.
Part of #280
@github-actions

Copy link
Copy Markdown

PR #311: ✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two)

17 files, +1915 / -0

Scope

🔴 PR has 1915 lines changed. Split into focused PRs.

🟡 1915 lines changed. PRs under 400 receive more thorough review.

🟡 PR mixes config and source changes.

Structural

🟡 18 console statements.

Slop

✅ Slop indicators look low.

Static Analysis

✅ Oxlint found no issues.

Correctness

No extraneous code patterns detected.

@taras

taras commented Aug 3, 2026

Copy link
Copy Markdown
OwnerAuthor

Closing without merging: the prerequisite is unmet. #279 Stage 2 has not merged and #279
is still open, and the #280 ruling puts PR A after both. The branch
feat/issue-280-affected-tests is preserved at 490de0b; this is a sequencing close, not
an abandonment.

Recorded for the rebase, once #279 closes — revalidate against its final interfaces first
(the inert flag set probe and run must adopt together, the task-entry lock protection,
and the scripts/lib/ helpers), then repair:

1. Code rules. Swept the additions; exactly four violations, no others:

  • scripts/affected-tests.ts:298 and scripts/measure-weights.ts:98 — top-level await main(...)
  • scripts/affected-tests.ts:46as const on RUNTIMES
  • scripts/tests/affected-selection.test.ts:23 — the fixture's explicit Promise<unknown> annotation

2. Complete the approved automated matrix. Currently missing: cross-workspace
transitivity; the --no-check negative case as its own assertion rather than only the
positive type-only case; a real copy record and a real staged deletion (git rm) driven
through git rather than the parser alone; renames into and out of a triggering directory;
both dynamic-import cases (analyzable template and computed target); every classification
exercised through scratch selection rather than the pure table; and ignored output versus
tracked generator inputs.

3. Command-level tests. None exist today — scripts/affected-tests.ts is covered only
end to end by hand. Needed: remediation on an unreadable base, full-corpus fallback after a
probe failure, allSettled behaviour when one runtime fails, complete first-failure output,
cancellation terminating every child process, and clean state after success, failure, and
cancellation.

4. AGENTS.md stays unchanged in PR A — it already does here, and it stays that way.
Agents do not learn to run affected tests until PR B lands it beside exhaustive sharded CI.

5. Re-verification. Complete pinned battery on the rebased head, tracked-tree
cleanliness proven, fully green CI and zero unresolved threads before re-review.

Then stop after PR A. PR B changes the agent default and closes #280.

@tarastaras closed this Aug 3, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Shard exhaustive CI runtime suites under five minutes

1 participant

@taras
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two) - #311

Closed
taras wants to merge 1 commit into
mainfrom
feat/issue-280-affected-tests
Closed

✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two)#311
taras wants to merge 1 commit into
mainfrom
feat/issue-280-affected-tests

Conversation

@taras

@tarastaras commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Why

Agents run the whole corpus under three runtimes after every change, and CI spends
306–316 s doing the same. #280 rules that agents run only the tests their branch and
worktree affect, decided by Deno's native selection, while CI keeps running everything.

This is PR A of two: the mechanism, with no policy change. AGENTS.md's default and
the exhaustive CI jobs are untouched, so the command ships alongside the existing battery
and can be exercised before anything depends on it. PR B records CI-measured weights,
installs the sharded matrix and its aggregate, updates the docs, and closes#280.

What changes

Before: the only local test commands run the complete corpus — deno task test,
pnpm test:node, bun run test:bun.

After: deno task test:affected runs the tests a change affects, under all three
runtimes, from one selected set.

$ deno task test:affected --base HEAD # one changed file in packages/acp/src
changed paths (1)
M packages/acp/src/session-key.ts
selected 9 of 174 test files
d n b packages/acp/tests/acceptance.test.ts
…
results
deno passed in 47s
node passed in 56.2s
bun passed in 40.9s
this command runs tests only — still applicable:
deno task lint / check / check:jsr / compiled smoke / xmd test packages/core/src --raw
CI runs the complete corpus under every runtime. This run is not equivalent to it.

66 s against roughly ten minutes for the three full suites.

How it works

git change set → classification → reachability proof → Deno's selection → 3 runtimes
  1. Change set (scripts/lib/git-changes.ts) — merge-base diff, staged, unstaged, and
    untracked, parsed NUL-safely and status-aware with explicit -M -C. A rename
    contributes both paths, the old one as a deletion; a copy contributes only its new
    path, because the source still exists.
  2. Classification (scripts/lib/change-classes.ts) — every path lands in exactly one
    class, and anything unmatched is unknown, which runs everything. Deletions,
    configuration, dependency state, the selector itself, the shared harness, runtime-read
    documents and fixtures, and bundle inputs each run the full corpus.
  3. Reachability proof (--related) — selection says nothing about a file in no test's
    graph, so any changed TypeScript with no static dependent runs the whole corpus rather
    than reporting a narrow green.
  4. Selection (--changed + --no-run, one candidate per probe) — Deno's answer,
    captured before anything runs.
  5. Three runtimes — the same set, minus each runtime's recorded exclusions, run
    concurrently under allSettled.

Review guide

Start with:scripts/lib/affected.ts — the module doc records why every other seam
was rejected.

Then review:

  1. scripts/lib/change-classes.ts — the trigger table and its unknown fallback
  2. scripts/lib/git-changes.ts — rename versus copy, and the statuses it refuses
  3. scripts/affected-tests.ts — failure semantics and the report
  4. scripts/lib/shard.ts, scripts/measure-weights.ts — weighted partitioning (used by PR B)

Look carefully at:

  • The two different failures in scripts/affected-tests.ts: an unreadable change set
    stops with remediation; a probe that fails afterwards widens to the full corpus.
    Narrowing is never the response to an error.

What must stay true

  • Type-checking stays on in the probe — under --no-check the sweep silently loses
    every test that reaches a change only through import type. Enforced by the shared
    FLAGS and checked by "selects a test that reaches the change only through
    import type"
    .
  • Escalation is one-way — checked by the deletion, rename, fixture, document, unknown,
    unreachable, and probe-failure cases.
  • One discovery — CI will hand explicit lists to all three runners, so
    scripts/lib/test-files.ts must find everything Deno's glob would. Checked by
    scripts/tests/test-file-discovery.test.ts.
  • Every file, exactly once, in the partition — checked by scripts/tests/shard.test.ts.

How to verify it

Every correctness case runs against a scratch git repository the test builds, so no
fixture depends on the state of this repository.

  • "selects the tests that import a changed module, and nothing else" fails if selection
    widens or narrows.
  • "selects a test that reaches the change only through import type" fails if the probe
    ever adopts --no-check.
  • "runs everything when a changed module is in no test's graph" fails if a
    dynamically-loaded or subprocess-loaded file is trusted to select honestly.
  • "runs everything for a deletion, which selects nothing on its own" fails if a deleted
    path is treated as an ordinary change — the false green Deno produces natively.
  • "reports both sides of a committed rename" and "takes only the new path of a copy"
    fail if rename and copy are conflated.
  • "fails rather than reporting no changes when the base cannot be resolved" fails if an
    unreadable git state degrades to "nothing changed".
  • "runs everything and says which probe failed" fails if an error narrows the run.
  • "still covers the corpus when nothing has been measured" fails if unweighted files
    pile into one shard — it caught exactly that defect during development.

Manual: deno task test:affected --base HEAD after touching one source file.

Scope

Included

  • deno task test:affected and its libraries, with the full regression suite
  • deno task weights:measure and the workflow_dispatch job that runs it on ubuntu-latest

Intentionally unchanged

  • AGENTS.md still requires the full battery; affected mode is not yet the default
  • ci.yml still runs the unsharded corpus; the sharded matrix and runtime-tests
    aggregate are PR B
  • test-weights.json is not committed here — weights are recorded from a CI run in PR B
  • Deno --parallel stays out of Shard exhaustive CI runtime suites under five minutes #280: it is 3.7× faster locally and currently exits 1

New abstractions

  • Probe (scripts/lib/affected.ts) exists so selection can be driven by the real
    deno test and by a stub that fails on demand.
  • captured() (scripts/lib/captured.ts) exists because exec(...).join() forwards child
    output as it arrives, and three concurrent suites would interleave into one terminal.

Risks and limitations

Scope confirmation

  • Every changed file supports the purpose described above.
  • Unrelated cleanup and formatting changes are excluded.
  • Generated or mechanical changes are clearly identified.
  • The description matches the final diff and test results.

Part of #280

Deno decides which tests a change affects; this adds the machinery around
that answer and nothing that depends on it yet.
`deno task test:affected` reads the change set with status intact, widens the
run for every input the module graph cannot see, asks Deno which tests the
surviving TypeScript affects, and runs that one set under all three runtimes
minus their recorded exclusions.
The seam is measured, not assumed. Deno 2.9.1 exposes selection but not the
selected set: reporters lose module identity under `--parallel`, JUnit names
the module that registered each test — every suite here is attributed to
`packages/test-support/bdd.ts` — and coverage needs the run to finish. Explicit
file arguments intersect with `--changed`, so `--no-run` decides one candidate
without executing anything, and sweeping the corpus that way reproduced Deno's
own discovery-driven selection exactly. Type-checking stays on: under
`--no-check` the same sweep lost every test that reaches a change through
`import type`.
Selection cannot speak for a file in no test's graph, so `--related` proves
reachability first. A path with no static dependent — a CLI entry point tests
spawn, a component loaded through `import(file://…)`, a file nothing imports
yet — runs the whole corpus rather than reporting a narrow green.
Sharding is weighted rather than counted: `deno test --shard` splits the sorted
list into consecutive groups, which lands this corpus's heavyweights together.
`weights:measure` records what each file costs on the runner CI uses.
Part of #280
@github-actions

Copy link
Copy Markdown

PR #311: ✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two)

17 files, +1915 / -0

Scope

🔴 PR has 1915 lines changed. Split into focused PRs.

🟡 1915 lines changed. PRs under 400 receive more thorough review.

🟡 PR mixes config and source changes.

Structural

🟡 18 console statements.

Slop

✅ Slop indicators look low.

Static Analysis

✅ Oxlint found no issues.

Correctness

No extraneous code patterns detected.

@taras

taras commented Aug 3, 2026

Copy link
Copy Markdown
OwnerAuthor

Closing without merging: the prerequisite is unmet. #279 Stage 2 has not merged and #279
is still open, and the #280 ruling puts PR A after both. The branch
feat/issue-280-affected-tests is preserved at 490de0b; this is a sequencing close, not
an abandonment.

Recorded for the rebase, once #279 closes — revalidate against its final interfaces first
(the inert flag set probe and run must adopt together, the task-entry lock protection,
and the scripts/lib/ helpers), then repair:

1. Code rules. Swept the additions; exactly four violations, no others:

  • scripts/affected-tests.ts:298 and scripts/measure-weights.ts:98 — top-level await main(...)
  • scripts/affected-tests.ts:46as const on RUNTIMES
  • scripts/tests/affected-selection.test.ts:23 — the fixture's explicit Promise<unknown> annotation

2. Complete the approved automated matrix. Currently missing: cross-workspace
transitivity; the --no-check negative case as its own assertion rather than only the
positive type-only case; a real copy record and a real staged deletion (git rm) driven
through git rather than the parser alone; renames into and out of a triggering directory;
both dynamic-import cases (analyzable template and computed target); every classification
exercised through scratch selection rather than the pure table; and ignored output versus
tracked generator inputs.

3. Command-level tests. None exist today — scripts/affected-tests.ts is covered only
end to end by hand. Needed: remediation on an unreadable base, full-corpus fallback after a
probe failure, allSettled behaviour when one runtime fails, complete first-failure output,
cancellation terminating every child process, and clean state after success, failure, and
cancellation.

4. AGENTS.md stays unchanged in PR A — it already does here, and it stays that way.
Agents do not learn to run affected tests until PR B lands it beside exhaustive sharded CI.

5. Re-verification. Complete pinned battery on the rebased head, tracked-tree
cleanliness proven, fully green CI and zero unresolved threads before re-review.

Then stop after PR A. PR B changes the agent default and closes #280.

@tarastaras closed this Aug 3, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Shard exhaustive CI runtime suites under five minutes

1 participant

@taras
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two) - #311

Closed
taras wants to merge 1 commit into
mainfrom
feat/issue-280-affected-tests
Closed

✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two)#311
taras wants to merge 1 commit into
mainfrom
feat/issue-280-affected-tests

Conversation

@taras

@tarastaras commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Why

Agents run the whole corpus under three runtimes after every change, and CI spends
306–316 s doing the same. #280 rules that agents run only the tests their branch and
worktree affect, decided by Deno's native selection, while CI keeps running everything.

This is PR A of two: the mechanism, with no policy change. AGENTS.md's default and
the exhaustive CI jobs are untouched, so the command ships alongside the existing battery
and can be exercised before anything depends on it. PR B records CI-measured weights,
installs the sharded matrix and its aggregate, updates the docs, and closes#280.

What changes

Before: the only local test commands run the complete corpus — deno task test,
pnpm test:node, bun run test:bun.

After: deno task test:affected runs the tests a change affects, under all three
runtimes, from one selected set.

$ deno task test:affected --base HEAD # one changed file in packages/acp/src
changed paths (1)
M packages/acp/src/session-key.ts
selected 9 of 174 test files
d n b packages/acp/tests/acceptance.test.ts
…
results
deno passed in 47s
node passed in 56.2s
bun passed in 40.9s
this command runs tests only — still applicable:
deno task lint / check / check:jsr / compiled smoke / xmd test packages/core/src --raw
CI runs the complete corpus under every runtime. This run is not equivalent to it.

66 s against roughly ten minutes for the three full suites.

How it works

git change set → classification → reachability proof → Deno's selection → 3 runtimes
  1. Change set (scripts/lib/git-changes.ts) — merge-base diff, staged, unstaged, and
    untracked, parsed NUL-safely and status-aware with explicit -M -C. A rename
    contributes both paths, the old one as a deletion; a copy contributes only its new
    path, because the source still exists.
  2. Classification (scripts/lib/change-classes.ts) — every path lands in exactly one
    class, and anything unmatched is unknown, which runs everything. Deletions,
    configuration, dependency state, the selector itself, the shared harness, runtime-read
    documents and fixtures, and bundle inputs each run the full corpus.
  3. Reachability proof (--related) — selection says nothing about a file in no test's
    graph, so any changed TypeScript with no static dependent runs the whole corpus rather
    than reporting a narrow green.
  4. Selection (--changed + --no-run, one candidate per probe) — Deno's answer,
    captured before anything runs.
  5. Three runtimes — the same set, minus each runtime's recorded exclusions, run
    concurrently under allSettled.

Review guide

Start with:scripts/lib/affected.ts — the module doc records why every other seam
was rejected.

Then review:

  1. scripts/lib/change-classes.ts — the trigger table and its unknown fallback
  2. scripts/lib/git-changes.ts — rename versus copy, and the statuses it refuses
  3. scripts/affected-tests.ts — failure semantics and the report
  4. scripts/lib/shard.ts, scripts/measure-weights.ts — weighted partitioning (used by PR B)

Look carefully at:

  • The two different failures in scripts/affected-tests.ts: an unreadable change set
    stops with remediation; a probe that fails afterwards widens to the full corpus.
    Narrowing is never the response to an error.

What must stay true

  • Type-checking stays on in the probe — under --no-check the sweep silently loses
    every test that reaches a change only through import type. Enforced by the shared
    FLAGS and checked by "selects a test that reaches the change only through
    import type"
    .
  • Escalation is one-way — checked by the deletion, rename, fixture, document, unknown,
    unreachable, and probe-failure cases.
  • One discovery — CI will hand explicit lists to all three runners, so
    scripts/lib/test-files.ts must find everything Deno's glob would. Checked by
    scripts/tests/test-file-discovery.test.ts.
  • Every file, exactly once, in the partition — checked by scripts/tests/shard.test.ts.

How to verify it

Every correctness case runs against a scratch git repository the test builds, so no
fixture depends on the state of this repository.

  • "selects the tests that import a changed module, and nothing else" fails if selection
    widens or narrows.
  • "selects a test that reaches the change only through import type" fails if the probe
    ever adopts --no-check.
  • "runs everything when a changed module is in no test's graph" fails if a
    dynamically-loaded or subprocess-loaded file is trusted to select honestly.
  • "runs everything for a deletion, which selects nothing on its own" fails if a deleted
    path is treated as an ordinary change — the false green Deno produces natively.
  • "reports both sides of a committed rename" and "takes only the new path of a copy"
    fail if rename and copy are conflated.
  • "fails rather than reporting no changes when the base cannot be resolved" fails if an
    unreadable git state degrades to "nothing changed".
  • "runs everything and says which probe failed" fails if an error narrows the run.
  • "still covers the corpus when nothing has been measured" fails if unweighted files
    pile into one shard — it caught exactly that defect during development.

Manual: deno task test:affected --base HEAD after touching one source file.

Scope

Included

  • deno task test:affected and its libraries, with the full regression suite
  • deno task weights:measure and the workflow_dispatch job that runs it on ubuntu-latest

Intentionally unchanged

  • AGENTS.md still requires the full battery; affected mode is not yet the default
  • ci.yml still runs the unsharded corpus; the sharded matrix and runtime-tests
    aggregate are PR B
  • test-weights.json is not committed here — weights are recorded from a CI run in PR B
  • Deno --parallel stays out of Shard exhaustive CI runtime suites under five minutes #280: it is 3.7× faster locally and currently exits 1

New abstractions

  • Probe (scripts/lib/affected.ts) exists so selection can be driven by the real
    deno test and by a stub that fails on demand.
  • captured() (scripts/lib/captured.ts) exists because exec(...).join() forwards child
    output as it arrives, and three concurrent suites would interleave into one terminal.

Risks and limitations

Scope confirmation

  • Every changed file supports the purpose described above.
  • Unrelated cleanup and formatting changes are excluded.
  • Generated or mechanical changes are clearly identified.
  • The description matches the final diff and test results.

Part of #280

Deno decides which tests a change affects; this adds the machinery around
that answer and nothing that depends on it yet.
`deno task test:affected` reads the change set with status intact, widens the
run for every input the module graph cannot see, asks Deno which tests the
surviving TypeScript affects, and runs that one set under all three runtimes
minus their recorded exclusions.
The seam is measured, not assumed. Deno 2.9.1 exposes selection but not the
selected set: reporters lose module identity under `--parallel`, JUnit names
the module that registered each test — every suite here is attributed to
`packages/test-support/bdd.ts` — and coverage needs the run to finish. Explicit
file arguments intersect with `--changed`, so `--no-run` decides one candidate
without executing anything, and sweeping the corpus that way reproduced Deno's
own discovery-driven selection exactly. Type-checking stays on: under
`--no-check` the same sweep lost every test that reaches a change through
`import type`.
Selection cannot speak for a file in no test's graph, so `--related` proves
reachability first. A path with no static dependent — a CLI entry point tests
spawn, a component loaded through `import(file://…)`, a file nothing imports
yet — runs the whole corpus rather than reporting a narrow green.
Sharding is weighted rather than counted: `deno test --shard` splits the sorted
list into consecutive groups, which lands this corpus's heavyweights together.
`weights:measure` records what each file costs on the runner CI uses.
Part of #280
@github-actions

Copy link
Copy Markdown

PR #311: ✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two)

17 files, +1915 / -0

Scope

🔴 PR has 1915 lines changed. Split into focused PRs.

🟡 1915 lines changed. PRs under 400 receive more thorough review.

🟡 PR mixes config and source changes.

Structural

🟡 18 console statements.

Slop

✅ Slop indicators look low.

Static Analysis

✅ Oxlint found no issues.

Correctness

No extraneous code patterns detected.

@taras

taras commented Aug 3, 2026

Copy link
Copy Markdown
OwnerAuthor

Closing without merging: the prerequisite is unmet. #279 Stage 2 has not merged and #279
is still open, and the #280 ruling puts PR A after both. The branch
feat/issue-280-affected-tests is preserved at 490de0b; this is a sequencing close, not
an abandonment.

Recorded for the rebase, once #279 closes — revalidate against its final interfaces first
(the inert flag set probe and run must adopt together, the task-entry lock protection,
and the scripts/lib/ helpers), then repair:

1. Code rules. Swept the additions; exactly four violations, no others:

  • scripts/affected-tests.ts:298 and scripts/measure-weights.ts:98 — top-level await main(...)
  • scripts/affected-tests.ts:46as const on RUNTIMES
  • scripts/tests/affected-selection.test.ts:23 — the fixture's explicit Promise<unknown> annotation

2. Complete the approved automated matrix. Currently missing: cross-workspace
transitivity; the --no-check negative case as its own assertion rather than only the
positive type-only case; a real copy record and a real staged deletion (git rm) driven
through git rather than the parser alone; renames into and out of a triggering directory;
both dynamic-import cases (analyzable template and computed target); every classification
exercised through scratch selection rather than the pure table; and ignored output versus
tracked generator inputs.

3. Command-level tests. None exist today — scripts/affected-tests.ts is covered only
end to end by hand. Needed: remediation on an unreadable base, full-corpus fallback after a
probe failure, allSettled behaviour when one runtime fails, complete first-failure output,
cancellation terminating every child process, and clean state after success, failure, and
cancellation.

4. AGENTS.md stays unchanged in PR A — it already does here, and it stays that way.
Agents do not learn to run affected tests until PR B lands it beside exhaustive sharded CI.

5. Re-verification. Complete pinned battery on the rebased head, tracked-tree
cleanliness proven, fully green CI and zero unresolved threads before re-review.

Then stop after PR A. PR B changes the agent default and closes #280.

@tarastaras closed this Aug 3, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Shard exhaustive CI runtime suites under five minutes

1 participant

@taras
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two) - #311

Closed
taras wants to merge 1 commit into
mainfrom
feat/issue-280-affected-tests
Closed

✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two)#311
taras wants to merge 1 commit into
mainfrom
feat/issue-280-affected-tests

Conversation

@taras

@tarastaras commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Why

Agents run the whole corpus under three runtimes after every change, and CI spends
306–316 s doing the same. #280 rules that agents run only the tests their branch and
worktree affect, decided by Deno's native selection, while CI keeps running everything.

This is PR A of two: the mechanism, with no policy change. AGENTS.md's default and
the exhaustive CI jobs are untouched, so the command ships alongside the existing battery
and can be exercised before anything depends on it. PR B records CI-measured weights,
installs the sharded matrix and its aggregate, updates the docs, and closes#280.

What changes

Before: the only local test commands run the complete corpus — deno task test,
pnpm test:node, bun run test:bun.

After: deno task test:affected runs the tests a change affects, under all three
runtimes, from one selected set.

$ deno task test:affected --base HEAD # one changed file in packages/acp/src
changed paths (1)
M packages/acp/src/session-key.ts
selected 9 of 174 test files
d n b packages/acp/tests/acceptance.test.ts
…
results
deno passed in 47s
node passed in 56.2s
bun passed in 40.9s
this command runs tests only — still applicable:
deno task lint / check / check:jsr / compiled smoke / xmd test packages/core/src --raw
CI runs the complete corpus under every runtime. This run is not equivalent to it.

66 s against roughly ten minutes for the three full suites.

How it works

git change set → classification → reachability proof → Deno's selection → 3 runtimes
  1. Change set (scripts/lib/git-changes.ts) — merge-base diff, staged, unstaged, and
    untracked, parsed NUL-safely and status-aware with explicit -M -C. A rename
    contributes both paths, the old one as a deletion; a copy contributes only its new
    path, because the source still exists.
  2. Classification (scripts/lib/change-classes.ts) — every path lands in exactly one
    class, and anything unmatched is unknown, which runs everything. Deletions,
    configuration, dependency state, the selector itself, the shared harness, runtime-read
    documents and fixtures, and bundle inputs each run the full corpus.
  3. Reachability proof (--related) — selection says nothing about a file in no test's
    graph, so any changed TypeScript with no static dependent runs the whole corpus rather
    than reporting a narrow green.
  4. Selection (--changed + --no-run, one candidate per probe) — Deno's answer,
    captured before anything runs.
  5. Three runtimes — the same set, minus each runtime's recorded exclusions, run
    concurrently under allSettled.

Review guide

Start with:scripts/lib/affected.ts — the module doc records why every other seam
was rejected.

Then review:

  1. scripts/lib/change-classes.ts — the trigger table and its unknown fallback
  2. scripts/lib/git-changes.ts — rename versus copy, and the statuses it refuses
  3. scripts/affected-tests.ts — failure semantics and the report
  4. scripts/lib/shard.ts, scripts/measure-weights.ts — weighted partitioning (used by PR B)

Look carefully at:

  • The two different failures in scripts/affected-tests.ts: an unreadable change set
    stops with remediation; a probe that fails afterwards widens to the full corpus.
    Narrowing is never the response to an error.

What must stay true

  • Type-checking stays on in the probe — under --no-check the sweep silently loses
    every test that reaches a change only through import type. Enforced by the shared
    FLAGS and checked by "selects a test that reaches the change only through
    import type"
    .
  • Escalation is one-way — checked by the deletion, rename, fixture, document, unknown,
    unreachable, and probe-failure cases.
  • One discovery — CI will hand explicit lists to all three runners, so
    scripts/lib/test-files.ts must find everything Deno's glob would. Checked by
    scripts/tests/test-file-discovery.test.ts.
  • Every file, exactly once, in the partition — checked by scripts/tests/shard.test.ts.

How to verify it

Every correctness case runs against a scratch git repository the test builds, so no
fixture depends on the state of this repository.

  • "selects the tests that import a changed module, and nothing else" fails if selection
    widens or narrows.
  • "selects a test that reaches the change only through import type" fails if the probe
    ever adopts --no-check.
  • "runs everything when a changed module is in no test's graph" fails if a
    dynamically-loaded or subprocess-loaded file is trusted to select honestly.
  • "runs everything for a deletion, which selects nothing on its own" fails if a deleted
    path is treated as an ordinary change — the false green Deno produces natively.
  • "reports both sides of a committed rename" and "takes only the new path of a copy"
    fail if rename and copy are conflated.
  • "fails rather than reporting no changes when the base cannot be resolved" fails if an
    unreadable git state degrades to "nothing changed".
  • "runs everything and says which probe failed" fails if an error narrows the run.
  • "still covers the corpus when nothing has been measured" fails if unweighted files
    pile into one shard — it caught exactly that defect during development.

Manual: deno task test:affected --base HEAD after touching one source file.

Scope

Included

  • deno task test:affected and its libraries, with the full regression suite
  • deno task weights:measure and the workflow_dispatch job that runs it on ubuntu-latest

Intentionally unchanged

  • AGENTS.md still requires the full battery; affected mode is not yet the default
  • ci.yml still runs the unsharded corpus; the sharded matrix and runtime-tests
    aggregate are PR B
  • test-weights.json is not committed here — weights are recorded from a CI run in PR B
  • Deno --parallel stays out of Shard exhaustive CI runtime suites under five minutes #280: it is 3.7× faster locally and currently exits 1

New abstractions

  • Probe (scripts/lib/affected.ts) exists so selection can be driven by the real
    deno test and by a stub that fails on demand.
  • captured() (scripts/lib/captured.ts) exists because exec(...).join() forwards child
    output as it arrives, and three concurrent suites would interleave into one terminal.

Risks and limitations

Scope confirmation

  • Every changed file supports the purpose described above.
  • Unrelated cleanup and formatting changes are excluded.
  • Generated or mechanical changes are clearly identified.
  • The description matches the final diff and test results.

Part of #280

Deno decides which tests a change affects; this adds the machinery around
that answer and nothing that depends on it yet.
`deno task test:affected` reads the change set with status intact, widens the
run for every input the module graph cannot see, asks Deno which tests the
surviving TypeScript affects, and runs that one set under all three runtimes
minus their recorded exclusions.
The seam is measured, not assumed. Deno 2.9.1 exposes selection but not the
selected set: reporters lose module identity under `--parallel`, JUnit names
the module that registered each test — every suite here is attributed to
`packages/test-support/bdd.ts` — and coverage needs the run to finish. Explicit
file arguments intersect with `--changed`, so `--no-run` decides one candidate
without executing anything, and sweeping the corpus that way reproduced Deno's
own discovery-driven selection exactly. Type-checking stays on: under
`--no-check` the same sweep lost every test that reaches a change through
`import type`.
Selection cannot speak for a file in no test's graph, so `--related` proves
reachability first. A path with no static dependent — a CLI entry point tests
spawn, a component loaded through `import(file://…)`, a file nothing imports
yet — runs the whole corpus rather than reporting a narrow green.
Sharding is weighted rather than counted: `deno test --shard` splits the sorted
list into consecutive groups, which lands this corpus's heavyweights together.
`weights:measure` records what each file costs on the runner CI uses.
Part of #280
@github-actions

Copy link
Copy Markdown

PR #311: ✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two)

17 files, +1915 / -0

Scope

🔴 PR has 1915 lines changed. Split into focused PRs.

🟡 1915 lines changed. PRs under 400 receive more thorough review.

🟡 PR mixes config and source changes.

Structural

🟡 18 console statements.

Slop

✅ Slop indicators look low.

Static Analysis

✅ Oxlint found no issues.

Correctness

No extraneous code patterns detected.

@taras

taras commented Aug 3, 2026

Copy link
Copy Markdown
OwnerAuthor

Closing without merging: the prerequisite is unmet. #279 Stage 2 has not merged and #279
is still open, and the #280 ruling puts PR A after both. The branch
feat/issue-280-affected-tests is preserved at 490de0b; this is a sequencing close, not
an abandonment.

Recorded for the rebase, once #279 closes — revalidate against its final interfaces first
(the inert flag set probe and run must adopt together, the task-entry lock protection,
and the scripts/lib/ helpers), then repair:

1. Code rules. Swept the additions; exactly four violations, no others:

  • scripts/affected-tests.ts:298 and scripts/measure-weights.ts:98 — top-level await main(...)
  • scripts/affected-tests.ts:46as const on RUNTIMES
  • scripts/tests/affected-selection.test.ts:23 — the fixture's explicit Promise<unknown> annotation

2. Complete the approved automated matrix. Currently missing: cross-workspace
transitivity; the --no-check negative case as its own assertion rather than only the
positive type-only case; a real copy record and a real staged deletion (git rm) driven
through git rather than the parser alone; renames into and out of a triggering directory;
both dynamic-import cases (analyzable template and computed target); every classification
exercised through scratch selection rather than the pure table; and ignored output versus
tracked generator inputs.

3. Command-level tests. None exist today — scripts/affected-tests.ts is covered only
end to end by hand. Needed: remediation on an unreadable base, full-corpus fallback after a
probe failure, allSettled behaviour when one runtime fails, complete first-failure output,
cancellation terminating every child process, and clean state after success, failure, and
cancellation.

4. AGENTS.md stays unchanged in PR A — it already does here, and it stays that way.
Agents do not learn to run affected tests until PR B lands it beside exhaustive sharded CI.

5. Re-verification. Complete pinned battery on the rebased head, tracked-tree
cleanliness proven, fully green CI and zero unresolved threads before re-review.

Then stop after PR A. PR B changes the agent default and closes #280.

@tarastaras closed this Aug 3, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Shard exhaustive CI runtime suites under five minutes

1 participant

@taras
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two) - #311

Closed
taras wants to merge 1 commit into
mainfrom
feat/issue-280-affected-tests
Closed

✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two)#311
taras wants to merge 1 commit into
mainfrom
feat/issue-280-affected-tests

Conversation

@taras

@tarastaras commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Why

Agents run the whole corpus under three runtimes after every change, and CI spends
306–316 s doing the same. #280 rules that agents run only the tests their branch and
worktree affect, decided by Deno's native selection, while CI keeps running everything.

This is PR A of two: the mechanism, with no policy change. AGENTS.md's default and
the exhaustive CI jobs are untouched, so the command ships alongside the existing battery
and can be exercised before anything depends on it. PR B records CI-measured weights,
installs the sharded matrix and its aggregate, updates the docs, and closes#280.

What changes

Before: the only local test commands run the complete corpus — deno task test,
pnpm test:node, bun run test:bun.

After: deno task test:affected runs the tests a change affects, under all three
runtimes, from one selected set.

$ deno task test:affected --base HEAD # one changed file in packages/acp/src
changed paths (1)
M packages/acp/src/session-key.ts
selected 9 of 174 test files
d n b packages/acp/tests/acceptance.test.ts
…
results
deno passed in 47s
node passed in 56.2s
bun passed in 40.9s
this command runs tests only — still applicable:
deno task lint / check / check:jsr / compiled smoke / xmd test packages/core/src --raw
CI runs the complete corpus under every runtime. This run is not equivalent to it.

66 s against roughly ten minutes for the three full suites.

How it works

git change set → classification → reachability proof → Deno's selection → 3 runtimes
  1. Change set (scripts/lib/git-changes.ts) — merge-base diff, staged, unstaged, and
    untracked, parsed NUL-safely and status-aware with explicit -M -C. A rename
    contributes both paths, the old one as a deletion; a copy contributes only its new
    path, because the source still exists.
  2. Classification (scripts/lib/change-classes.ts) — every path lands in exactly one
    class, and anything unmatched is unknown, which runs everything. Deletions,
    configuration, dependency state, the selector itself, the shared harness, runtime-read
    documents and fixtures, and bundle inputs each run the full corpus.
  3. Reachability proof (--related) — selection says nothing about a file in no test's
    graph, so any changed TypeScript with no static dependent runs the whole corpus rather
    than reporting a narrow green.
  4. Selection (--changed + --no-run, one candidate per probe) — Deno's answer,
    captured before anything runs.
  5. Three runtimes — the same set, minus each runtime's recorded exclusions, run
    concurrently under allSettled.

Review guide

Start with:scripts/lib/affected.ts — the module doc records why every other seam
was rejected.

Then review:

  1. scripts/lib/change-classes.ts — the trigger table and its unknown fallback
  2. scripts/lib/git-changes.ts — rename versus copy, and the statuses it refuses
  3. scripts/affected-tests.ts — failure semantics and the report
  4. scripts/lib/shard.ts, scripts/measure-weights.ts — weighted partitioning (used by PR B)

Look carefully at:

  • The two different failures in scripts/affected-tests.ts: an unreadable change set
    stops with remediation; a probe that fails afterwards widens to the full corpus.
    Narrowing is never the response to an error.

What must stay true

  • Type-checking stays on in the probe — under --no-check the sweep silently loses
    every test that reaches a change only through import type. Enforced by the shared
    FLAGS and checked by "selects a test that reaches the change only through
    import type"
    .
  • Escalation is one-way — checked by the deletion, rename, fixture, document, unknown,
    unreachable, and probe-failure cases.
  • One discovery — CI will hand explicit lists to all three runners, so
    scripts/lib/test-files.ts must find everything Deno's glob would. Checked by
    scripts/tests/test-file-discovery.test.ts.
  • Every file, exactly once, in the partition — checked by scripts/tests/shard.test.ts.

How to verify it

Every correctness case runs against a scratch git repository the test builds, so no
fixture depends on the state of this repository.

  • "selects the tests that import a changed module, and nothing else" fails if selection
    widens or narrows.
  • "selects a test that reaches the change only through import type" fails if the probe
    ever adopts --no-check.
  • "runs everything when a changed module is in no test's graph" fails if a
    dynamically-loaded or subprocess-loaded file is trusted to select honestly.
  • "runs everything for a deletion, which selects nothing on its own" fails if a deleted
    path is treated as an ordinary change — the false green Deno produces natively.
  • "reports both sides of a committed rename" and "takes only the new path of a copy"
    fail if rename and copy are conflated.
  • "fails rather than reporting no changes when the base cannot be resolved" fails if an
    unreadable git state degrades to "nothing changed".
  • "runs everything and says which probe failed" fails if an error narrows the run.
  • "still covers the corpus when nothing has been measured" fails if unweighted files
    pile into one shard — it caught exactly that defect during development.

Manual: deno task test:affected --base HEAD after touching one source file.

Scope

Included

  • deno task test:affected and its libraries, with the full regression suite
  • deno task weights:measure and the workflow_dispatch job that runs it on ubuntu-latest

Intentionally unchanged

  • AGENTS.md still requires the full battery; affected mode is not yet the default
  • ci.yml still runs the unsharded corpus; the sharded matrix and runtime-tests
    aggregate are PR B
  • test-weights.json is not committed here — weights are recorded from a CI run in PR B
  • Deno --parallel stays out of Shard exhaustive CI runtime suites under five minutes #280: it is 3.7× faster locally and currently exits 1

New abstractions

  • Probe (scripts/lib/affected.ts) exists so selection can be driven by the real
    deno test and by a stub that fails on demand.
  • captured() (scripts/lib/captured.ts) exists because exec(...).join() forwards child
    output as it arrives, and three concurrent suites would interleave into one terminal.

Risks and limitations

Scope confirmation

  • Every changed file supports the purpose described above.
  • Unrelated cleanup and formatting changes are excluded.
  • Generated or mechanical changes are clearly identified.
  • The description matches the final diff and test results.

Part of #280

Deno decides which tests a change affects; this adds the machinery around
that answer and nothing that depends on it yet.
`deno task test:affected` reads the change set with status intact, widens the
run for every input the module graph cannot see, asks Deno which tests the
surviving TypeScript affects, and runs that one set under all three runtimes
minus their recorded exclusions.
The seam is measured, not assumed. Deno 2.9.1 exposes selection but not the
selected set: reporters lose module identity under `--parallel`, JUnit names
the module that registered each test — every suite here is attributed to
`packages/test-support/bdd.ts` — and coverage needs the run to finish. Explicit
file arguments intersect with `--changed`, so `--no-run` decides one candidate
without executing anything, and sweeping the corpus that way reproduced Deno's
own discovery-driven selection exactly. Type-checking stays on: under
`--no-check` the same sweep lost every test that reaches a change through
`import type`.
Selection cannot speak for a file in no test's graph, so `--related` proves
reachability first. A path with no static dependent — a CLI entry point tests
spawn, a component loaded through `import(file://…)`, a file nothing imports
yet — runs the whole corpus rather than reporting a narrow green.
Sharding is weighted rather than counted: `deno test --shard` splits the sorted
list into consecutive groups, which lands this corpus's heavyweights together.
`weights:measure` records what each file costs on the runner CI uses.
Part of #280
@github-actions

Copy link
Copy Markdown

PR #311: ✨ Run the tests a change affects, and weigh the corpus for shards (#280, PR A of two)

17 files, +1915 / -0

Scope

🔴 PR has 1915 lines changed. Split into focused PRs.

🟡 1915 lines changed. PRs under 400 receive more thorough review.

🟡 PR mixes config and source changes.

Structural

🟡 18 console statements.

Slop

✅ Slop indicators look low.

Static Analysis

✅ Oxlint found no issues.

Correctness

No extraneous code patterns detected.

@taras

taras commented Aug 3, 2026

Copy link
Copy Markdown
OwnerAuthor

Closing without merging: the prerequisite is unmet. #279 Stage 2 has not merged and #279
is still open, and the #280 ruling puts PR A after both. The branch
feat/issue-280-affected-tests is preserved at 490de0b; this is a sequencing close, not
an abandonment.

Recorded for the rebase, once #279 closes — revalidate against its final interfaces first
(the inert flag set probe and run must adopt together, the task-entry lock protection,
and the scripts/lib/ helpers), then repair:

1. Code rules. Swept the additions; exactly four violations, no others:

  • scripts/affected-tests.ts:298 and scripts/measure-weights.ts:98 — top-level await main(...)
  • scripts/affected-tests.ts:46as const on RUNTIMES
  • scripts/tests/affected-selection.test.ts:23 — the fixture's explicit Promise<unknown> annotation

2. Complete the approved automated matrix. Currently missing: cross-workspace
transitivity; the --no-check negative case as its own assertion rather than only the
positive type-only case; a real copy record and a real staged deletion (git rm) driven
through git rather than the parser alone; renames into and out of a triggering directory;
both dynamic-import cases (analyzable template and computed target); every classification
exercised through scratch selection rather than the pure table; and ignored output versus
tracked generator inputs.

3. Command-level tests. None exist today — scripts/affected-tests.ts is covered only
end to end by hand. Needed: remediation on an unreadable base, full-corpus fallback after a
probe failure, allSettled behaviour when one runtime fails, complete first-failure output,
cancellation terminating every child process, and clean state after success, failure, and
cancellation.

4. AGENTS.md stays unchanged in PR A — it already does here, and it stays that way.
Agents do not learn to run affected tests until PR B lands it beside exhaustive sharded CI.

5. Re-verification. Complete pinned battery on the rebased head, tracked-tree
cleanliness proven, fully green CI and zero unresolved threads before re-review.

Then stop after PR A. PR B changes the agent default and closes #280.

@tarastaras closed this Aug 3, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Shard exhaustive CI runtime suites under five minutes

1 participant

@taras