Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 134 additions & 10 deletions packages/cli/test/run-dev-unbuilt-workspace.e2e.test.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@

import { describe, it, expect, beforeAll, afterAll } from 'vitest';
import { execFile, spawn } from 'node:child_process';
import { mkdtempSync, rmSync } from 'node:fs';
import { mkdtempSync, readFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join, resolve } from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
Expand All@@ -96,6 +96,12 @@ interface Run {
code: number;
stdout: string;
stderr: string;
/**
* Wall clock for the whole child, spawn to callback. Read by case 5, which
* sizes its own ceiling against a run of the SAME child on the SAME runner
* rather than against a constant measured somewhere else.
*/
elapsedMs: number;
}

/**
Expand All@@ -105,9 +111,11 @@ interface Run {
* started with — and the control legs' whole job is to be un-simulated.
*/
function runCli(args: string[], cwd: string, nodeOptions: string | undefined): Promise<Run> {
const started = Date.now();
return new Promise((resolvePromise) => {
execFile(TSX, [CLI, ...args], { cwd, maxBuffer: 32 * 1024 * 1024, env: childEnv({ NO_COLOR: '1', NODE_OPTIONS: nodeOptions }) }, (err, stdout, stderr) => {
resolvePromise({
elapsedMs: Date.now() - started,
// `err.code` is the real exit status; `null`/undefined means the child
// was signalled — a failure of a different kind, never reported as 0.
code: err ? (typeof (err as { code?: unknown }).code === 'number' ? (err as unknown as { code: number }).code : 1) : 0,
Expand DownExpand Up@@ -147,10 +155,79 @@ const PIPE_BUFFER_BYTES = 65_536;
const STALL_MS = 10_000;

/**
* Ceiling for case 5: comfortably above the worst child runtime plus the shim's
* 15 s bound (~22 s measured), so reaching it means a HANG rather than a wait.
* The shim's own no-progress bound, mirrored from `bin/run-dev.js`
* (`STDERR_DRAIN_STALL_MS`) and held equal to it by a case below rather than
* trusted. Case 5's ceiling no longer budgets it — that ceiling is a constant
* now — but two cases here are still sized against it and would quietly stop
* discriminating if it moved:
*
* • `STALL_MS` above must stay strictly BELOW it, or case 4's stalled reader
* outlasts the shim's own give-up and reds against a WORKING fix;
* • case 6 reads the closed-reader path as released in less than `STALL_MS`,
* which is evidence of a fast path only while `STALL_MS` is itself below
* the bound.
*/
const UNREAD_HARD_CAP_MS = 40_000;
const SHIM_DRAIN_STALL_MS = 15_000;

/** Where that constant is written — read by the parity case, never imported. */
const SHIM = resolve(HERE, '../bin/run-dev.js');

/**
* Case 5's ceiling — a CONSTANT, and deliberately this file's existing
* per-child budget rather than a number of its own.
*
* ⚠️ Two ceilings have been tried here and both failed the same way, so the
* history is written down instead of left to be rediscovered:
*
* • `UNREAD_HARD_CAP_MS = 40_000`, read as "comfortably above the worst child
* runtime plus the shim's 15 s bound (~22 s measured)". Merge-queue shards
* running the full suite six ways sharded went over it three times in a
* day, on three trees that cannot reach this file.
* • then a per-run derivation, `clamp(40_000, RUN_TIMEOUT_MS, 4 x a case-1
* calibration + 2 x the shim's bound)`, on the theory that a contended
* shard can calibrate itself. It was evicted from the queue by its own new
* assertion at `cap 61464 ms = clamp(40000, 180000, 4 x 7866 ms measured
* child runtime + 2 x 15000 ms shim bound)`: the child outlived a ceiling
* built from a sample taken minutes earlier on that same runner by more
* than 7.8x that sample, against a FACTOR of 4.
*
* Raising the factor would be the same move a third time. Both ceilings were
* sized comfortably above the worst thing on record when they were written, and
* both were beaten by a runner that got busier afterwards. Nothing measures the
* spread between a calibration and a later run on a shared, six-way-sharded
* queue runner, so no factor can be justified as ENOUGH — only as not beaten
* yet, which is what the constant it replaced could also say.
*
* ⭐ What removes the choice is the property this case actually pins. The
* failure it was written against is an UNBOUNDED wait: a drain wait with no
* bound armed at all, observed alive at 25 s, 30 s and 60 s and ending only
* when something else killed it. ANY finite ceiling catches that. Tightening a
* ceiling buys no detection at all — it buys false reds, and each one here
* costs a queue rebuild. So the ceiling wants to be the LARGEST value that
* keeps the failure legible, and it must not track load: a term tracking load
* is a prediction about contention drawn from a sample of the past, which is
* the one thing a shared runner will not honour.
*
* `RUN_TIMEOUT_MS` is that largest legible value, and it is not a new number:
*
* • past it this case stops reporting a SIGKILL and starts reporting the
* `beforeAll` timeout, which reds all six cases and names none of them. So
* it is where legibility ends, not a preference;
* • it is already this file's budget for ONE child of this suite, and cases
* 1-4 run the same child. A child here that legitimately needs more than
* 180 s has broken the whole file, not this case — one number to get
* wrong instead of two;
* • every load figure on record clears it by an order of magnitude: 23x the
* 7.9 s calibration, and 3.4x the worst legitimate lifetime yet measured
* (22.6 s of contended work against 8 competing copies of this child, plus
* both of the shim's 15 s bounds).
*
* The measurement is KEPT — as evidence in the failure message, never as an
* input to the threshold. That is the whole correction: case 1's wall clock
* tells a triage whether a red is a hang or a runner on fire, and it decides
* nothing.
*/
const UNREAD_HARD_CAP_MS = RUN_TIMEOUT_MS;

interface Lifetime {
code: number | null;
Expand All@@ -168,6 +245,7 @@ function runCliAgainstDeadReader(
cwd: string,
nodeOptions: string,
mode: 'never-read' | 'destroy-read-end',
capMs: number,
): Promise<Lifetime> {
return new Promise((resolvePromise) => {
const child = spawn(TSX, [CLI, ...args], {
Expand All@@ -183,7 +261,7 @@ function runCliAgainstDeadReader(
const started = Date.now();
// Ours, and it must be the ONLY thing that can end a hang — a child that
// reaches it is the failure this case exists to catch.
const cap = setTimeout(() => child.kill('SIGKILL'), UNREAD_HARD_CAP_MS);
const cap = setTimeout(() => child.kill('SIGKILL'), capMs);
child.once('exit', (code, signal) => {
clearTimeout(cap);
resolvePromise({ code, signal, elapsedMs: Date.now() - started });
Expand DownExpand Up@@ -222,8 +300,11 @@ beforeAll(async () => {
built = await runCli(REAL_COMMAND, dir, undefined);
genuinelyMissing = await runCli(['definitely-not-a-command'], dir, undefined);
stalled = await runCliWhileParentStalls(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`);
unread = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'never-read');
closedEnd = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'destroy-read-end');
// ⛔ Nothing measured above is consulted here. Case 1's wall clock is read by
// the failure message below, as evidence; the ceiling is a constant, so a
// slow sample can no longer size the instrument that judges the next run.
unread = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'never-read', UNREAD_HARD_CAP_MS);
closedEnd = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'destroy-read-end', UNREAD_HARD_CAP_MS);
}, RUN_TIMEOUT_MS * 6);

afterAll(() => {
Expand DownExpand Up@@ -314,9 +395,28 @@ describe('the mirror direction: a reader that is never coming back', () => {
it('gives up and exits instead of waiting forever', () => {
// A child still alive at the cap was SIGKILLed: signal set, code null.
// That is the hang, and it is the whole point of this case.
expect(unread.signal).toBeNull();
expect(unread.code).toBe(2);
expect(unread.elapsedMs).toBeLessThan(UNREAD_HARD_CAP_MS);
//
// ⚠️ Both surviving assertions are about the PRODUCT: the child ends on
// its OWN, and it ends with the status any other reader would have got.
//
// ⛔ The third assertion this case used to carry — `elapsedMs` below the
// ceiling — is deliberately gone. Against a constant cap it asserts nothing
// the first line does not: the harness kills at exactly that cap, so a
// child that was not killed ran less than it. What it added was a race, at
// the one instant where a child exiting on its own and the timer firing
// are simultaneous, and it was the only reading here that a slower box
// could move on its own. Detection unchanged, one fewer way to red.
//
// The numbers move into the message, because `expected 'SIGKILL' to be
// null` alone does not tell a merge-queue triage which of two readings it
// has. A child killed at 180 s whose calibration was 8 s is a hang; one
// whose calibration was also minutes indicts the runner, not this code.
const evidence =
`cap ${UNREAD_HARD_CAP_MS} ms (RUN_TIMEOUT_MS, constant and load-independent by design); ` +
`this child ran ${unread.elapsedMs} ms; case 1 measured the same child at ` +
`${unbuilt.elapsedMs} ms on this runner minutes earlier`;
expect(unread.signal, `the harness SIGKILLed the child — it was still alive at the ceiling. ${evidence}`).toBeNull();
expect(unread.code, `the child did not exit 2 on its own. ${evidence}`).toBe(2);
});

// ⛔ There is deliberately NO assertion here that the child WAITED for the
Expand All@@ -332,6 +432,30 @@ describe('the mirror direction: a reader that is never coming back', () => {
// process ENDS. That the bound itself runs and trips is shown out of band,
// by tracing a run whose reader blocks its loop for the whole run — see the
// PR for the `BOUND TRIPPED` trace.
//
// ⚠️ The same nondeterminism means an ABLATION of the bound can come back
// GREEN, and a single green one here is a ZERO READING rather than evidence
// this case has stopped discriminating. Measured on one box minutes apart,
// same tree: disabling the no-progress branch red this case at 180072 ms
// once, and passed it in 31.9 s the run before — that run's backlog fit in
// what the kernel and node happened to absorb, so the write callback
// resolved on its own and the branch was never reached. Re-run it, or drive
// the child OUT OF BAND (`spawn`, `stderr.pause()`, never read) where the
// pending bytes can actually be counted: 135408 bytes still held at a clean
// exit 2 in 19271 ms, against 145638 held by a child still alive at 90 s
// with the branch disabled.

it("keeps its mirror of the shim's bound equal to the shim's own", () => {
// The derivation above is only as good as `SHIM_DRAIN_STALL_MS` still being
// what `bin/run-dev.js` waits. There is no import to take it from — that
// file runs the CLI at module top — so it is mirrored, and a mirror with
// nothing holding it is how a ceiling ends up sized around a bound that
// moved. Same discipline as `INVOCATION_PREFIX` vs `CLI_NAME`: kept in
// sync by a case, not by an import.
const declared = /const STDERR_DRAIN_STALL_MS = ([\d_]+);/.exec(readFileSync(SHIM, 'utf8'))?.[1];
expect(declared, `no STDERR_DRAIN_STALL_MS declaration found in ${SHIM}`).toBeDefined();
expect(Number(String(declared).replaceAll('_', ''))).toBe(SHIM_DRAIN_STALL_MS);
});

it('a CLOSED read end is released at once, not held for the bound (EPIPE reaches the callback)', () => {
// Pins the fast path measured alongside the hang: when the reader is gone
Expand Down
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 134 additions & 10 deletions packages/cli/test/run-dev-unbuilt-workspace.e2e.test.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@

import { describe, it, expect, beforeAll, afterAll } from 'vitest';
import { execFile, spawn } from 'node:child_process';
import { mkdtempSync, rmSync } from 'node:fs';
import { mkdtempSync, readFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join, resolve } from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
Expand All@@ -96,6 +96,12 @@ interface Run {
code: number;
stdout: string;
stderr: string;
/**
* Wall clock for the whole child, spawn to callback. Read by case 5, which
* sizes its own ceiling against a run of the SAME child on the SAME runner
* rather than against a constant measured somewhere else.
*/
elapsedMs: number;
}

/**
Expand All@@ -105,9 +111,11 @@ interface Run {
* started with — and the control legs' whole job is to be un-simulated.
*/
function runCli(args: string[], cwd: string, nodeOptions: string | undefined): Promise<Run> {
const started = Date.now();
return new Promise((resolvePromise) => {
execFile(TSX, [CLI, ...args], { cwd, maxBuffer: 32 * 1024 * 1024, env: childEnv({ NO_COLOR: '1', NODE_OPTIONS: nodeOptions }) }, (err, stdout, stderr) => {
resolvePromise({
elapsedMs: Date.now() - started,
// `err.code` is the real exit status; `null`/undefined means the child
// was signalled — a failure of a different kind, never reported as 0.
code: err ? (typeof (err as { code?: unknown }).code === 'number' ? (err as unknown as { code: number }).code : 1) : 0,
Expand DownExpand Up@@ -147,10 +155,79 @@ const PIPE_BUFFER_BYTES = 65_536;
const STALL_MS = 10_000;

/**
* Ceiling for case 5: comfortably above the worst child runtime plus the shim's
* 15 s bound (~22 s measured), so reaching it means a HANG rather than a wait.
* The shim's own no-progress bound, mirrored from `bin/run-dev.js`
* (`STDERR_DRAIN_STALL_MS`) and held equal to it by a case below rather than
* trusted. Case 5's ceiling no longer budgets it — that ceiling is a constant
* now — but two cases here are still sized against it and would quietly stop
* discriminating if it moved:
*
* • `STALL_MS` above must stay strictly BELOW it, or case 4's stalled reader
* outlasts the shim's own give-up and reds against a WORKING fix;
* • case 6 reads the closed-reader path as released in less than `STALL_MS`,
* which is evidence of a fast path only while `STALL_MS` is itself below
* the bound.
*/
const UNREAD_HARD_CAP_MS = 40_000;
const SHIM_DRAIN_STALL_MS = 15_000;

/** Where that constant is written — read by the parity case, never imported. */
const SHIM = resolve(HERE, '../bin/run-dev.js');

/**
* Case 5's ceiling — a CONSTANT, and deliberately this file's existing
* per-child budget rather than a number of its own.
*
* ⚠️ Two ceilings have been tried here and both failed the same way, so the
* history is written down instead of left to be rediscovered:
*
* • `UNREAD_HARD_CAP_MS = 40_000`, read as "comfortably above the worst child
* runtime plus the shim's 15 s bound (~22 s measured)". Merge-queue shards
* running the full suite six ways sharded went over it three times in a
* day, on three trees that cannot reach this file.
* • then a per-run derivation, `clamp(40_000, RUN_TIMEOUT_MS, 4 x a case-1
* calibration + 2 x the shim's bound)`, on the theory that a contended
* shard can calibrate itself. It was evicted from the queue by its own new
* assertion at `cap 61464 ms = clamp(40000, 180000, 4 x 7866 ms measured
* child runtime + 2 x 15000 ms shim bound)`: the child outlived a ceiling
* built from a sample taken minutes earlier on that same runner by more
* than 7.8x that sample, against a FACTOR of 4.
*
* Raising the factor would be the same move a third time. Both ceilings were
* sized comfortably above the worst thing on record when they were written, and
* both were beaten by a runner that got busier afterwards. Nothing measures the
* spread between a calibration and a later run on a shared, six-way-sharded
* queue runner, so no factor can be justified as ENOUGH — only as not beaten
* yet, which is what the constant it replaced could also say.
*
* ⭐ What removes the choice is the property this case actually pins. The
* failure it was written against is an UNBOUNDED wait: a drain wait with no
* bound armed at all, observed alive at 25 s, 30 s and 60 s and ending only
* when something else killed it. ANY finite ceiling catches that. Tightening a
* ceiling buys no detection at all — it buys false reds, and each one here
* costs a queue rebuild. So the ceiling wants to be the LARGEST value that
* keeps the failure legible, and it must not track load: a term tracking load
* is a prediction about contention drawn from a sample of the past, which is
* the one thing a shared runner will not honour.
*
* `RUN_TIMEOUT_MS` is that largest legible value, and it is not a new number:
*
* • past it this case stops reporting a SIGKILL and starts reporting the
* `beforeAll` timeout, which reds all six cases and names none of them. So
* it is where legibility ends, not a preference;
* • it is already this file's budget for ONE child of this suite, and cases
* 1-4 run the same child. A child here that legitimately needs more than
* 180 s has broken the whole file, not this case — one number to get
* wrong instead of two;
* • every load figure on record clears it by an order of magnitude: 23x the
* 7.9 s calibration, and 3.4x the worst legitimate lifetime yet measured
* (22.6 s of contended work against 8 competing copies of this child, plus
* both of the shim's 15 s bounds).
*
* The measurement is KEPT — as evidence in the failure message, never as an
* input to the threshold. That is the whole correction: case 1's wall clock
* tells a triage whether a red is a hang or a runner on fire, and it decides
* nothing.
*/
const UNREAD_HARD_CAP_MS = RUN_TIMEOUT_MS;

interface Lifetime {
code: number | null;
Expand All@@ -168,6 +245,7 @@ function runCliAgainstDeadReader(
cwd: string,
nodeOptions: string,
mode: 'never-read' | 'destroy-read-end',
capMs: number,
): Promise<Lifetime> {
return new Promise((resolvePromise) => {
const child = spawn(TSX, [CLI, ...args], {
Expand All@@ -183,7 +261,7 @@ function runCliAgainstDeadReader(
const started = Date.now();
// Ours, and it must be the ONLY thing that can end a hang — a child that
// reaches it is the failure this case exists to catch.
const cap = setTimeout(() => child.kill('SIGKILL'), UNREAD_HARD_CAP_MS);
const cap = setTimeout(() => child.kill('SIGKILL'), capMs);
child.once('exit', (code, signal) => {
clearTimeout(cap);
resolvePromise({ code, signal, elapsedMs: Date.now() - started });
Expand DownExpand Up@@ -222,8 +300,11 @@ beforeAll(async () => {
built = await runCli(REAL_COMMAND, dir, undefined);
genuinelyMissing = await runCli(['definitely-not-a-command'], dir, undefined);
stalled = await runCliWhileParentStalls(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`);
unread = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'never-read');
closedEnd = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'destroy-read-end');
// ⛔ Nothing measured above is consulted here. Case 1's wall clock is read by
// the failure message below, as evidence; the ceiling is a constant, so a
// slow sample can no longer size the instrument that judges the next run.
unread = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'never-read', UNREAD_HARD_CAP_MS);
closedEnd = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'destroy-read-end', UNREAD_HARD_CAP_MS);
}, RUN_TIMEOUT_MS * 6);

afterAll(() => {
Expand DownExpand Up@@ -314,9 +395,28 @@ describe('the mirror direction: a reader that is never coming back', () => {
it('gives up and exits instead of waiting forever', () => {
// A child still alive at the cap was SIGKILLed: signal set, code null.
// That is the hang, and it is the whole point of this case.
expect(unread.signal).toBeNull();
expect(unread.code).toBe(2);
expect(unread.elapsedMs).toBeLessThan(UNREAD_HARD_CAP_MS);
//
// ⚠️ Both surviving assertions are about the PRODUCT: the child ends on
// its OWN, and it ends with the status any other reader would have got.
//
// ⛔ The third assertion this case used to carry — `elapsedMs` below the
// ceiling — is deliberately gone. Against a constant cap it asserts nothing
// the first line does not: the harness kills at exactly that cap, so a
// child that was not killed ran less than it. What it added was a race, at
// the one instant where a child exiting on its own and the timer firing
// are simultaneous, and it was the only reading here that a slower box
// could move on its own. Detection unchanged, one fewer way to red.
//
// The numbers move into the message, because `expected 'SIGKILL' to be
// null` alone does not tell a merge-queue triage which of two readings it
// has. A child killed at 180 s whose calibration was 8 s is a hang; one
// whose calibration was also minutes indicts the runner, not this code.
const evidence =
`cap ${UNREAD_HARD_CAP_MS} ms (RUN_TIMEOUT_MS, constant and load-independent by design); ` +
`this child ran ${unread.elapsedMs} ms; case 1 measured the same child at ` +
`${unbuilt.elapsedMs} ms on this runner minutes earlier`;
expect(unread.signal, `the harness SIGKILLed the child — it was still alive at the ceiling. ${evidence}`).toBeNull();
expect(unread.code, `the child did not exit 2 on its own. ${evidence}`).toBe(2);
});

// ⛔ There is deliberately NO assertion here that the child WAITED for the
Expand All@@ -332,6 +432,30 @@ describe('the mirror direction: a reader that is never coming back', () => {
// process ENDS. That the bound itself runs and trips is shown out of band,
// by tracing a run whose reader blocks its loop for the whole run — see the
// PR for the `BOUND TRIPPED` trace.
//
// ⚠️ The same nondeterminism means an ABLATION of the bound can come back
// GREEN, and a single green one here is a ZERO READING rather than evidence
// this case has stopped discriminating. Measured on one box minutes apart,
// same tree: disabling the no-progress branch red this case at 180072 ms
// once, and passed it in 31.9 s the run before — that run's backlog fit in
// what the kernel and node happened to absorb, so the write callback
// resolved on its own and the branch was never reached. Re-run it, or drive
// the child OUT OF BAND (`spawn`, `stderr.pause()`, never read) where the
// pending bytes can actually be counted: 135408 bytes still held at a clean
// exit 2 in 19271 ms, against 145638 held by a child still alive at 90 s
// with the branch disabled.

it("keeps its mirror of the shim's bound equal to the shim's own", () => {
// The derivation above is only as good as `SHIM_DRAIN_STALL_MS` still being
// what `bin/run-dev.js` waits. There is no import to take it from — that
// file runs the CLI at module top — so it is mirrored, and a mirror with
// nothing holding it is how a ceiling ends up sized around a bound that
// moved. Same discipline as `INVOCATION_PREFIX` vs `CLI_NAME`: kept in
// sync by a case, not by an import.
const declared = /const STDERR_DRAIN_STALL_MS = ([\d_]+);/.exec(readFileSync(SHIM, 'utf8'))?.[1];
expect(declared, `no STDERR_DRAIN_STALL_MS declaration found in ${SHIM}`).toBeDefined();
expect(Number(String(declared).replaceAll('_', ''))).toBe(SHIM_DRAIN_STALL_MS);
});

it('a CLOSED read end is released at once, not held for the bound (EPIPE reaches the callback)', () => {
// Pins the fast path measured alongside the hang: when the reader is gone
Expand Down
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 134 additions & 10 deletions packages/cli/test/run-dev-unbuilt-workspace.e2e.test.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@

import { describe, it, expect, beforeAll, afterAll } from 'vitest';
import { execFile, spawn } from 'node:child_process';
import { mkdtempSync, rmSync } from 'node:fs';
import { mkdtempSync, readFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join, resolve } from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
Expand All@@ -96,6 +96,12 @@ interface Run {
code: number;
stdout: string;
stderr: string;
/**
* Wall clock for the whole child, spawn to callback. Read by case 5, which
* sizes its own ceiling against a run of the SAME child on the SAME runner
* rather than against a constant measured somewhere else.
*/
elapsedMs: number;
}

/**
Expand All@@ -105,9 +111,11 @@ interface Run {
* started with — and the control legs' whole job is to be un-simulated.
*/
function runCli(args: string[], cwd: string, nodeOptions: string | undefined): Promise<Run> {
const started = Date.now();
return new Promise((resolvePromise) => {
execFile(TSX, [CLI, ...args], { cwd, maxBuffer: 32 * 1024 * 1024, env: childEnv({ NO_COLOR: '1', NODE_OPTIONS: nodeOptions }) }, (err, stdout, stderr) => {
resolvePromise({
elapsedMs: Date.now() - started,
// `err.code` is the real exit status; `null`/undefined means the child
// was signalled — a failure of a different kind, never reported as 0.
code: err ? (typeof (err as { code?: unknown }).code === 'number' ? (err as unknown as { code: number }).code : 1) : 0,
Expand DownExpand Up@@ -147,10 +155,79 @@ const PIPE_BUFFER_BYTES = 65_536;
const STALL_MS = 10_000;

/**
* Ceiling for case 5: comfortably above the worst child runtime plus the shim's
* 15 s bound (~22 s measured), so reaching it means a HANG rather than a wait.
* The shim's own no-progress bound, mirrored from `bin/run-dev.js`
* (`STDERR_DRAIN_STALL_MS`) and held equal to it by a case below rather than
* trusted. Case 5's ceiling no longer budgets it — that ceiling is a constant
* now — but two cases here are still sized against it and would quietly stop
* discriminating if it moved:
*
* • `STALL_MS` above must stay strictly BELOW it, or case 4's stalled reader
* outlasts the shim's own give-up and reds against a WORKING fix;
* • case 6 reads the closed-reader path as released in less than `STALL_MS`,
* which is evidence of a fast path only while `STALL_MS` is itself below
* the bound.
*/
const UNREAD_HARD_CAP_MS = 40_000;
const SHIM_DRAIN_STALL_MS = 15_000;

/** Where that constant is written — read by the parity case, never imported. */
const SHIM = resolve(HERE, '../bin/run-dev.js');

/**
* Case 5's ceiling — a CONSTANT, and deliberately this file's existing
* per-child budget rather than a number of its own.
*
* ⚠️ Two ceilings have been tried here and both failed the same way, so the
* history is written down instead of left to be rediscovered:
*
* • `UNREAD_HARD_CAP_MS = 40_000`, read as "comfortably above the worst child
* runtime plus the shim's 15 s bound (~22 s measured)". Merge-queue shards
* running the full suite six ways sharded went over it three times in a
* day, on three trees that cannot reach this file.
* • then a per-run derivation, `clamp(40_000, RUN_TIMEOUT_MS, 4 x a case-1
* calibration + 2 x the shim's bound)`, on the theory that a contended
* shard can calibrate itself. It was evicted from the queue by its own new
* assertion at `cap 61464 ms = clamp(40000, 180000, 4 x 7866 ms measured
* child runtime + 2 x 15000 ms shim bound)`: the child outlived a ceiling
* built from a sample taken minutes earlier on that same runner by more
* than 7.8x that sample, against a FACTOR of 4.
*
* Raising the factor would be the same move a third time. Both ceilings were
* sized comfortably above the worst thing on record when they were written, and
* both were beaten by a runner that got busier afterwards. Nothing measures the
* spread between a calibration and a later run on a shared, six-way-sharded
* queue runner, so no factor can be justified as ENOUGH — only as not beaten
* yet, which is what the constant it replaced could also say.
*
* ⭐ What removes the choice is the property this case actually pins. The
* failure it was written against is an UNBOUNDED wait: a drain wait with no
* bound armed at all, observed alive at 25 s, 30 s and 60 s and ending only
* when something else killed it. ANY finite ceiling catches that. Tightening a
* ceiling buys no detection at all — it buys false reds, and each one here
* costs a queue rebuild. So the ceiling wants to be the LARGEST value that
* keeps the failure legible, and it must not track load: a term tracking load
* is a prediction about contention drawn from a sample of the past, which is
* the one thing a shared runner will not honour.
*
* `RUN_TIMEOUT_MS` is that largest legible value, and it is not a new number:
*
* • past it this case stops reporting a SIGKILL and starts reporting the
* `beforeAll` timeout, which reds all six cases and names none of them. So
* it is where legibility ends, not a preference;
* • it is already this file's budget for ONE child of this suite, and cases
* 1-4 run the same child. A child here that legitimately needs more than
* 180 s has broken the whole file, not this case — one number to get
* wrong instead of two;
* • every load figure on record clears it by an order of magnitude: 23x the
* 7.9 s calibration, and 3.4x the worst legitimate lifetime yet measured
* (22.6 s of contended work against 8 competing copies of this child, plus
* both of the shim's 15 s bounds).
*
* The measurement is KEPT — as evidence in the failure message, never as an
* input to the threshold. That is the whole correction: case 1's wall clock
* tells a triage whether a red is a hang or a runner on fire, and it decides
* nothing.
*/
const UNREAD_HARD_CAP_MS = RUN_TIMEOUT_MS;

interface Lifetime {
code: number | null;
Expand All@@ -168,6 +245,7 @@ function runCliAgainstDeadReader(
cwd: string,
nodeOptions: string,
mode: 'never-read' | 'destroy-read-end',
capMs: number,
): Promise<Lifetime> {
return new Promise((resolvePromise) => {
const child = spawn(TSX, [CLI, ...args], {
Expand All@@ -183,7 +261,7 @@ function runCliAgainstDeadReader(
const started = Date.now();
// Ours, and it must be the ONLY thing that can end a hang — a child that
// reaches it is the failure this case exists to catch.
const cap = setTimeout(() => child.kill('SIGKILL'), UNREAD_HARD_CAP_MS);
const cap = setTimeout(() => child.kill('SIGKILL'), capMs);
child.once('exit', (code, signal) => {
clearTimeout(cap);
resolvePromise({ code, signal, elapsedMs: Date.now() - started });
Expand DownExpand Up@@ -222,8 +300,11 @@ beforeAll(async () => {
built = await runCli(REAL_COMMAND, dir, undefined);
genuinelyMissing = await runCli(['definitely-not-a-command'], dir, undefined);
stalled = await runCliWhileParentStalls(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`);
unread = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'never-read');
closedEnd = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'destroy-read-end');
// ⛔ Nothing measured above is consulted here. Case 1's wall clock is read by
// the failure message below, as evidence; the ceiling is a constant, so a
// slow sample can no longer size the instrument that judges the next run.
unread = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'never-read', UNREAD_HARD_CAP_MS);
closedEnd = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'destroy-read-end', UNREAD_HARD_CAP_MS);
}, RUN_TIMEOUT_MS * 6);

afterAll(() => {
Expand DownExpand Up@@ -314,9 +395,28 @@ describe('the mirror direction: a reader that is never coming back', () => {
it('gives up and exits instead of waiting forever', () => {
// A child still alive at the cap was SIGKILLed: signal set, code null.
// That is the hang, and it is the whole point of this case.
expect(unread.signal).toBeNull();
expect(unread.code).toBe(2);
expect(unread.elapsedMs).toBeLessThan(UNREAD_HARD_CAP_MS);
//
// ⚠️ Both surviving assertions are about the PRODUCT: the child ends on
// its OWN, and it ends with the status any other reader would have got.
//
// ⛔ The third assertion this case used to carry — `elapsedMs` below the
// ceiling — is deliberately gone. Against a constant cap it asserts nothing
// the first line does not: the harness kills at exactly that cap, so a
// child that was not killed ran less than it. What it added was a race, at
// the one instant where a child exiting on its own and the timer firing
// are simultaneous, and it was the only reading here that a slower box
// could move on its own. Detection unchanged, one fewer way to red.
//
// The numbers move into the message, because `expected 'SIGKILL' to be
// null` alone does not tell a merge-queue triage which of two readings it
// has. A child killed at 180 s whose calibration was 8 s is a hang; one
// whose calibration was also minutes indicts the runner, not this code.
const evidence =
`cap ${UNREAD_HARD_CAP_MS} ms (RUN_TIMEOUT_MS, constant and load-independent by design); ` +
`this child ran ${unread.elapsedMs} ms; case 1 measured the same child at ` +
`${unbuilt.elapsedMs} ms on this runner minutes earlier`;
expect(unread.signal, `the harness SIGKILLed the child — it was still alive at the ceiling. ${evidence}`).toBeNull();
expect(unread.code, `the child did not exit 2 on its own. ${evidence}`).toBe(2);
});

// ⛔ There is deliberately NO assertion here that the child WAITED for the
Expand All@@ -332,6 +432,30 @@ describe('the mirror direction: a reader that is never coming back', () => {
// process ENDS. That the bound itself runs and trips is shown out of band,
// by tracing a run whose reader blocks its loop for the whole run — see the
// PR for the `BOUND TRIPPED` trace.
//
// ⚠️ The same nondeterminism means an ABLATION of the bound can come back
// GREEN, and a single green one here is a ZERO READING rather than evidence
// this case has stopped discriminating. Measured on one box minutes apart,
// same tree: disabling the no-progress branch red this case at 180072 ms
// once, and passed it in 31.9 s the run before — that run's backlog fit in
// what the kernel and node happened to absorb, so the write callback
// resolved on its own and the branch was never reached. Re-run it, or drive
// the child OUT OF BAND (`spawn`, `stderr.pause()`, never read) where the
// pending bytes can actually be counted: 135408 bytes still held at a clean
// exit 2 in 19271 ms, against 145638 held by a child still alive at 90 s
// with the branch disabled.

it("keeps its mirror of the shim's bound equal to the shim's own", () => {
// The derivation above is only as good as `SHIM_DRAIN_STALL_MS` still being
// what `bin/run-dev.js` waits. There is no import to take it from — that
// file runs the CLI at module top — so it is mirrored, and a mirror with
// nothing holding it is how a ceiling ends up sized around a bound that
// moved. Same discipline as `INVOCATION_PREFIX` vs `CLI_NAME`: kept in
// sync by a case, not by an import.
const declared = /const STDERR_DRAIN_STALL_MS = ([\d_]+);/.exec(readFileSync(SHIM, 'utf8'))?.[1];
expect(declared, `no STDERR_DRAIN_STALL_MS declaration found in ${SHIM}`).toBeDefined();
expect(Number(String(declared).replaceAll('_', ''))).toBe(SHIM_DRAIN_STALL_MS);
});

it('a CLOSED read end is released at once, not held for the bound (EPIPE reaches the callback)', () => {
// Pins the fast path measured alongside the hang: when the reader is gone
Expand Down
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 134 additions & 10 deletions packages/cli/test/run-dev-unbuilt-workspace.e2e.test.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@

import { describe, it, expect, beforeAll, afterAll } from 'vitest';
import { execFile, spawn } from 'node:child_process';
import { mkdtempSync, rmSync } from 'node:fs';
import { mkdtempSync, readFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join, resolve } from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
Expand All@@ -96,6 +96,12 @@ interface Run {
code: number;
stdout: string;
stderr: string;
/**
* Wall clock for the whole child, spawn to callback. Read by case 5, which
* sizes its own ceiling against a run of the SAME child on the SAME runner
* rather than against a constant measured somewhere else.
*/
elapsedMs: number;
}

/**
Expand All@@ -105,9 +111,11 @@ interface Run {
* started with — and the control legs' whole job is to be un-simulated.
*/
function runCli(args: string[], cwd: string, nodeOptions: string | undefined): Promise<Run> {
const started = Date.now();
return new Promise((resolvePromise) => {
execFile(TSX, [CLI, ...args], { cwd, maxBuffer: 32 * 1024 * 1024, env: childEnv({ NO_COLOR: '1', NODE_OPTIONS: nodeOptions }) }, (err, stdout, stderr) => {
resolvePromise({
elapsedMs: Date.now() - started,
// `err.code` is the real exit status; `null`/undefined means the child
// was signalled — a failure of a different kind, never reported as 0.
code: err ? (typeof (err as { code?: unknown }).code === 'number' ? (err as unknown as { code: number }).code : 1) : 0,
Expand DownExpand Up@@ -147,10 +155,79 @@ const PIPE_BUFFER_BYTES = 65_536;
const STALL_MS = 10_000;

/**
* Ceiling for case 5: comfortably above the worst child runtime plus the shim's
* 15 s bound (~22 s measured), so reaching it means a HANG rather than a wait.
* The shim's own no-progress bound, mirrored from `bin/run-dev.js`
* (`STDERR_DRAIN_STALL_MS`) and held equal to it by a case below rather than
* trusted. Case 5's ceiling no longer budgets it — that ceiling is a constant
* now — but two cases here are still sized against it and would quietly stop
* discriminating if it moved:
*
* • `STALL_MS` above must stay strictly BELOW it, or case 4's stalled reader
* outlasts the shim's own give-up and reds against a WORKING fix;
* • case 6 reads the closed-reader path as released in less than `STALL_MS`,
* which is evidence of a fast path only while `STALL_MS` is itself below
* the bound.
*/
const UNREAD_HARD_CAP_MS = 40_000;
const SHIM_DRAIN_STALL_MS = 15_000;

/** Where that constant is written — read by the parity case, never imported. */
const SHIM = resolve(HERE, '../bin/run-dev.js');

/**
* Case 5's ceiling — a CONSTANT, and deliberately this file's existing
* per-child budget rather than a number of its own.
*
* ⚠️ Two ceilings have been tried here and both failed the same way, so the
* history is written down instead of left to be rediscovered:
*
* • `UNREAD_HARD_CAP_MS = 40_000`, read as "comfortably above the worst child
* runtime plus the shim's 15 s bound (~22 s measured)". Merge-queue shards
* running the full suite six ways sharded went over it three times in a
* day, on three trees that cannot reach this file.
* • then a per-run derivation, `clamp(40_000, RUN_TIMEOUT_MS, 4 x a case-1
* calibration + 2 x the shim's bound)`, on the theory that a contended
* shard can calibrate itself. It was evicted from the queue by its own new
* assertion at `cap 61464 ms = clamp(40000, 180000, 4 x 7866 ms measured
* child runtime + 2 x 15000 ms shim bound)`: the child outlived a ceiling
* built from a sample taken minutes earlier on that same runner by more
* than 7.8x that sample, against a FACTOR of 4.
*
* Raising the factor would be the same move a third time. Both ceilings were
* sized comfortably above the worst thing on record when they were written, and
* both were beaten by a runner that got busier afterwards. Nothing measures the
* spread between a calibration and a later run on a shared, six-way-sharded
* queue runner, so no factor can be justified as ENOUGH — only as not beaten
* yet, which is what the constant it replaced could also say.
*
* ⭐ What removes the choice is the property this case actually pins. The
* failure it was written against is an UNBOUNDED wait: a drain wait with no
* bound armed at all, observed alive at 25 s, 30 s and 60 s and ending only
* when something else killed it. ANY finite ceiling catches that. Tightening a
* ceiling buys no detection at all — it buys false reds, and each one here
* costs a queue rebuild. So the ceiling wants to be the LARGEST value that
* keeps the failure legible, and it must not track load: a term tracking load
* is a prediction about contention drawn from a sample of the past, which is
* the one thing a shared runner will not honour.
*
* `RUN_TIMEOUT_MS` is that largest legible value, and it is not a new number:
*
* • past it this case stops reporting a SIGKILL and starts reporting the
* `beforeAll` timeout, which reds all six cases and names none of them. So
* it is where legibility ends, not a preference;
* • it is already this file's budget for ONE child of this suite, and cases
* 1-4 run the same child. A child here that legitimately needs more than
* 180 s has broken the whole file, not this case — one number to get
* wrong instead of two;
* • every load figure on record clears it by an order of magnitude: 23x the
* 7.9 s calibration, and 3.4x the worst legitimate lifetime yet measured
* (22.6 s of contended work against 8 competing copies of this child, plus
* both of the shim's 15 s bounds).
*
* The measurement is KEPT — as evidence in the failure message, never as an
* input to the threshold. That is the whole correction: case 1's wall clock
* tells a triage whether a red is a hang or a runner on fire, and it decides
* nothing.
*/
const UNREAD_HARD_CAP_MS = RUN_TIMEOUT_MS;

interface Lifetime {
code: number | null;
Expand All@@ -168,6 +245,7 @@ function runCliAgainstDeadReader(
cwd: string,
nodeOptions: string,
mode: 'never-read' | 'destroy-read-end',
capMs: number,
): Promise<Lifetime> {
return new Promise((resolvePromise) => {
const child = spawn(TSX, [CLI, ...args], {
Expand All@@ -183,7 +261,7 @@ function runCliAgainstDeadReader(
const started = Date.now();
// Ours, and it must be the ONLY thing that can end a hang — a child that
// reaches it is the failure this case exists to catch.
const cap = setTimeout(() => child.kill('SIGKILL'), UNREAD_HARD_CAP_MS);
const cap = setTimeout(() => child.kill('SIGKILL'), capMs);
child.once('exit', (code, signal) => {
clearTimeout(cap);
resolvePromise({ code, signal, elapsedMs: Date.now() - started });
Expand DownExpand Up@@ -222,8 +300,11 @@ beforeAll(async () => {
built = await runCli(REAL_COMMAND, dir, undefined);
genuinelyMissing = await runCli(['definitely-not-a-command'], dir, undefined);
stalled = await runCliWhileParentStalls(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`);
unread = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'never-read');
closedEnd = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'destroy-read-end');
// ⛔ Nothing measured above is consulted here. Case 1's wall clock is read by
// the failure message below, as evidence; the ceiling is a constant, so a
// slow sample can no longer size the instrument that judges the next run.
unread = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'never-read', UNREAD_HARD_CAP_MS);
closedEnd = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'destroy-read-end', UNREAD_HARD_CAP_MS);
}, RUN_TIMEOUT_MS * 6);

afterAll(() => {
Expand DownExpand Up@@ -314,9 +395,28 @@ describe('the mirror direction: a reader that is never coming back', () => {
it('gives up and exits instead of waiting forever', () => {
// A child still alive at the cap was SIGKILLed: signal set, code null.
// That is the hang, and it is the whole point of this case.
expect(unread.signal).toBeNull();
expect(unread.code).toBe(2);
expect(unread.elapsedMs).toBeLessThan(UNREAD_HARD_CAP_MS);
//
// ⚠️ Both surviving assertions are about the PRODUCT: the child ends on
// its OWN, and it ends with the status any other reader would have got.
//
// ⛔ The third assertion this case used to carry — `elapsedMs` below the
// ceiling — is deliberately gone. Against a constant cap it asserts nothing
// the first line does not: the harness kills at exactly that cap, so a
// child that was not killed ran less than it. What it added was a race, at
// the one instant where a child exiting on its own and the timer firing
// are simultaneous, and it was the only reading here that a slower box
// could move on its own. Detection unchanged, one fewer way to red.
//
// The numbers move into the message, because `expected 'SIGKILL' to be
// null` alone does not tell a merge-queue triage which of two readings it
// has. A child killed at 180 s whose calibration was 8 s is a hang; one
// whose calibration was also minutes indicts the runner, not this code.
const evidence =
`cap ${UNREAD_HARD_CAP_MS} ms (RUN_TIMEOUT_MS, constant and load-independent by design); ` +
`this child ran ${unread.elapsedMs} ms; case 1 measured the same child at ` +
`${unbuilt.elapsedMs} ms on this runner minutes earlier`;
expect(unread.signal, `the harness SIGKILLed the child — it was still alive at the ceiling. ${evidence}`).toBeNull();
expect(unread.code, `the child did not exit 2 on its own. ${evidence}`).toBe(2);
});

// ⛔ There is deliberately NO assertion here that the child WAITED for the
Expand All@@ -332,6 +432,30 @@ describe('the mirror direction: a reader that is never coming back', () => {
// process ENDS. That the bound itself runs and trips is shown out of band,
// by tracing a run whose reader blocks its loop for the whole run — see the
// PR for the `BOUND TRIPPED` trace.
//
// ⚠️ The same nondeterminism means an ABLATION of the bound can come back
// GREEN, and a single green one here is a ZERO READING rather than evidence
// this case has stopped discriminating. Measured on one box minutes apart,
// same tree: disabling the no-progress branch red this case at 180072 ms
// once, and passed it in 31.9 s the run before — that run's backlog fit in
// what the kernel and node happened to absorb, so the write callback
// resolved on its own and the branch was never reached. Re-run it, or drive
// the child OUT OF BAND (`spawn`, `stderr.pause()`, never read) where the
// pending bytes can actually be counted: 135408 bytes still held at a clean
// exit 2 in 19271 ms, against 145638 held by a child still alive at 90 s
// with the branch disabled.

it("keeps its mirror of the shim's bound equal to the shim's own", () => {
// The derivation above is only as good as `SHIM_DRAIN_STALL_MS` still being
// what `bin/run-dev.js` waits. There is no import to take it from — that
// file runs the CLI at module top — so it is mirrored, and a mirror with
// nothing holding it is how a ceiling ends up sized around a bound that
// moved. Same discipline as `INVOCATION_PREFIX` vs `CLI_NAME`: kept in
// sync by a case, not by an import.
const declared = /const STDERR_DRAIN_STALL_MS = ([\d_]+);/.exec(readFileSync(SHIM, 'utf8'))?.[1];
expect(declared, `no STDERR_DRAIN_STALL_MS declaration found in ${SHIM}`).toBeDefined();
expect(Number(String(declared).replaceAll('_', ''))).toBe(SHIM_DRAIN_STALL_MS);
});

it('a CLOSED read end is released at once, not held for the bound (EPIPE reaches the callback)', () => {
// Pins the fast path measured alongside the hang: when the reader is gone
Expand Down
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 134 additions & 10 deletions packages/cli/test/run-dev-unbuilt-workspace.e2e.test.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@

import { describe, it, expect, beforeAll, afterAll } from 'vitest';
import { execFile, spawn } from 'node:child_process';
import { mkdtempSync, rmSync } from 'node:fs';
import { mkdtempSync, readFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join, resolve } from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
Expand All@@ -96,6 +96,12 @@ interface Run {
code: number;
stdout: string;
stderr: string;
/**
* Wall clock for the whole child, spawn to callback. Read by case 5, which
* sizes its own ceiling against a run of the SAME child on the SAME runner
* rather than against a constant measured somewhere else.
*/
elapsedMs: number;
}

/**
Expand All@@ -105,9 +111,11 @@ interface Run {
* started with — and the control legs' whole job is to be un-simulated.
*/
function runCli(args: string[], cwd: string, nodeOptions: string | undefined): Promise<Run> {
const started = Date.now();
return new Promise((resolvePromise) => {
execFile(TSX, [CLI, ...args], { cwd, maxBuffer: 32 * 1024 * 1024, env: childEnv({ NO_COLOR: '1', NODE_OPTIONS: nodeOptions }) }, (err, stdout, stderr) => {
resolvePromise({
elapsedMs: Date.now() - started,
// `err.code` is the real exit status; `null`/undefined means the child
// was signalled — a failure of a different kind, never reported as 0.
code: err ? (typeof (err as { code?: unknown }).code === 'number' ? (err as unknown as { code: number }).code : 1) : 0,
Expand DownExpand Up@@ -147,10 +155,79 @@ const PIPE_BUFFER_BYTES = 65_536;
const STALL_MS = 10_000;

/**
* Ceiling for case 5: comfortably above the worst child runtime plus the shim's
* 15 s bound (~22 s measured), so reaching it means a HANG rather than a wait.
* The shim's own no-progress bound, mirrored from `bin/run-dev.js`
* (`STDERR_DRAIN_STALL_MS`) and held equal to it by a case below rather than
* trusted. Case 5's ceiling no longer budgets it — that ceiling is a constant
* now — but two cases here are still sized against it and would quietly stop
* discriminating if it moved:
*
* • `STALL_MS` above must stay strictly BELOW it, or case 4's stalled reader
* outlasts the shim's own give-up and reds against a WORKING fix;
* • case 6 reads the closed-reader path as released in less than `STALL_MS`,
* which is evidence of a fast path only while `STALL_MS` is itself below
* the bound.
*/
const UNREAD_HARD_CAP_MS = 40_000;
const SHIM_DRAIN_STALL_MS = 15_000;

/** Where that constant is written — read by the parity case, never imported. */
const SHIM = resolve(HERE, '../bin/run-dev.js');

/**
* Case 5's ceiling — a CONSTANT, and deliberately this file's existing
* per-child budget rather than a number of its own.
*
* ⚠️ Two ceilings have been tried here and both failed the same way, so the
* history is written down instead of left to be rediscovered:
*
* • `UNREAD_HARD_CAP_MS = 40_000`, read as "comfortably above the worst child
* runtime plus the shim's 15 s bound (~22 s measured)". Merge-queue shards
* running the full suite six ways sharded went over it three times in a
* day, on three trees that cannot reach this file.
* • then a per-run derivation, `clamp(40_000, RUN_TIMEOUT_MS, 4 x a case-1
* calibration + 2 x the shim's bound)`, on the theory that a contended
* shard can calibrate itself. It was evicted from the queue by its own new
* assertion at `cap 61464 ms = clamp(40000, 180000, 4 x 7866 ms measured
* child runtime + 2 x 15000 ms shim bound)`: the child outlived a ceiling
* built from a sample taken minutes earlier on that same runner by more
* than 7.8x that sample, against a FACTOR of 4.
*
* Raising the factor would be the same move a third time. Both ceilings were
* sized comfortably above the worst thing on record when they were written, and
* both were beaten by a runner that got busier afterwards. Nothing measures the
* spread between a calibration and a later run on a shared, six-way-sharded
* queue runner, so no factor can be justified as ENOUGH — only as not beaten
* yet, which is what the constant it replaced could also say.
*
* ⭐ What removes the choice is the property this case actually pins. The
* failure it was written against is an UNBOUNDED wait: a drain wait with no
* bound armed at all, observed alive at 25 s, 30 s and 60 s and ending only
* when something else killed it. ANY finite ceiling catches that. Tightening a
* ceiling buys no detection at all — it buys false reds, and each one here
* costs a queue rebuild. So the ceiling wants to be the LARGEST value that
* keeps the failure legible, and it must not track load: a term tracking load
* is a prediction about contention drawn from a sample of the past, which is
* the one thing a shared runner will not honour.
*
* `RUN_TIMEOUT_MS` is that largest legible value, and it is not a new number:
*
* • past it this case stops reporting a SIGKILL and starts reporting the
* `beforeAll` timeout, which reds all six cases and names none of them. So
* it is where legibility ends, not a preference;
* • it is already this file's budget for ONE child of this suite, and cases
* 1-4 run the same child. A child here that legitimately needs more than
* 180 s has broken the whole file, not this case — one number to get
* wrong instead of two;
* • every load figure on record clears it by an order of magnitude: 23x the
* 7.9 s calibration, and 3.4x the worst legitimate lifetime yet measured
* (22.6 s of contended work against 8 competing copies of this child, plus
* both of the shim's 15 s bounds).
*
* The measurement is KEPT — as evidence in the failure message, never as an
* input to the threshold. That is the whole correction: case 1's wall clock
* tells a triage whether a red is a hang or a runner on fire, and it decides
* nothing.
*/
const UNREAD_HARD_CAP_MS = RUN_TIMEOUT_MS;

interface Lifetime {
code: number | null;
Expand All@@ -168,6 +245,7 @@ function runCliAgainstDeadReader(
cwd: string,
nodeOptions: string,
mode: 'never-read' | 'destroy-read-end',
capMs: number,
): Promise<Lifetime> {
return new Promise((resolvePromise) => {
const child = spawn(TSX, [CLI, ...args], {
Expand All@@ -183,7 +261,7 @@ function runCliAgainstDeadReader(
const started = Date.now();
// Ours, and it must be the ONLY thing that can end a hang — a child that
// reaches it is the failure this case exists to catch.
const cap = setTimeout(() => child.kill('SIGKILL'), UNREAD_HARD_CAP_MS);
const cap = setTimeout(() => child.kill('SIGKILL'), capMs);
child.once('exit', (code, signal) => {
clearTimeout(cap);
resolvePromise({ code, signal, elapsedMs: Date.now() - started });
Expand DownExpand Up@@ -222,8 +300,11 @@ beforeAll(async () => {
built = await runCli(REAL_COMMAND, dir, undefined);
genuinelyMissing = await runCli(['definitely-not-a-command'], dir, undefined);
stalled = await runCliWhileParentStalls(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`);
unread = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'never-read');
closedEnd = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'destroy-read-end');
// ⛔ Nothing measured above is consulted here. Case 1's wall clock is read by
// the failure message below, as evidence; the ceiling is a constant, so a
// slow sample can no longer size the instrument that judges the next run.
unread = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'never-read', UNREAD_HARD_CAP_MS);
closedEnd = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'destroy-read-end', UNREAD_HARD_CAP_MS);
}, RUN_TIMEOUT_MS * 6);

afterAll(() => {
Expand DownExpand Up@@ -314,9 +395,28 @@ describe('the mirror direction: a reader that is never coming back', () => {
it('gives up and exits instead of waiting forever', () => {
// A child still alive at the cap was SIGKILLed: signal set, code null.
// That is the hang, and it is the whole point of this case.
expect(unread.signal).toBeNull();
expect(unread.code).toBe(2);
expect(unread.elapsedMs).toBeLessThan(UNREAD_HARD_CAP_MS);
//
// ⚠️ Both surviving assertions are about the PRODUCT: the child ends on
// its OWN, and it ends with the status any other reader would have got.
//
// ⛔ The third assertion this case used to carry — `elapsedMs` below the
// ceiling — is deliberately gone. Against a constant cap it asserts nothing
// the first line does not: the harness kills at exactly that cap, so a
// child that was not killed ran less than it. What it added was a race, at
// the one instant where a child exiting on its own and the timer firing
// are simultaneous, and it was the only reading here that a slower box
// could move on its own. Detection unchanged, one fewer way to red.
//
// The numbers move into the message, because `expected 'SIGKILL' to be
// null` alone does not tell a merge-queue triage which of two readings it
// has. A child killed at 180 s whose calibration was 8 s is a hang; one
// whose calibration was also minutes indicts the runner, not this code.
const evidence =
`cap ${UNREAD_HARD_CAP_MS} ms (RUN_TIMEOUT_MS, constant and load-independent by design); ` +
`this child ran ${unread.elapsedMs} ms; case 1 measured the same child at ` +
`${unbuilt.elapsedMs} ms on this runner minutes earlier`;
expect(unread.signal, `the harness SIGKILLed the child — it was still alive at the ceiling. ${evidence}`).toBeNull();
expect(unread.code, `the child did not exit 2 on its own. ${evidence}`).toBe(2);
});

// ⛔ There is deliberately NO assertion here that the child WAITED for the
Expand All@@ -332,6 +432,30 @@ describe('the mirror direction: a reader that is never coming back', () => {
// process ENDS. That the bound itself runs and trips is shown out of band,
// by tracing a run whose reader blocks its loop for the whole run — see the
// PR for the `BOUND TRIPPED` trace.
//
// ⚠️ The same nondeterminism means an ABLATION of the bound can come back
// GREEN, and a single green one here is a ZERO READING rather than evidence
// this case has stopped discriminating. Measured on one box minutes apart,
// same tree: disabling the no-progress branch red this case at 180072 ms
// once, and passed it in 31.9 s the run before — that run's backlog fit in
// what the kernel and node happened to absorb, so the write callback
// resolved on its own and the branch was never reached. Re-run it, or drive
// the child OUT OF BAND (`spawn`, `stderr.pause()`, never read) where the
// pending bytes can actually be counted: 135408 bytes still held at a clean
// exit 2 in 19271 ms, against 145638 held by a child still alive at 90 s
// with the branch disabled.

it("keeps its mirror of the shim's bound equal to the shim's own", () => {
// The derivation above is only as good as `SHIM_DRAIN_STALL_MS` still being
// what `bin/run-dev.js` waits. There is no import to take it from — that
// file runs the CLI at module top — so it is mirrored, and a mirror with
// nothing holding it is how a ceiling ends up sized around a bound that
// moved. Same discipline as `INVOCATION_PREFIX` vs `CLI_NAME`: kept in
// sync by a case, not by an import.
const declared = /const STDERR_DRAIN_STALL_MS = ([\d_]+);/.exec(readFileSync(SHIM, 'utf8'))?.[1];
expect(declared, `no STDERR_DRAIN_STALL_MS declaration found in ${SHIM}`).toBeDefined();
expect(Number(String(declared).replaceAll('_', ''))).toBe(SHIM_DRAIN_STALL_MS);
});

it('a CLOSED read end is released at once, not held for the bound (EPIPE reaches the callback)', () => {
// Pins the fast path measured alongside the hang: when the reader is gone
Expand Down
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 134 additions & 10 deletions packages/cli/test/run-dev-unbuilt-workspace.e2e.test.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@

import { describe, it, expect, beforeAll, afterAll } from 'vitest';
import { execFile, spawn } from 'node:child_process';
import { mkdtempSync, rmSync } from 'node:fs';
import { mkdtempSync, readFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join, resolve } from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
Expand All@@ -96,6 +96,12 @@ interface Run {
code: number;
stdout: string;
stderr: string;
/**
* Wall clock for the whole child, spawn to callback. Read by case 5, which
* sizes its own ceiling against a run of the SAME child on the SAME runner
* rather than against a constant measured somewhere else.
*/
elapsedMs: number;
}

/**
Expand All@@ -105,9 +111,11 @@ interface Run {
* started with — and the control legs' whole job is to be un-simulated.
*/
function runCli(args: string[], cwd: string, nodeOptions: string | undefined): Promise<Run> {
const started = Date.now();
return new Promise((resolvePromise) => {
execFile(TSX, [CLI, ...args], { cwd, maxBuffer: 32 * 1024 * 1024, env: childEnv({ NO_COLOR: '1', NODE_OPTIONS: nodeOptions }) }, (err, stdout, stderr) => {
resolvePromise({
elapsedMs: Date.now() - started,
// `err.code` is the real exit status; `null`/undefined means the child
// was signalled — a failure of a different kind, never reported as 0.
code: err ? (typeof (err as { code?: unknown }).code === 'number' ? (err as unknown as { code: number }).code : 1) : 0,
Expand DownExpand Up@@ -147,10 +155,79 @@ const PIPE_BUFFER_BYTES = 65_536;
const STALL_MS = 10_000;

/**
* Ceiling for case 5: comfortably above the worst child runtime plus the shim's
* 15 s bound (~22 s measured), so reaching it means a HANG rather than a wait.
* The shim's own no-progress bound, mirrored from `bin/run-dev.js`
* (`STDERR_DRAIN_STALL_MS`) and held equal to it by a case below rather than
* trusted. Case 5's ceiling no longer budgets it — that ceiling is a constant
* now — but two cases here are still sized against it and would quietly stop
* discriminating if it moved:
*
* • `STALL_MS` above must stay strictly BELOW it, or case 4's stalled reader
* outlasts the shim's own give-up and reds against a WORKING fix;
* • case 6 reads the closed-reader path as released in less than `STALL_MS`,
* which is evidence of a fast path only while `STALL_MS` is itself below
* the bound.
*/
const UNREAD_HARD_CAP_MS = 40_000;
const SHIM_DRAIN_STALL_MS = 15_000;

/** Where that constant is written — read by the parity case, never imported. */
const SHIM = resolve(HERE, '../bin/run-dev.js');

/**
* Case 5's ceiling — a CONSTANT, and deliberately this file's existing
* per-child budget rather than a number of its own.
*
* ⚠️ Two ceilings have been tried here and both failed the same way, so the
* history is written down instead of left to be rediscovered:
*
* • `UNREAD_HARD_CAP_MS = 40_000`, read as "comfortably above the worst child
* runtime plus the shim's 15 s bound (~22 s measured)". Merge-queue shards
* running the full suite six ways sharded went over it three times in a
* day, on three trees that cannot reach this file.
* • then a per-run derivation, `clamp(40_000, RUN_TIMEOUT_MS, 4 x a case-1
* calibration + 2 x the shim's bound)`, on the theory that a contended
* shard can calibrate itself. It was evicted from the queue by its own new
* assertion at `cap 61464 ms = clamp(40000, 180000, 4 x 7866 ms measured
* child runtime + 2 x 15000 ms shim bound)`: the child outlived a ceiling
* built from a sample taken minutes earlier on that same runner by more
* than 7.8x that sample, against a FACTOR of 4.
*
* Raising the factor would be the same move a third time. Both ceilings were
* sized comfortably above the worst thing on record when they were written, and
* both were beaten by a runner that got busier afterwards. Nothing measures the
* spread between a calibration and a later run on a shared, six-way-sharded
* queue runner, so no factor can be justified as ENOUGH — only as not beaten
* yet, which is what the constant it replaced could also say.
*
* ⭐ What removes the choice is the property this case actually pins. The
* failure it was written against is an UNBOUNDED wait: a drain wait with no
* bound armed at all, observed alive at 25 s, 30 s and 60 s and ending only
* when something else killed it. ANY finite ceiling catches that. Tightening a
* ceiling buys no detection at all — it buys false reds, and each one here
* costs a queue rebuild. So the ceiling wants to be the LARGEST value that
* keeps the failure legible, and it must not track load: a term tracking load
* is a prediction about contention drawn from a sample of the past, which is
* the one thing a shared runner will not honour.
*
* `RUN_TIMEOUT_MS` is that largest legible value, and it is not a new number:
*
* • past it this case stops reporting a SIGKILL and starts reporting the
* `beforeAll` timeout, which reds all six cases and names none of them. So
* it is where legibility ends, not a preference;
* • it is already this file's budget for ONE child of this suite, and cases
* 1-4 run the same child. A child here that legitimately needs more than
* 180 s has broken the whole file, not this case — one number to get
* wrong instead of two;
* • every load figure on record clears it by an order of magnitude: 23x the
* 7.9 s calibration, and 3.4x the worst legitimate lifetime yet measured
* (22.6 s of contended work against 8 competing copies of this child, plus
* both of the shim's 15 s bounds).
*
* The measurement is KEPT — as evidence in the failure message, never as an
* input to the threshold. That is the whole correction: case 1's wall clock
* tells a triage whether a red is a hang or a runner on fire, and it decides
* nothing.
*/
const UNREAD_HARD_CAP_MS = RUN_TIMEOUT_MS;

interface Lifetime {
code: number | null;
Expand All@@ -168,6 +245,7 @@ function runCliAgainstDeadReader(
cwd: string,
nodeOptions: string,
mode: 'never-read' | 'destroy-read-end',
capMs: number,
): Promise<Lifetime> {
return new Promise((resolvePromise) => {
const child = spawn(TSX, [CLI, ...args], {
Expand All@@ -183,7 +261,7 @@ function runCliAgainstDeadReader(
const started = Date.now();
// Ours, and it must be the ONLY thing that can end a hang — a child that
// reaches it is the failure this case exists to catch.
const cap = setTimeout(() => child.kill('SIGKILL'), UNREAD_HARD_CAP_MS);
const cap = setTimeout(() => child.kill('SIGKILL'), capMs);
child.once('exit', (code, signal) => {
clearTimeout(cap);
resolvePromise({ code, signal, elapsedMs: Date.now() - started });
Expand DownExpand Up@@ -222,8 +300,11 @@ beforeAll(async () => {
built = await runCli(REAL_COMMAND, dir, undefined);
genuinelyMissing = await runCli(['definitely-not-a-command'], dir, undefined);
stalled = await runCliWhileParentStalls(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`);
unread = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'never-read');
closedEnd = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'destroy-read-end');
// ⛔ Nothing measured above is consulted here. Case 1's wall clock is read by
// the failure message below, as evidence; the ceiling is a constant, so a
// slow sample can no longer size the instrument that judges the next run.
unread = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'never-read', UNREAD_HARD_CAP_MS);
closedEnd = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'destroy-read-end', UNREAD_HARD_CAP_MS);
}, RUN_TIMEOUT_MS * 6);

afterAll(() => {
Expand DownExpand Up@@ -314,9 +395,28 @@ describe('the mirror direction: a reader that is never coming back', () => {
it('gives up and exits instead of waiting forever', () => {
// A child still alive at the cap was SIGKILLed: signal set, code null.
// That is the hang, and it is the whole point of this case.
expect(unread.signal).toBeNull();
expect(unread.code).toBe(2);
expect(unread.elapsedMs).toBeLessThan(UNREAD_HARD_CAP_MS);
//
// ⚠️ Both surviving assertions are about the PRODUCT: the child ends on
// its OWN, and it ends with the status any other reader would have got.
//
// ⛔ The third assertion this case used to carry — `elapsedMs` below the
// ceiling — is deliberately gone. Against a constant cap it asserts nothing
// the first line does not: the harness kills at exactly that cap, so a
// child that was not killed ran less than it. What it added was a race, at
// the one instant where a child exiting on its own and the timer firing
// are simultaneous, and it was the only reading here that a slower box
// could move on its own. Detection unchanged, one fewer way to red.
//
// The numbers move into the message, because `expected 'SIGKILL' to be
// null` alone does not tell a merge-queue triage which of two readings it
// has. A child killed at 180 s whose calibration was 8 s is a hang; one
// whose calibration was also minutes indicts the runner, not this code.
const evidence =
`cap ${UNREAD_HARD_CAP_MS} ms (RUN_TIMEOUT_MS, constant and load-independent by design); ` +
`this child ran ${unread.elapsedMs} ms; case 1 measured the same child at ` +
`${unbuilt.elapsedMs} ms on this runner minutes earlier`;
expect(unread.signal, `the harness SIGKILLed the child — it was still alive at the ceiling. ${evidence}`).toBeNull();
expect(unread.code, `the child did not exit 2 on its own. ${evidence}`).toBe(2);
});

// ⛔ There is deliberately NO assertion here that the child WAITED for the
Expand All@@ -332,6 +432,30 @@ describe('the mirror direction: a reader that is never coming back', () => {
// process ENDS. That the bound itself runs and trips is shown out of band,
// by tracing a run whose reader blocks its loop for the whole run — see the
// PR for the `BOUND TRIPPED` trace.
//
// ⚠️ The same nondeterminism means an ABLATION of the bound can come back
// GREEN, and a single green one here is a ZERO READING rather than evidence
// this case has stopped discriminating. Measured on one box minutes apart,
// same tree: disabling the no-progress branch red this case at 180072 ms
// once, and passed it in 31.9 s the run before — that run's backlog fit in
// what the kernel and node happened to absorb, so the write callback
// resolved on its own and the branch was never reached. Re-run it, or drive
// the child OUT OF BAND (`spawn`, `stderr.pause()`, never read) where the
// pending bytes can actually be counted: 135408 bytes still held at a clean
// exit 2 in 19271 ms, against 145638 held by a child still alive at 90 s
// with the branch disabled.

it("keeps its mirror of the shim's bound equal to the shim's own", () => {
// The derivation above is only as good as `SHIM_DRAIN_STALL_MS` still being
// what `bin/run-dev.js` waits. There is no import to take it from — that
// file runs the CLI at module top — so it is mirrored, and a mirror with
// nothing holding it is how a ceiling ends up sized around a bound that
// moved. Same discipline as `INVOCATION_PREFIX` vs `CLI_NAME`: kept in
// sync by a case, not by an import.
const declared = /const STDERR_DRAIN_STALL_MS = ([\d_]+);/.exec(readFileSync(SHIM, 'utf8'))?.[1];
expect(declared, `no STDERR_DRAIN_STALL_MS declaration found in ${SHIM}`).toBeDefined();
expect(Number(String(declared).replaceAll('_', ''))).toBe(SHIM_DRAIN_STALL_MS);
});

it('a CLOSED read end is released at once, not held for the bound (EPIPE reaches the callback)', () => {
// Pins the fast path measured alongside the hang: when the reader is gone
Expand Down
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 134 additions & 10 deletions packages/cli/test/run-dev-unbuilt-workspace.e2e.test.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@

import { describe, it, expect, beforeAll, afterAll } from 'vitest';
import { execFile, spawn } from 'node:child_process';
import { mkdtempSync, rmSync } from 'node:fs';
import { mkdtempSync, readFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join, resolve } from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
Expand All@@ -96,6 +96,12 @@ interface Run {
code: number;
stdout: string;
stderr: string;
/**
* Wall clock for the whole child, spawn to callback. Read by case 5, which
* sizes its own ceiling against a run of the SAME child on the SAME runner
* rather than against a constant measured somewhere else.
*/
elapsedMs: number;
}

/**
Expand All@@ -105,9 +111,11 @@ interface Run {
* started with — and the control legs' whole job is to be un-simulated.
*/
function runCli(args: string[], cwd: string, nodeOptions: string | undefined): Promise<Run> {
const started = Date.now();
return new Promise((resolvePromise) => {
execFile(TSX, [CLI, ...args], { cwd, maxBuffer: 32 * 1024 * 1024, env: childEnv({ NO_COLOR: '1', NODE_OPTIONS: nodeOptions }) }, (err, stdout, stderr) => {
resolvePromise({
elapsedMs: Date.now() - started,
// `err.code` is the real exit status; `null`/undefined means the child
// was signalled — a failure of a different kind, never reported as 0.
code: err ? (typeof (err as { code?: unknown }).code === 'number' ? (err as unknown as { code: number }).code : 1) : 0,
Expand DownExpand Up@@ -147,10 +155,79 @@ const PIPE_BUFFER_BYTES = 65_536;
const STALL_MS = 10_000;

/**
* Ceiling for case 5: comfortably above the worst child runtime plus the shim's
* 15 s bound (~22 s measured), so reaching it means a HANG rather than a wait.
* The shim's own no-progress bound, mirrored from `bin/run-dev.js`
* (`STDERR_DRAIN_STALL_MS`) and held equal to it by a case below rather than
* trusted. Case 5's ceiling no longer budgets it — that ceiling is a constant
* now — but two cases here are still sized against it and would quietly stop
* discriminating if it moved:
*
* • `STALL_MS` above must stay strictly BELOW it, or case 4's stalled reader
* outlasts the shim's own give-up and reds against a WORKING fix;
* • case 6 reads the closed-reader path as released in less than `STALL_MS`,
* which is evidence of a fast path only while `STALL_MS` is itself below
* the bound.
*/
const UNREAD_HARD_CAP_MS = 40_000;
const SHIM_DRAIN_STALL_MS = 15_000;

/** Where that constant is written — read by the parity case, never imported. */
const SHIM = resolve(HERE, '../bin/run-dev.js');

/**
* Case 5's ceiling — a CONSTANT, and deliberately this file's existing
* per-child budget rather than a number of its own.
*
* ⚠️ Two ceilings have been tried here and both failed the same way, so the
* history is written down instead of left to be rediscovered:
*
* • `UNREAD_HARD_CAP_MS = 40_000`, read as "comfortably above the worst child
* runtime plus the shim's 15 s bound (~22 s measured)". Merge-queue shards
* running the full suite six ways sharded went over it three times in a
* day, on three trees that cannot reach this file.
* • then a per-run derivation, `clamp(40_000, RUN_TIMEOUT_MS, 4 x a case-1
* calibration + 2 x the shim's bound)`, on the theory that a contended
* shard can calibrate itself. It was evicted from the queue by its own new
* assertion at `cap 61464 ms = clamp(40000, 180000, 4 x 7866 ms measured
* child runtime + 2 x 15000 ms shim bound)`: the child outlived a ceiling
* built from a sample taken minutes earlier on that same runner by more
* than 7.8x that sample, against a FACTOR of 4.
*
* Raising the factor would be the same move a third time. Both ceilings were
* sized comfortably above the worst thing on record when they were written, and
* both were beaten by a runner that got busier afterwards. Nothing measures the
* spread between a calibration and a later run on a shared, six-way-sharded
* queue runner, so no factor can be justified as ENOUGH — only as not beaten
* yet, which is what the constant it replaced could also say.
*
* ⭐ What removes the choice is the property this case actually pins. The
* failure it was written against is an UNBOUNDED wait: a drain wait with no
* bound armed at all, observed alive at 25 s, 30 s and 60 s and ending only
* when something else killed it. ANY finite ceiling catches that. Tightening a
* ceiling buys no detection at all — it buys false reds, and each one here
* costs a queue rebuild. So the ceiling wants to be the LARGEST value that
* keeps the failure legible, and it must not track load: a term tracking load
* is a prediction about contention drawn from a sample of the past, which is
* the one thing a shared runner will not honour.
*
* `RUN_TIMEOUT_MS` is that largest legible value, and it is not a new number:
*
* • past it this case stops reporting a SIGKILL and starts reporting the
* `beforeAll` timeout, which reds all six cases and names none of them. So
* it is where legibility ends, not a preference;
* • it is already this file's budget for ONE child of this suite, and cases
* 1-4 run the same child. A child here that legitimately needs more than
* 180 s has broken the whole file, not this case — one number to get
* wrong instead of two;
* • every load figure on record clears it by an order of magnitude: 23x the
* 7.9 s calibration, and 3.4x the worst legitimate lifetime yet measured
* (22.6 s of contended work against 8 competing copies of this child, plus
* both of the shim's 15 s bounds).
*
* The measurement is KEPT — as evidence in the failure message, never as an
* input to the threshold. That is the whole correction: case 1's wall clock
* tells a triage whether a red is a hang or a runner on fire, and it decides
* nothing.
*/
const UNREAD_HARD_CAP_MS = RUN_TIMEOUT_MS;

interface Lifetime {
code: number | null;
Expand All@@ -168,6 +245,7 @@ function runCliAgainstDeadReader(
cwd: string,
nodeOptions: string,
mode: 'never-read' | 'destroy-read-end',
capMs: number,
): Promise<Lifetime> {
return new Promise((resolvePromise) => {
const child = spawn(TSX, [CLI, ...args], {
Expand All@@ -183,7 +261,7 @@ function runCliAgainstDeadReader(
const started = Date.now();
// Ours, and it must be the ONLY thing that can end a hang — a child that
// reaches it is the failure this case exists to catch.
const cap = setTimeout(() => child.kill('SIGKILL'), UNREAD_HARD_CAP_MS);
const cap = setTimeout(() => child.kill('SIGKILL'), capMs);
child.once('exit', (code, signal) => {
clearTimeout(cap);
resolvePromise({ code, signal, elapsedMs: Date.now() - started });
Expand DownExpand Up@@ -222,8 +300,11 @@ beforeAll(async () => {
built = await runCli(REAL_COMMAND, dir, undefined);
genuinelyMissing = await runCli(['definitely-not-a-command'], dir, undefined);
stalled = await runCliWhileParentStalls(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`);
unread = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'never-read');
closedEnd = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'destroy-read-end');
// ⛔ Nothing measured above is consulted here. Case 1's wall clock is read by
// the failure message below, as evidence; the ceiling is a constant, so a
// slow sample can no longer size the instrument that judges the next run.
unread = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'never-read', UNREAD_HARD_CAP_MS);
closedEnd = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'destroy-read-end', UNREAD_HARD_CAP_MS);
}, RUN_TIMEOUT_MS * 6);

afterAll(() => {
Expand DownExpand Up@@ -314,9 +395,28 @@ describe('the mirror direction: a reader that is never coming back', () => {
it('gives up and exits instead of waiting forever', () => {
// A child still alive at the cap was SIGKILLed: signal set, code null.
// That is the hang, and it is the whole point of this case.
expect(unread.signal).toBeNull();
expect(unread.code).toBe(2);
expect(unread.elapsedMs).toBeLessThan(UNREAD_HARD_CAP_MS);
//
// ⚠️ Both surviving assertions are about the PRODUCT: the child ends on
// its OWN, and it ends with the status any other reader would have got.
//
// ⛔ The third assertion this case used to carry — `elapsedMs` below the
// ceiling — is deliberately gone. Against a constant cap it asserts nothing
// the first line does not: the harness kills at exactly that cap, so a
// child that was not killed ran less than it. What it added was a race, at
// the one instant where a child exiting on its own and the timer firing
// are simultaneous, and it was the only reading here that a slower box
// could move on its own. Detection unchanged, one fewer way to red.
//
// The numbers move into the message, because `expected 'SIGKILL' to be
// null` alone does not tell a merge-queue triage which of two readings it
// has. A child killed at 180 s whose calibration was 8 s is a hang; one
// whose calibration was also minutes indicts the runner, not this code.
const evidence =
`cap ${UNREAD_HARD_CAP_MS} ms (RUN_TIMEOUT_MS, constant and load-independent by design); ` +
`this child ran ${unread.elapsedMs} ms; case 1 measured the same child at ` +
`${unbuilt.elapsedMs} ms on this runner minutes earlier`;
expect(unread.signal, `the harness SIGKILLed the child — it was still alive at the ceiling. ${evidence}`).toBeNull();
expect(unread.code, `the child did not exit 2 on its own. ${evidence}`).toBe(2);
});

// ⛔ There is deliberately NO assertion here that the child WAITED for the
Expand All@@ -332,6 +432,30 @@ describe('the mirror direction: a reader that is never coming back', () => {
// process ENDS. That the bound itself runs and trips is shown out of band,
// by tracing a run whose reader blocks its loop for the whole run — see the
// PR for the `BOUND TRIPPED` trace.
//
// ⚠️ The same nondeterminism means an ABLATION of the bound can come back
// GREEN, and a single green one here is a ZERO READING rather than evidence
// this case has stopped discriminating. Measured on one box minutes apart,
// same tree: disabling the no-progress branch red this case at 180072 ms
// once, and passed it in 31.9 s the run before — that run's backlog fit in
// what the kernel and node happened to absorb, so the write callback
// resolved on its own and the branch was never reached. Re-run it, or drive
// the child OUT OF BAND (`spawn`, `stderr.pause()`, never read) where the
// pending bytes can actually be counted: 135408 bytes still held at a clean
// exit 2 in 19271 ms, against 145638 held by a child still alive at 90 s
// with the branch disabled.

it("keeps its mirror of the shim's bound equal to the shim's own", () => {
// The derivation above is only as good as `SHIM_DRAIN_STALL_MS` still being
// what `bin/run-dev.js` waits. There is no import to take it from — that
// file runs the CLI at module top — so it is mirrored, and a mirror with
// nothing holding it is how a ceiling ends up sized around a bound that
// moved. Same discipline as `INVOCATION_PREFIX` vs `CLI_NAME`: kept in
// sync by a case, not by an import.
const declared = /const STDERR_DRAIN_STALL_MS = ([\d_]+);/.exec(readFileSync(SHIM, 'utf8'))?.[1];
expect(declared, `no STDERR_DRAIN_STALL_MS declaration found in ${SHIM}`).toBeDefined();
expect(Number(String(declared).replaceAll('_', ''))).toBe(SHIM_DRAIN_STALL_MS);
});

it('a CLOSED read end is released at once, not held for the bound (EPIPE reaches the callback)', () => {
// Pins the fast path measured alongside the hang: when the reader is gone
Expand Down
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 134 additions & 10 deletions packages/cli/test/run-dev-unbuilt-workspace.e2e.test.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -70,7 +70,7 @@

import { describe, it, expect, beforeAll, afterAll } from 'vitest';
import { execFile, spawn } from 'node:child_process';
import { mkdtempSync, rmSync } from 'node:fs';
import { mkdtempSync, readFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join, resolve } from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
Expand All@@ -96,6 +96,12 @@ interface Run {
code: number;
stdout: string;
stderr: string;
/**
* Wall clock for the whole child, spawn to callback. Read by case 5, which
* sizes its own ceiling against a run of the SAME child on the SAME runner
* rather than against a constant measured somewhere else.
*/
elapsedMs: number;
}

/**
Expand All@@ -105,9 +111,11 @@ interface Run {
* started with — and the control legs' whole job is to be un-simulated.
*/
function runCli(args: string[], cwd: string, nodeOptions: string | undefined): Promise<Run> {
const started = Date.now();
return new Promise((resolvePromise) => {
execFile(TSX, [CLI, ...args], { cwd, maxBuffer: 32 * 1024 * 1024, env: childEnv({ NO_COLOR: '1', NODE_OPTIONS: nodeOptions }) }, (err, stdout, stderr) => {
resolvePromise({
elapsedMs: Date.now() - started,
// `err.code` is the real exit status; `null`/undefined means the child
// was signalled — a failure of a different kind, never reported as 0.
code: err ? (typeof (err as { code?: unknown }).code === 'number' ? (err as unknown as { code: number }).code : 1) : 0,
Expand DownExpand Up@@ -147,10 +155,79 @@ const PIPE_BUFFER_BYTES = 65_536;
const STALL_MS = 10_000;

/**
* Ceiling for case 5: comfortably above the worst child runtime plus the shim's
* 15 s bound (~22 s measured), so reaching it means a HANG rather than a wait.
* The shim's own no-progress bound, mirrored from `bin/run-dev.js`
* (`STDERR_DRAIN_STALL_MS`) and held equal to it by a case below rather than
* trusted. Case 5's ceiling no longer budgets it — that ceiling is a constant
* now — but two cases here are still sized against it and would quietly stop
* discriminating if it moved:
*
* • `STALL_MS` above must stay strictly BELOW it, or case 4's stalled reader
* outlasts the shim's own give-up and reds against a WORKING fix;
* • case 6 reads the closed-reader path as released in less than `STALL_MS`,
* which is evidence of a fast path only while `STALL_MS` is itself below
* the bound.
*/
const UNREAD_HARD_CAP_MS = 40_000;
const SHIM_DRAIN_STALL_MS = 15_000;

/** Where that constant is written — read by the parity case, never imported. */
const SHIM = resolve(HERE, '../bin/run-dev.js');

/**
* Case 5's ceiling — a CONSTANT, and deliberately this file's existing
* per-child budget rather than a number of its own.
*
* ⚠️ Two ceilings have been tried here and both failed the same way, so the
* history is written down instead of left to be rediscovered:
*
* • `UNREAD_HARD_CAP_MS = 40_000`, read as "comfortably above the worst child
* runtime plus the shim's 15 s bound (~22 s measured)". Merge-queue shards
* running the full suite six ways sharded went over it three times in a
* day, on three trees that cannot reach this file.
* • then a per-run derivation, `clamp(40_000, RUN_TIMEOUT_MS, 4 x a case-1
* calibration + 2 x the shim's bound)`, on the theory that a contended
* shard can calibrate itself. It was evicted from the queue by its own new
* assertion at `cap 61464 ms = clamp(40000, 180000, 4 x 7866 ms measured
* child runtime + 2 x 15000 ms shim bound)`: the child outlived a ceiling
* built from a sample taken minutes earlier on that same runner by more
* than 7.8x that sample, against a FACTOR of 4.
*
* Raising the factor would be the same move a third time. Both ceilings were
* sized comfortably above the worst thing on record when they were written, and
* both were beaten by a runner that got busier afterwards. Nothing measures the
* spread between a calibration and a later run on a shared, six-way-sharded
* queue runner, so no factor can be justified as ENOUGH — only as not beaten
* yet, which is what the constant it replaced could also say.
*
* ⭐ What removes the choice is the property this case actually pins. The
* failure it was written against is an UNBOUNDED wait: a drain wait with no
* bound armed at all, observed alive at 25 s, 30 s and 60 s and ending only
* when something else killed it. ANY finite ceiling catches that. Tightening a
* ceiling buys no detection at all — it buys false reds, and each one here
* costs a queue rebuild. So the ceiling wants to be the LARGEST value that
* keeps the failure legible, and it must not track load: a term tracking load
* is a prediction about contention drawn from a sample of the past, which is
* the one thing a shared runner will not honour.
*
* `RUN_TIMEOUT_MS` is that largest legible value, and it is not a new number:
*
* • past it this case stops reporting a SIGKILL and starts reporting the
* `beforeAll` timeout, which reds all six cases and names none of them. So
* it is where legibility ends, not a preference;
* • it is already this file's budget for ONE child of this suite, and cases
* 1-4 run the same child. A child here that legitimately needs more than
* 180 s has broken the whole file, not this case — one number to get
* wrong instead of two;
* • every load figure on record clears it by an order of magnitude: 23x the
* 7.9 s calibration, and 3.4x the worst legitimate lifetime yet measured
* (22.6 s of contended work against 8 competing copies of this child, plus
* both of the shim's 15 s bounds).
*
* The measurement is KEPT — as evidence in the failure message, never as an
* input to the threshold. That is the whole correction: case 1's wall clock
* tells a triage whether a red is a hang or a runner on fire, and it decides
* nothing.
*/
const UNREAD_HARD_CAP_MS = RUN_TIMEOUT_MS;

interface Lifetime {
code: number | null;
Expand All@@ -168,6 +245,7 @@ function runCliAgainstDeadReader(
cwd: string,
nodeOptions: string,
mode: 'never-read' | 'destroy-read-end',
capMs: number,
): Promise<Lifetime> {
return new Promise((resolvePromise) => {
const child = spawn(TSX, [CLI, ...args], {
Expand All@@ -183,7 +261,7 @@ function runCliAgainstDeadReader(
const started = Date.now();
// Ours, and it must be the ONLY thing that can end a hang — a child that
// reaches it is the failure this case exists to catch.
const cap = setTimeout(() => child.kill('SIGKILL'), UNREAD_HARD_CAP_MS);
const cap = setTimeout(() => child.kill('SIGKILL'), capMs);
child.once('exit', (code, signal) => {
clearTimeout(cap);
resolvePromise({ code, signal, elapsedMs: Date.now() - started });
Expand DownExpand Up@@ -222,8 +300,11 @@ beforeAll(async () => {
built = await runCli(REAL_COMMAND, dir, undefined);
genuinelyMissing = await runCli(['definitely-not-a-command'], dir, undefined);
stalled = await runCliWhileParentStalls(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`);
unread = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'never-read');
closedEnd = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'destroy-read-end');
// ⛔ Nothing measured above is consulted here. Case 1's wall clock is read by
// the failure message below, as evidence; the ceiling is a constant, so a
// slow sample can no longer size the instrument that judges the next run.
unread = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'never-read', UNREAD_HARD_CAP_MS);
closedEnd = await runCliAgainstDeadReader(REAL_COMMAND, dir, `--import ${UNBUILT_HOOK}`, 'destroy-read-end', UNREAD_HARD_CAP_MS);
}, RUN_TIMEOUT_MS * 6);

afterAll(() => {
Expand DownExpand Up@@ -314,9 +395,28 @@ describe('the mirror direction: a reader that is never coming back', () => {
it('gives up and exits instead of waiting forever', () => {
// A child still alive at the cap was SIGKILLed: signal set, code null.
// That is the hang, and it is the whole point of this case.
expect(unread.signal).toBeNull();
expect(unread.code).toBe(2);
expect(unread.elapsedMs).toBeLessThan(UNREAD_HARD_CAP_MS);
//
// ⚠️ Both surviving assertions are about the PRODUCT: the child ends on
// its OWN, and it ends with the status any other reader would have got.
//
// ⛔ The third assertion this case used to carry — `elapsedMs` below the
// ceiling — is deliberately gone. Against a constant cap it asserts nothing
// the first line does not: the harness kills at exactly that cap, so a
// child that was not killed ran less than it. What it added was a race, at
// the one instant where a child exiting on its own and the timer firing
// are simultaneous, and it was the only reading here that a slower box
// could move on its own. Detection unchanged, one fewer way to red.
//
// The numbers move into the message, because `expected 'SIGKILL' to be
// null` alone does not tell a merge-queue triage which of two readings it
// has. A child killed at 180 s whose calibration was 8 s is a hang; one
// whose calibration was also minutes indicts the runner, not this code.
const evidence =
`cap ${UNREAD_HARD_CAP_MS} ms (RUN_TIMEOUT_MS, constant and load-independent by design); ` +
`this child ran ${unread.elapsedMs} ms; case 1 measured the same child at ` +
`${unbuilt.elapsedMs} ms on this runner minutes earlier`;
expect(unread.signal, `the harness SIGKILLed the child — it was still alive at the ceiling. ${evidence}`).toBeNull();
expect(unread.code, `the child did not exit 2 on its own. ${evidence}`).toBe(2);
});

// ⛔ There is deliberately NO assertion here that the child WAITED for the
Expand All@@ -332,6 +432,30 @@ describe('the mirror direction: a reader that is never coming back', () => {
// process ENDS. That the bound itself runs and trips is shown out of band,
// by tracing a run whose reader blocks its loop for the whole run — see the
// PR for the `BOUND TRIPPED` trace.
//
// ⚠️ The same nondeterminism means an ABLATION of the bound can come back
// GREEN, and a single green one here is a ZERO READING rather than evidence
// this case has stopped discriminating. Measured on one box minutes apart,
// same tree: disabling the no-progress branch red this case at 180072 ms
// once, and passed it in 31.9 s the run before — that run's backlog fit in
// what the kernel and node happened to absorb, so the write callback
// resolved on its own and the branch was never reached. Re-run it, or drive
// the child OUT OF BAND (`spawn`, `stderr.pause()`, never read) where the
// pending bytes can actually be counted: 135408 bytes still held at a clean
// exit 2 in 19271 ms, against 145638 held by a child still alive at 90 s
// with the branch disabled.

it("keeps its mirror of the shim's bound equal to the shim's own", () => {
// The derivation above is only as good as `SHIM_DRAIN_STALL_MS` still being
// what `bin/run-dev.js` waits. There is no import to take it from — that
// file runs the CLI at module top — so it is mirrored, and a mirror with
// nothing holding it is how a ceiling ends up sized around a bound that
// moved. Same discipline as `INVOCATION_PREFIX` vs `CLI_NAME`: kept in
// sync by a case, not by an import.
const declared = /const STDERR_DRAIN_STALL_MS = ([\d_]+);/.exec(readFileSync(SHIM, 'utf8'))?.[1];
expect(declared, `no STDERR_DRAIN_STALL_MS declaration found in ${SHIM}`).toBeDefined();
expect(Number(String(declared).replaceAll('_', ''))).toBe(SHIM_DRAIN_STALL_MS);
});

it('a CLOSED read end is released at once, not held for the bound (EPIPE reaches the callback)', () => {
// Pins the fast path measured alongside the hang: when the reader is gone
Expand Down
Loading