diff --git a/.github/workflows/shell-escape-residue.yml b/.github/workflows/shell-escape-residue.yml new file mode 100644 index 0000000000..d14f47e0e7 --- /dev/null +++ b/.github/workflows/shell-escape-residue.yml @@ -0,0 +1,84 @@ +name: Shell Escape Residue + +# Why this is its own workflow rather than a step in `ci.yml` or `lint.yml`: both +# of those decide inside the job whether the change "needs a full run", with an +# exclusion list that skips every expensive step on a markdown-only or +# changeset-only change — and a markdown-only change is precisely the shape that +# can introduce this defect. A gate that cannot see the pull request shape most +# likely to trip it "rebuilds the hole it exists to close": the conclusion +# `docs-links.yml`, `control-bytes.yml`, `skills-paths.yml`, +# `changeset-presence.yml`, `pre-install-import-graph.yml` and +# `vi-mock-specifiers.yml` have each reached in their own headers. One gate, one +# home. +# +# Hence: no `paths` and no `paths-ignore` here, deliberately. +# `scripts/__tests__/check-shell-escape-residue.test.ts` fails if either is ever +# added, and fails too if a second workflow starts running the same script. +# Reporting on every pull request is also what makes the check requirable, and +# `scripts/dependabot-merge-gate.mjs` classifies it as a required context — an +# unclassified blocking check is one a Dependabot merge would be let past +# (objectui#6135), and the `merge_group` floor now DERIVES from that same list +# (objectui#6160). +# +# It needs no install and no build — a checkout plus one `node` call over ~200 +# markdown documents and ~1300 fenced blocks. Keep it that way; the import graph +# is builtins plus repo-relative modules only, which `pre-install-import-graph.yml` +# enforces (objectui#6148). + +on: + pull_request: + branches: [main, develop] + push: + branches: [main, develop] + # Merge queue (objectui#3523 — see `ci.yml`'s trigger block for the full note + # and the measurements behind it). A required check that does not report on a + # queue build stalls the queue until the ruleset's 60-minute timeout fails it, + # so an unfiltered gate that can become required subscribes here from the + # start. `types:` is named although `checks_requested` is currently the only + # activity type GitHub defines for `merge_group`. + merge_group: + types: [checks_requested] + workflow_dispatch: + +concurrency: + group: shell-escape-residue-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: true + +permissions: + contents: read + +jobs: + shell-escape-residue: + name: Shell Escape Residue Scan + runs-on: ubuntu-latest + timeout-minutes: 5 + + steps: + - name: Checkout code + uses: actions/checkout@v7 + + - name: Setup Node.js + uses: actions/setup-node@v7 + with: + node-version: '22.x' + + # objectui#5150: the `git commit -F -` example in AGENTS.md §9 shipped with + # its heredoc terminator wrapped in the single-quote-inside-single-quote + # shell escape. Copied verbatim it does not fail with a message — it HANGS, + # on a terminator that never matches, and a reader does not attribute a hung + # terminal to the document. objectui#5151 measured the full derived gate + # union against the replanted bytes: control-bytes, doc-links and both + # changeset gates all exited 0, none of them negligently — the residue is + # printable ASCII inside a code block and no scan surface reached it. + # + # ⛔ This gate checks an ENUMERATED LITERAL. It does NOT make fenced shell + # examples executable-by-construction, and nothing in this repository does; + # `bash -n` per block is objectui#5151's unbuilt "direction 1". The script + # header states the boundary and the test suite asserts it as a fact. + # + # GREEN AT REST — there is nothing to find on an ordinary day — so it prints + # its per-root census rather than a bare "OK", and it FAILS when a scan root + # does not resolve or the population collapses. A scan that silently finds + # nothing reads as coverage, which is this gate's own defect one level up. + - name: Check for machine-produced shell-escape residue in fenced examples + run: node scripts/check-shell-escape-residue.mjs diff --git a/content/docs/guide/ci-cd-pipeline.md b/content/docs/guide/ci-cd-pipeline.md index 6e70bb68ea..61230ee566 100644 --- a/content/docs/guide/ci-cd-pipeline.md +++ b/content/docs/guide/ci-cd-pipeline.md @@ -35,6 +35,7 @@ one has its own section below. | `doc-fence-languages.yml` | Doc Fence Language Check | Push / PR to `main`, `develop` — **no path filter**; merge-queue builds; manual | **Yes** — when a TypeScript block sits under a fence the snippet gate does not read | | `pre-install-import-graph.yml` | Pre-Install Import Graph Check | Push / PR to `main`, `develop` — **no path filter**; merge-queue builds; manual | **Yes** — when a gate a workflow runs *before* `pnpm install` reaches a package anywhere in its import graph | | `vi-mock-specifiers.yml` | Inert vi.mock Specifier Check | Push / PR to `main`, `develop` — **no path filter**; merge-queue builds; manual | **Yes** — when a `vi.mock` / `vi.doMock` relative specifier resolves to no file, or the scan's population collapses | +| `shell-escape-residue.yml` | Shell Escape Residue Scan | Push / PR to `main`, `develop` — **no path filter**; merge-queue builds; manual | **Yes** — when a fenced block in `AGENTS.md`, `CLAUDE.md`, `skills/**` or `content/docs/**` carries the enumerated machine-produced shell escape, or a scan root fails to resolve | | `performance-budget.yml` | Bundle Analysis | Push / PR touching `packages/**`, `apps/console/**`, `pnpm-lock.yaml` | **Yes** — the console entry gzip budget | | `live-e2e.yml` | Live E2E (informational) | PR to `main`, `develop` (code paths); nightly cron `30 6 * * *`; manual | No — informational lane, `continue-on-error` | | `labeler.yml` | Auto Label PRs | PR `opened`, `synchronize`, `reopened` | No | @@ -831,6 +832,62 @@ that the suite goes red. Run it locally with `pnpm check:vi-mock-specifiers`, or `node scripts/check-vi-mock-specifiers.mjs --list` to see every call site the walk found. It needs no install and no build. +## Shell Escape Residue (`shell-escape-residue.yml`) + +**Triggers:** Push and PR to `main`/`develop`, merge-queue builds, plus manual dispatch — with **no +path filter at all**. The scan surface is markdown that any shape of pull request can touch, and a +markdown-only change is exactly the shape `ci.yml` and `lint.yml` skip their expensive steps on. It +appears in the checks list as **Shell Escape Residue Scan**. + +Runs `scripts/check-shell-escape-residue.mjs`. It walks `AGENTS.md`, `CLAUDE.md`, every `.md`/`.mdx` +under `skills/` and every one under `content/docs/`, and fails when a **fenced code block** contains +one of the enumerated machine-produced shell-quote escape runs. + +**Why it needed a gate.** In [#5150](https://github.com/objectstack-ai/objectui/issues/5150) the +`git commit -F -` example in `AGENTS.md` §9 shipped with its heredoc terminator wrapped in the +single-quote-inside-single-quote shell escape. Copied verbatim, that example does not fail with a +message — it **hangs**, on a terminator that never matches, and a reader does not attribute a hung +terminal to the document. [#5151](https://github.com/objectstack-ai/objectui/issues/5151) then ran the +full derived gate union against the replanted bytes: `check-control-bytes`, `check-doc-links`, +`check-changeset-presence` and `check-changeset-no-major` **all exited 0**. None of them was +negligent — the residue is printable ASCII inside a code block, and no scan surface in this repository +reached it. The amplifier is that `AGENTS.md`, `CLAUDE.md` and `skills/**` are re-read **once per +session** by every agent seat, so a bad example is not paid once; it is paid by every reader. + +**⛔ What this gate does not do.** It checks an **enumerated literal** — one entry today, the sequence +#5150 leaked. It does **not** make fenced shell examples executable-by-construction, and nothing in +this repository does: a ```bash block may be syntactically invalid, may never terminate, or may name +a flag that does not exist, and this gate is green on all of it. Running `bash -n` over every block is +#5151's **unbuilt** "direction 1"; it was ruled out of that card rather than rejected on the merits, +and it carries a dependency worth recording — it is only as good as its **extraction convention**. In +#5150's own example the block sat inside a numbered list, so both lines carried a two-space indent, +and a quoted heredoc terminator must reach **column 0**. Rendered markdown strips the container indent +and the block looks fine; agents read these files by `cat`, not by rendering them, so a verbatim copy +including the indent hangs exactly as the original defect did. The boundary is asserted as a *fact* in +`scripts/__tests__/check-shell-escape-residue.test.ts` — broken shell is fed to the gate and a pass is +required — rather than pinned as a sentence, so the claim cannot rot into a false one. + +**It is green at rest, so its census is part of the verdict.** There are zero occurrences in the tree +and there should stay zero, which means the run's output alone cannot distinguish a working gate from +one that matches nothing. The verdict line therefore prints the **per-root population** — files and +fenced blocks for each of the four roots — rather than a bare `OK`, and the scan **fails when that +population collapses**: a root that does not resolve, a root that walks to fewer documents than its +floor, or a total fence count under the floor is a broken walk, not a clean tree. A scan root that has +moved or been mistyped is reported **by name**, because a mistyped root and a clean root produce +identical output otherwise. The evidence that the gate works is the ablation in its test suite, which +replants #5150's exact line in each root on a fixture tree. + +**Scope:** fenced blocks only. An occurrence in prose or an inline code span is **counted in the +census and not judged**, because documentation about this defect class has to be able to name the +literal. That is a known narrowing, and the census figure is what keeps it visible. + +**If it fails:** it names the file, line and column, the fence language and the line the fence opened +on. Note that `AGENTS.md`, `CLAUDE.md` and `skills/**` are **governed surface** — a finding in one of +those is reported for a human to fix in its own change, not folded into an unrelated pull request. A +finding under `content/docs/**` is an ordinary docs fix. Run it locally with +`pnpm check:shell-escape-residue`, or `node scripts/check-shell-escape-residue.mjs --list` to see the +per-root census. It needs no install and no build. + ## Link Checking (`check-links.yml`) **Trigger:** Weekly cron (`17 4 * * 0` — Sundays, off the top of the hour, when the scheduled-run diff --git a/package.json b/package.json index 1697fe87ea..9193857327 100644 --- a/package.json +++ b/package.json @@ -59,6 +59,7 @@ "check:entry-guard": "node scripts/check-entry-guard.mjs", "check:pre-install-import-graph": "node scripts/check-pre-install-import-graph.mjs", "check:vi-mock-specifiers": "node scripts/check-vi-mock-specifiers.mjs", + "check:shell-escape-residue": "node scripts/check-shell-escape-residue.mjs", "cli": "node packages/cli/dist/cli.js", "objectui": "node packages/cli/dist/cli.js", "create-plugin": "node packages/create-plugin/dist/index.js", diff --git a/scripts/__tests__/check-shell-escape-residue.test.ts b/scripts/__tests__/check-shell-escape-residue.test.ts new file mode 100644 index 0000000000..bb55613e3a --- /dev/null +++ b/scripts/__tests__/check-shell-escape-residue.test.ts @@ -0,0 +1,458 @@ +import { describe, expect, it } from 'vitest'; +import { execFileSync } from 'node:child_process'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; + +// Plain-JS CI helper. Its types are INFERRED from the .mjs source by +// `tsconfig.scripts.json` (`allowJs`), so no `@ts-expect-error` here — see +// objectui#3494. +import { + FENCE_FLOOR, + RESIDUE_PATTERNS, + SCAN_ROOTS, + findResidue, + resolveRoot, + scan, + summarise, +} from '../check-shell-escape-residue.mjs'; +import { REQUIRED_CONTEXTS } from '../dependabot-merge-gate.mjs'; + +/** + * objectui#5151 — the test for `scripts/check-shell-escape-residue.mjs`. + * + * ## Why this file carries the weight + * + * The gate is GREEN AT REST. objectui#5150's fix (PR #5152) removed the only + * occurrence that has ever existed, and the measured population across all four + * scan roots is ZERO. So a green run of the gate over this repository proves + * only that this repository is clean; it cannot distinguish a working gate from + * one that matches nothing at all — which is objectui#5151's own defect, one + * level up. ⭐ THE ABLATION BELOW IS THE ONLY EVIDENCE THE GATE EXISTS. + * + * It is not synthetic. `HISTORICAL_LINE` reconstructs the exact line objectui#5150 + * shipped, in its exact geometry: two-space indented (it sat inside a numbered + * list) inside a ```bash fence, carrying the escape run twice — once around the + * heredoc introducer's terminator and once around its repeat. + * + * ## ⛔ The limitation is asserted as a FACT, never as a sentence + * + * The gate's header says executability is unguarded. A `toContain` pin over that + * sentence would assert that the CLAIM EXISTS, not that it is TRUE — the failure + * this lane spent a round on in objectui#6186, where a pin read as coverage while + * the claim underneath it rotted. So the boundary cases below FEED THE GATE + * broken shell and require a PASS. If someone ever widens this gate into a real + * syntax check, those cases go red and have to be rewritten deliberately, which + * is the correct amount of friction for changing what a gate promises. + * + * ## Fixture discipline + * + * `scripts/` is not in `SCAN_ROOTS`, so this file could carry the literal + * plainly. It builds it from code points anyway — belt and braces against a + * future widening of the scan surface turning this suite into the gate's own + * first finding — and then PINS the constructed value against the shipped + * `RESIDUE_PATTERNS` entry, so a typo in the source literal reddens here. + */ + +const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../..'); + +const SQ = String.fromCharCode(39); // ' +const DQ = String.fromCharCode(34); // " + +/** The residue, rebuilt from code points. */ +const RESIDUE = SQ + DQ + SQ + DQ + SQ; + +/** + * objectui#5150's shipped line, byte-for-byte, indent included. The indent is + * not decoration: the block sat inside a numbered list, and a heredoc terminator + * has to reach column 0 — which is the separate defect the triage note recorded + * against direction 1 and which this gate deliberately does not judge. + */ +const HISTORICAL_LINE = ` git commit -F - <<${RESIDUE}EOF${RESIDUE}`; + +/** A fenced block, as document source. */ +const fence = (language: string, ...body: string[]) => ['```' + language, ...body, '```'].join('\n'); + +/** Build a throwaway document tree and hand back its root. */ +function fixtureTree(files: Record): string { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'check-shell-residue-')); + for (const [rel, body] of Object.entries(files)) { + const full = path.join(root, rel); + fs.mkdirSync(path.dirname(full), { recursive: true }); + fs.writeFileSync(full, body); + } + return root; +} + +/** The fixture roots: same shape as `SCAN_ROOTS`, floors a fixture can meet. */ +const FIXTURE_ROOTS = [ + { spec: 'AGENTS.md', kind: 'file', minFiles: 1 }, + { spec: 'CLAUDE.md', kind: 'file', minFiles: 1 }, + { spec: 'skills', kind: 'dir', minFiles: 1 }, + { spec: 'content/docs', kind: 'dir', minFiles: 1 }, +]; + +const scanFixture = (root: string, roots = FIXTURE_ROOTS) => scan(root, { roots, fenceFloor: 0 }); + +/** A finding as `findResidue` returns it — no file, because it scans one source. */ +type Hit = { line: number; column: number; language: string; fenceLine: number; patternId: string }; +/** The same finding as `scan` returns it, carrying the document it came from. */ +type ScanHit = Hit & { file: string; root: string }; + +// --------------------------------------------------------------------------- +// The literal itself +// --------------------------------------------------------------------------- + +describe('the enumeration', () => { + it('ships exactly the literal objectui#5150 leaked, rebuilt from code points', () => { + // Pins the source constant against an independent construction: a typo in + // `RESIDUE_PATTERNS` — a gate matching a sequence that never occurs — is + // otherwise invisible, because both a working gate and a broken one are + // green on this tree. + expect(RESIDUE_PATTERNS.map((p: { literal: string }) => p.literal)).toEqual([RESIDUE]); + }); + + it('has ONE entry — a second needs an observed instance, not an argument', () => { + // The whole value of this direction over `bash -n` is a zero false-positive + // rate. The first speculative literal spends it, so the count is pinned and + // widening it is a deliberate edit to this line. + expect(RESIDUE_PATTERNS).toHaveLength(1); + }); + + it('reconstructs the historical line with the run appearing twice', () => { + expect(HISTORICAL_LINE.split(RESIDUE).length - 1).toBe(2); + expect(HISTORICAL_LINE.startsWith(' ')).toBe(true); + }); +}); + +// --------------------------------------------------------------------------- +// findResidue — the unit +// --------------------------------------------------------------------------- + +describe('findResidue — what is judged', () => { + it('flags every occurrence inside a fenced block, with line and column', () => { + const source = ['# Doc', '', fence('bash', HISTORICAL_LINE, ' EOF'), ''].join('\n'); + const { hits } = findResidue(source); + + expect(hits.map((h: Hit) => [h.line, h.column])).toEqual([ + [4, 21], + [4, 29], + ]); + expect(hits.every((h: Hit) => h.language === 'bash')).toBe(true); + expect(hits.every((h: Hit) => h.fenceLine === 3)).toBe(true); + }); + + it('judges a fence whatever its info string says — no language vocabulary to rot', () => { + // Deliberately NOT an allow-list of shell fence spellings. `sh`, `shell`, + // `console`, or no info string at all are all how this could be written, and + // an enumeration is the thing that rots (objectui#6135's standing lesson). + for (const language of ['bash', 'sh', 'shell', 'console', '']) { + const { hits } = findResidue(fence(language, HISTORICAL_LINE)); + expect(hits, `a ${language || '(bare)'} fence must be judged`).toHaveLength(2); + } + }); + + it('COUNTS an occurrence outside a fence and does not judge it', () => { + // The one deliberate narrowing. Prose about this defect class has to be able + // to NAME the literal — objectui#5151's own body does — and no mechanical + // rule separates quoting it from shipping it in running text. The census + // figure is what keeps the exclusion visible rather than silent. + const { hits, outsideFences } = findResidue(`The leaked run is \`${RESIDUE}\`, five printable bytes.`); + expect(hits).toEqual([]); + expect(outsideFences).toBe(1); + }); + + it('splits a document carrying both, so neither figure absorbs the other', () => { + const source = [`prose ${RESIDUE} prose`, '', fence('bash', HISTORICAL_LINE)].join('\n'); + const { hits, outsideFences } = findResidue(source); + expect(hits).toHaveLength(2); + expect(outsideFences).toBe(1); + }); + + it('counts the fences it examined, including clean ones', () => { + const source = [fence('ts', 'const a = 1;'), '', fence('bash', 'echo hi')].join('\n'); + const { fences, hits } = findResidue(source); + expect(fences).toBe(2); + expect(hits).toEqual([]); + }); + + it('is green on a document with no fences at all', () => { + expect(findResidue('# Just prose\n\nNothing fenced here.\n')).toEqual({ fences: 0, hits: [], outsideFences: 0 }); + }); +}); + +// --------------------------------------------------------------------------- +// ⛔ The boundary, asserted as a fact — see the header +// --------------------------------------------------------------------------- + +describe('⛔ what this gate does NOT do — asserted by behaviour, not by prose', () => { + it('PASSES a ```bash fence that cannot execute — executability is unguarded', () => { + // Every line here is broken shell. The gate is green on all of it, because + // it checks an enumerated literal and nothing else. Nothing in this + // repository checks that a fenced shell example runs; objectui#5151's + // "direction 1" (`bash -n` per block) is the unbuilt option. + const broken = fence( + 'bash', + 'if [ -f x ]', // no `then`, no `fi` + 'for i in', // truncated + 'echo "unterminated', // unbalanced quote + 'cat < { + // The scenario the triage note attached to direction 1: inside a numbered + // list both lines carry the container indent, and a quoted heredoc + // terminator must reach COLUMN 0. Rendered markdown hides it; agents read + // these files by `cat`. This gate does not judge it — stated in its header + // and pinned here, so the claim cannot rot into a false one. + const indented = fence('bash', ` git commit -F - <<${SQ}EOF${SQ}`, ' message', ' EOF'); + expect(indented).not.toContain(RESIDUE); + expect(findResidue(indented).hits).toEqual([]); + }); + + it("PASSES the equivalent '\\'' spelling — the documented remedy is not matched", () => { + const alternative = fence('bash', `git commit -F - <<${SQ}\\${SQ}${SQ}EOF`); + expect(findResidue(alternative).hits).toEqual([]); + }); + + it('looks at nothing outside SCAN_ROOTS', () => { + const root = fixtureTree({ + 'AGENTS.md': fence('bash', 'echo ok'), + 'CLAUDE.md': fence('bash', 'echo ok'), + 'skills/s/SKILL.md': fence('bash', 'echo ok'), + 'content/docs/a.md': fence('bash', 'echo ok'), + // Out of scope by construction: not under any declared root. + 'packages/thing/README.md': fence('bash', HISTORICAL_LINE), + }); + expect(scanFixture(root).hits).toEqual([]); + }); +}); + +// --------------------------------------------------------------------------- +// ⭐ The ablation — the gate's only evidence, because it is green at rest +// --------------------------------------------------------------------------- + +describe('⭐ ablation — objectui#5150 replanted in every scan root', () => { + const clean = { + 'AGENTS.md': ['# AGENTS', '', fence('bash', `git commit -F - <<${SQ}EOF${SQ}`, 'msg', 'EOF')].join('\n'), + 'CLAUDE.md': ['# CLAUDE', '', fence('bash', 'pnpm install')].join('\n'), + 'skills/objectui/SKILL.md': ['# Skill', '', fence('bash', 'pnpm build')].join('\n'), + 'content/docs/guide/a.md': ['# Guide', '', fence('bash', 'pnpm test')].join('\n'), + }; + + it('is GREEN on the clean fixture — the control leg', () => { + const result = scanFixture(fixtureTree(clean)); + expect(result.hits).toEqual([]); + expect(result.unresolved).toEqual([]); + expect(result.vacuous).toEqual([]); + expect(result.census.fences).toBe(4); + }); + + it('goes RED in each root separately, naming the file and the line', () => { + for (const target of Object.keys(clean)) { + const planted = { ...clean, [target]: `${clean[target as keyof typeof clean]}\n\n${fence('bash', HISTORICAL_LINE, ' EOF')}\n` }; + const result = scanFixture(fixtureTree(planted)); + + const hits: ScanHit[] = result.hits; + expect(hits.map((h) => h.file), `planting in ${target} must be found there and nowhere else`).toEqual([ + target, + target, + ]); + // The run appears twice on the one line: around the introducer's + // terminator and around its repeat. + expect(new Set(hits.map((h) => h.line)).size).toBe(1); + expect(hits[0].column).toBeLessThan(hits[1].column); + expect(hits[0].patternId).toBe(RESIDUE_PATTERNS[0].id); + } + }); + + it('finds all four at once, and is loud about none of the roots collapsing', () => { + const planted = Object.fromEntries( + Object.entries(clean).map(([rel, body]) => [rel, `${body}\n\n${fence('bash', HISTORICAL_LINE)}\n`]), + ); + const result = scanFixture(fixtureTree(planted)); + expect(result.hits).toHaveLength(8); + expect(result.census.rootsResolved).toBe(4); + expect(result.census.outsideFences).toBe(0); + }); +}); + +// --------------------------------------------------------------------------- +// Non-vacuity — a scan that found nothing must FAIL, not pass +// --------------------------------------------------------------------------- + +describe('non-vacuity — zero roots or zero files is a failure, not a green', () => { + it('⚠️ reports a MISSING root by name instead of scanning zero files quietly', () => { + // A mistyped root and a clean root produce identical output otherwise, and + // the mistyped one reads as coverage for as long as nobody checks. + const root = fixtureTree({ 'AGENTS.md': fence('bash', 'echo ok') }); + const result = scan(root, { roots: FIXTURE_ROOTS, fenceFloor: 0 }); + expect(result.unresolved.map((u: { spec: string }) => u.spec)).toEqual(['CLAUDE.md', 'skills', 'content/docs']); + expect(result.census.rootsResolved).toBe(1); + }); + + it('reports a root whose KIND changed — a file where a directory was declared', () => { + const root = fixtureTree({ 'skills': fence('bash', 'echo ok') }); + expect(resolveRoot(root, { spec: 'skills', kind: 'dir' })).toMatchObject({ + ok: false, + problem: 'is declared a directory but is a file', + }); + expect(resolveRoot(root, { spec: 'AGENTS.md', kind: 'file' })).toMatchObject({ ok: false, problem: 'does not exist' }); + }); + + it('reports a resolved-but-empty root as a COLLAPSE, not as clean', () => { + const root = fixtureTree({ 'AGENTS.md': '', 'CLAUDE.md': '', 'skills/.keep': '', 'content/docs/.keep': '' }); + const result = scan(root, { roots: FIXTURE_ROOTS, fenceFloor: 0 }); + // `.keep` is not a document, so both directory roots resolve and walk to nothing. + expect(result.vacuous.map((v: { what: string }) => v.what)).toEqual(['files under skills', 'files under content/docs']); + }); + + it('treats a fence count under the floor as a collapse of the walk', () => { + const root = fixtureTree({ + 'AGENTS.md': fence('bash', 'echo ok'), + 'CLAUDE.md': '# no fences', + 'skills/s/SKILL.md': '# no fences', + 'content/docs/a.md': '# no fences', + }); + const result = scan(root, { roots: FIXTURE_ROOTS, fenceFloor: 400 }); + expect(result.vacuous).toEqual([{ what: 'fenced blocks examined', value: 1, floor: 400 }]); + }); + + it('exits 1 for a collapsed population even with nothing to report', () => { + // The direction that matters: a broken walk must not be reported as OK. + const root = fixtureTree({ 'AGENTS.md': fence('bash', 'echo ok') }); + const result = scan(root, { roots: FIXTURE_ROOTS, fenceFloor: 400 }); + expect(result.hits).toEqual([]); + expect(result.unresolved.length + result.vacuous.length).toBeGreaterThan(0); + }); +}); + +// --------------------------------------------------------------------------- +// The tree +// --------------------------------------------------------------------------- + +describe('repo state — the gate is green on this tree, over a real population', () => { + const result = scan(repoRoot); + + it('scans exactly the four roots objectui#5151 ruled', () => { + expect(SCAN_ROOTS.map((r: { spec: string }) => r.spec)).toEqual([ + 'AGENTS.md', + 'CLAUDE.md', + 'skills', + 'content/docs', + ]); + }); + + it('has no residue in any fenced block', () => { + expect( + result.hits.map((h: ScanHit) => `${h.file}:${h.line}:${h.column}`), + 'Run `pnpm check:shell-escape-residue` for the full report and the remedy.', + ).toEqual([]); + }); + + it('resolved every root and actually walked them', () => { + // Without this the assertion above passes for the wrong reason the day a + // root moves. Floors, not exact counts — measured 204 files / 1305 fences + // when this landed. + expect(result.unresolved).toEqual([]); + expect(result.census.rootsResolved).toBe(SCAN_ROOTS.length); + expect(result.census.files).toBeGreaterThan(100); + expect(result.census.fences).toBeGreaterThan(FENCE_FLOOR); + expect(result.vacuous).toEqual([]); + }); + + it('puts the per-root census in the verdict, so a reader sees the population', () => { + // "OK" alone is what a gate that does nothing also prints. + const line = summarise(result); + for (const root of SCAN_ROOTS) expect(line).toContain(`${root.spec}: `); + expect(line).toMatch(/\d+ file\(s\) and \d+ fenced block\(s\) examined/); + + const out = execFileSync('node', ['scripts/check-shell-escape-residue.mjs'], { cwd: repoRoot, encoding: 'utf8' }); + expect(out).toMatch(/check-shell-escape-residue: OK/); + expect(out).toContain(`${result.census.fences} fenced block(s) examined`); + }); + + it('needs no install and no build — it is a cheap-tier gate', () => { + const src = fs.readFileSync(path.join(repoRoot, 'scripts/check-shell-escape-residue.mjs'), 'utf8'); + const imports = [...src.matchAll(/^import .*? from '([^']+)';$/gm)].map((m) => m[1]); + expect(imports.length).toBeGreaterThan(2); + for (const spec of imports) { + expect(spec.startsWith('node:') || spec.startsWith('./'), `${spec} would need an install`).toBe(true); + } + }); +}); + +// --------------------------------------------------------------------------- +// Wiring +// --------------------------------------------------------------------------- + +describe('wiring — the gate is reachable and every PR shape starts it', () => { + const SCRIPT = 'scripts/check-shell-escape-residue.mjs'; + const WORKFLOW = 'shell-escape-residue.yml'; + const CHECK_NAME = 'Shell Escape Residue Scan'; + const workflowDir = path.join(repoRoot, '.github/workflows'); + const workflowFiles = fs.readdirSync(workflowDir).filter((f) => f.endsWith('.yml')); + + /** A workflow's YAML with whole-line comments removed — headers discuss + * `paths` and each other's scripts in prose. */ + const yamlOf = (file: string) => + fs + .readFileSync(path.join(workflowDir, file), 'utf8') + .split('\n') + .filter((line) => !/^\s*#/.test(line)) + .join('\n'); + + it('is exposed as a root package script', () => { + const pkg = JSON.parse(fs.readFileSync(path.join(repoRoot, 'package.json'), 'utf8')); + expect(pkg.scripts['check:shell-escape-residue']).toBe(`node ${SCRIPT}`); + }); + + it('has a workflow that gates pull requests, not just pushes', () => { + expect(fs.existsSync(path.join(workflowDir, WORKFLOW)), 'a check nothing runs is not a gate').toBe(true); + const yaml = yamlOf(WORKFLOW); + expect(yaml).toMatch(new RegExp(`run:\\s*node\\s+${SCRIPT.replace(/[.]/g, '\\.')}`)); + expect(yaml).toMatch(/^\s*pull_request:/m); + expect(yaml).toMatch(/^\s*push:/m); + expect(yaml).toContain(`name: ${CHECK_NAME}`); + }); + + it('subscribes merge_group — a required check that skips a queue build stalls it', () => { + // objectui#3523: `main` sits behind an enforced queue, and a required + // context that never reports does not fail the queue, it hangs it until the + // ruleset's 60-minute timeout. + expect(yamlOf(WORKFLOW)).toMatch(/^\s*merge_group:/m); + }); + + it('is classified as a required context — the merge_group floor DERIVES from that list', () => { + // objectui#6160 / PR #6187: `merge-queue-reporting.test.ts` reads + // `REQUIRED_CONTEXTS` to decide which workflows must subscribe the queue, + // and `dependabot-merge-gate.mjs` would let a Dependabot merge past an + // unclassified blocking check (objectui#6135). + expect(REQUIRED_CONTEXTS).toContain(CHECK_NAME); + }); + + it('runs it in NO path-filtered workflow', () => { + // The scan surface is markdown that any pull request shape can touch, and + // reporting on every shape is also what makes the check requirable. + expect(workflowFiles.length, 'the workflow directory scan returned implausibly few files').toBeGreaterThan(5); + for (const file of workflowFiles) { + const yaml = yamlOf(file); + if (!yaml.includes(SCRIPT)) continue; + expect(yaml, `${file} runs ${SCRIPT} behind a paths-ignore`).not.toMatch(/paths-ignore:/); + expect(yaml, `${file} runs ${SCRIPT} behind a paths filter — see objectui#3448`).not.toMatch(/^\s+paths:/m); + } + }); + + it('has exactly one home', () => { + expect(workflowFiles.filter((f) => yamlOf(f).includes(SCRIPT))).toEqual([WORKFLOW]); + }); + + it('installs nothing before running the gate — the cheap tier, mechanically', () => { + const yaml = yamlOf(WORKFLOW); + expect(yaml).not.toMatch(/pnpm install/); + expect(yaml.indexOf(SCRIPT)).toBeGreaterThan(-1); + }); +}); diff --git a/scripts/check-shell-escape-residue.mjs b/scripts/check-shell-escape-residue.mjs new file mode 100644 index 0000000000..c79a487aa9 --- /dev/null +++ b/scripts/check-shell-escape-residue.mjs @@ -0,0 +1,488 @@ +#!/usr/bin/env node +/** + * check-shell-escape-residue -- rejects a known MACHINE-PRODUCED shell-quote + * escape run that has leaked into a fenced code block in agent-facing markdown. + * + * node scripts/check-shell-escape-residue.mjs (pnpm check:shell-escape-residue) + * node scripts/check-shell-escape-residue.mjs --list every fence the walk examined + * node scripts/check-shell-escape-residue.mjs --json + * + * Exit: 0 = no enumerated residue inside a fenced block, and the population did + * not collapse. 1 = one of those. + * + * ## ⛔ WHAT THIS GATE DOES NOT DO -- read this before citing it as coverage + * + * It checks ONE ENUMERATED LITERAL (`RESIDUE_PATTERNS`, currently a single + * entry) inside fenced blocks in four roots. That is the whole of it. + * + * ⛔ It does NOT make fenced shell examples executable-by-construction, and + * nothing in this repository does. A ```bash block may be syntactically + * invalid, may never terminate, may reference a flag that does not exist, + * may `rm -rf` the wrong path -- this gate is green on every one of them. + * EXECUTABILITY IS UNGUARDED. `scripts/__tests__/check-shell-escape-residue.test.ts` + * asserts that as a FACT rather than as a sentence in this header: it feeds + * the gate a fence carrying an unterminated heredoc and a broken `if`, and + * requires a PASS. + * + * ⛔ The unbuilt option is objectui#5151's "direction 1": run `bash -n` over + * every ```bash / ```sh block. It was ruled out of this card, not rejected + * on the merits, and it carries a dependency worth recording so nobody + * re-proposes it blindly -- IT IS ONLY AS GOOD AS ITS EXTRACTION + * CONVENTION. In objectui#5150's own example the block sits inside a + * numbered list, so in the raw file both lines carry a two-space indent, + * and a heredoc opened `<<` + a quoted terminator requires that terminator + * at COLUMN 0. Rendered markdown strips the container indent and the block + * looks fine. ⭐ Agents read these files by `cat`, not by rendering them, + * so a verbatim copy INCLUDING the indent hangs in exactly the way the + * original defect did. Whether `bash -n` catches that depends entirely on + * whether the extractor dedents first -- so if direction 1 is ever built, + * decide and STATE what the extractor does with container indentation, and + * pick the answer matching how the file is actually consumed. + * + * ⛔ It does not judge text OUTSIDE a fenced block. See "The one deliberate + * narrowing" below; the census counts those occurrences so the exclusion is + * a number rather than a silence. + * + * ⛔ It does not look outside `SCAN_ROOTS`. + * + * The name is chosen to say all of that: `shell-escape-residue`, not + * `shell-examples`. A gate named for a general property while checking a literal + * list is how "no gate covers this" becomes "a gate covers this" in someone's + * head a month later (objectui#5151 triage, made binding at dispatch). + * + * ## The defect (objectui#5150, fixed by PR #5152; the class is objectui#5151) + * + * `AGENTS.md` §9 recommends `git commit -F -` with a heredoc. The line landed as + * the heredoc introducer with its `EOF` wrapped in the single-quote-inside- + * single-quote shell escape -- five characters, all printable ASCII. Copied + * verbatim it opens a heredoc whose terminator is not `EOF`, so the closing line + * never matches. + * + * Two amplifiers, and the card's weight is both of them together: + * + * 1. **There is no error to look at.** It does not exit with a message. It + * HANGS. A reader will not attribute a hung terminal to the document, and + * will fall back to exactly the workaround the clause argues against. + * 2. **Agent-facing text is re-read once per session.** `AGENTS.md`, + * `CLAUDE.md` and `skills/**` are inputs to every seat, so a bad example is + * not paid once -- it is paid by every reader. Those three are also where + * this repository does most of its shell demonstrating. + * + * And the GENERATING MECHANISM IS FIXED AND REPRODUCIBLE: an agent writing a + * file through a shell heredoc that is itself nested inside a single-quoted + * argument leaks that escape run into the content. Neither side guarded it -- + * not the producing side, not the checking side. + * + * ## What was measured, and why a gate rather than a note (objectui#5151) + * + * The broken bytes were re-planted on the fix branch and the FULL derived gate + * union for a change to `AGENTS.md` was run against them: + * + * check-control-bytes exit 0 -- judges control bytes; 0x27 and 0x22 + * are printable, so out of set BY DESIGN + * check-doc-links exit 0 -- has AGENTS.md in scope, but parses + * LINKS; block contents are not read + * check-changeset-presence exit 0 -- judges declarations, never content + * check-changeset-no-major exit 0 -- likewise + * + * All green. Not one of them was negligent: the residue is printable ASCII in a + * code block, and no scan surface in this repository reached it. That is the + * hole, and this file is the narrow half of it. + * + * ## ⚠️ GREEN AT REST -- so the ablation is the only evidence this gate exists + * + * There were ZERO occurrences in the tree when this landed (PR #5152 removed the + * only one) and there should stay zero. A green run over today's tree therefore + * proves only that today's tree is clean; it cannot distinguish a working gate + * from one matching nothing at all -- which is objectui#5151's own defect, one + * level up. Two consequences, both load-bearing and both copied deliberately + * from `check-vi-mock-specifiers.mjs` (objectui#5646), which is green at rest + * for the same reason: + * + * 1. **The population must refuse to collapse.** Zero roots resolved, zero + * files scanned, or a fence count under `FENCE_FLOOR` is not a clean tree; + * it is a broken walk, and reporting OK for it would be this card's defect + * wearing the gate's uniform. Each is a FAILURE. + * 2. **⚠️ A root that does not exist is LOUD, never a silent zero.** A + * mistyped root and a clean root produce identical output otherwise, and + * the mistyped one reads as coverage forever. `SCAN_ROOTS` declares each + * root's `kind`, so a root that vanished, moved, or turned from a file into + * a directory is reported by name and separately from an ordinary finding. + * 3. **The verdict line carries the census** -- files and fences PER ROOT -- + * so a reader sees the population the green was computed over. + * + * `scripts/__tests__/check-shell-escape-residue.test.ts` carries the ablation: + * objectui#5150's exact shipped line, reconstructed on a fixture tree. + * + * ## The enumeration, and why it has exactly one member + * + * `RESIDUE_PATTERNS` is a list of literals OBSERVED to be produced by a machine + * and shipped. It has one entry because exactly one has been observed. ⛔ Adding + * a second requires an observed instance, not a plausible one: the entire value + * of this direction over direction 1 is that its false-positive rate is zero, + * and the first speculative entry spends that. + * + * The measured population of the current entry across the four roots, at the + * commit this landed on, is ZERO inside fences and ZERO outside them. + * + * ## ⚠️ The literal is legitimate shell -- so the claim here is narrower + * + * `'` + `"` + `'` + `"` + `'` is the standard way to put a literal single quote + * inside a single-quoted string, and a hand-written one-liner may use it + * correctly. This gate does not claim otherwise. It claims something smaller and + * checkable: in THESE FOUR ROOTS the sequence has never once been intentional, + * and every occurrence so far was a write-path leak. + * + * If a deliberate instance is ever genuinely needed in a documented example, the + * equivalent `'\''` spelling is not matched here and is the documented remedy. + * ⛔ There is deliberately no allowlist, baseline or inline opt-out: with a + * measured population of zero, a permit mechanism would be the only thing in + * this file anyone ever reached for. + * + * ## The one deliberate narrowing: fenced blocks only + * + * Occurrences are JUDGED inside a fenced code block and COUNTED everywhere else, + * with the outside-fence count reported in the census. The reason is that prose + * about this defect class has to be able to NAME the literal -- objectui#5151's + * own body does, and a future `AGENTS.md` clause documenting this very gate + * would too -- and no mechanical rule separates "quoting the residue" from + * "shipping it" in running text. + * + * ⚠️ This is a known gap, not a claim of completeness: residue inside an INLINE + * code span is just as copy-pasteable and is NOT judged here. The census figure + * is what keeps that visible -- a non-zero `outsideFences` is a number a reader + * can act on. An exclusion nobody can see in the census is how a scan narrows + * itself into vacuity. + * + * ## One fence walker, not a second copy + * + * `scanFences` is imported from `check-doc-fence-languages.mjs` rather than + * re-implemented. Fence scanning is already this repository's answer to "where + * does a code block start and end", it is pinned by that gate's own test, and + * objectui#3261/#3279 are the standing lesson that a second copy of one fact is + * a second answer that drifts. It costs nothing: that module's whole import + * graph is node builtins plus `./invoked-as.mjs`, so this gate stays in the + * cheap pre-install tier that `check-pre-install-import-graph.mjs` enforces. + */ + +import { existsSync, readFileSync, readdirSync, statSync } from 'node:fs'; +import { dirname, join, relative, resolve, sep } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import { isEntrypoint } from './invoked-as.mjs'; +import { scanFences } from './check-doc-fence-languages.mjs'; + +const HERE = dirname(fileURLToPath(import.meta.url)); + +/** The repository root -- this file lives at `scripts/` depth 0. */ +export function repoRoot() { + return resolve(HERE, '..'); +} + +/** Documents the walk reads. */ +const DOC_EXTENSIONS = ['.mdx', '.md']; + +/** + * The scan surface, declared -- objectui#5151's dispatch ruling, verbatim: + * `AGENTS.md`, `CLAUDE.md`, `skills/**`, `content/docs/**`. + * + * `kind` and `minFiles` exist for the "loud, never a silent zero" rule above. + * `minFiles` is a COLLAPSE floor set with room, not today's count: the point is + * to catch a walk that broke, not to pin figures that move every day. Measured + * when this landed: 1 / 1 / 18 / 184 files, 12 / 2 / 235 / 1056 fences. + * + * ⚠️ `AGENTS.md`, `CLAUDE.md` and `skills/**` are GOVERNED SURFACE (AGENTS.md + * §受管面). This gate READS them and never writes: a finding in one of those is + * reported for a human to act on, and fixing it is a separate, human-merged + * change. That is a property of the remedy, not of the scan. + */ +export const SCAN_ROOTS = Object.freeze([ + { spec: 'AGENTS.md', kind: 'file', minFiles: 1 }, + { spec: 'CLAUDE.md', kind: 'file', minFiles: 1 }, + { spec: 'skills', kind: 'dir', minFiles: 5 }, + { spec: 'content/docs', kind: 'dir', minFiles: 100 }, +]); + +/** + * Literals OBSERVED to be machine-produced and shipped. One entry; see the + * header for why a second one needs an observed instance rather than an + * argument. + * + * This source file is not itself in `SCAN_ROOTS`, so the literal is written out + * plainly here rather than assembled from code points. + */ +export const RESIDUE_PATTERNS = Object.freeze([ + { + id: 'single-quote-in-single-quote', + literal: `'"'"'`, + /** What a reader sees when this fires. */ + what: 'the single-quote-inside-single-quote shell escape', + instance: 'objectui#5150 -- AGENTS.md §9 shipped `git commit -F -` with its heredoc terminator wrapped in it', + remedy: "Delete the escape run. Inside a fenced example the quotes are already literal, so the intended text is the bare form (`<<` + 'EOF' quoted once). If a literal single quote inside a single-quoted string is genuinely meant, spell it '\\'' -- that form is not matched here.", + }, +]); + +/** + * Total fenced blocks below which the walk did not happen. 1305 were examined + * when this landed; the floor is set far under that on purpose -- it catches a + * fence walker that stopped matching, not a day when the docs got shorter. + */ +export const FENCE_FLOOR = 400; + +/** Every document under one resolved root, repo-relative, in a stable order. */ +export function listDocuments(root, spec, kind) { + const abs = join(root, spec); + if (kind === 'file') return [spec]; + const out = []; + const walk = (dir) => { + for (const entry of readdirSync(dir).sort()) { + const p = join(dir, entry); + if (statSync(p).isDirectory()) walk(p); + else if (DOC_EXTENSIONS.some((ext) => entry.endsWith(ext))) out.push(relative(root, p).split(sep).join('/')); + } + }; + walk(abs); + return out; +} + +/** + * Resolve one declared root, or say precisely how it failed to resolve. + * + * @returns {{ spec: string, kind: string, ok: boolean, problem: string | null }} + */ +export function resolveRoot(root, { spec, kind }) { + const abs = join(root, spec); + if (!existsSync(abs)) return { spec, kind, ok: false, problem: 'does not exist' }; + const isDir = statSync(abs).isDirectory(); + if (kind === 'file' && isDir) return { spec, kind, ok: false, problem: 'is declared a file but is a directory' }; + if (kind === 'dir' && !isDir) return { spec, kind, ok: false, problem: 'is declared a directory but is a file' }; + return { spec, kind, ok: true, problem: null }; +} + +/** Every occurrence of every pattern on one line, as 1-based columns. */ +function occurrencesOn(line) { + const found = []; + for (const pattern of RESIDUE_PATTERNS) { + let from = 0; + for (;;) { + const at = line.indexOf(pattern.literal, from); + if (at === -1) break; + found.push({ patternId: pattern.id, column: at + 1 }); + from = at + 1; // overlapping runs are separate findings, not one + } + } + return found; +} + +/** + * One finding: a residue occurrence inside a fenced block. + * + * @typedef {{ + * line: number, + * column: number, + * patternId: string, + * fenceLine: number, + * language: string, + * text: string, + * }} ResidueHit + */ + +/** + * One document's residue, split into what is judged and what is only counted. + * + * Pure -- the test drives it over fixture sources rather than over the tree. + * + * @returns {{ fences: number, hits: ResidueHit[], outsideFences: number }} + */ +export function findResidue(source) { + const lines = source.split('\n'); + const blocks = scanFences(source); + + const hits = []; + for (const block of blocks) { + const body = block.body === '' ? [] : block.body.split('\n'); + body.forEach((text, i) => { + for (const o of occurrencesOn(text)) { + hits.push({ + // `fenceLine` is the 1-based line of the OPENING fence, so the body + // starts on the line after it. + line: block.fenceLine + 1 + i, + column: o.column, + patternId: o.patternId, + fenceLine: block.fenceLine, + language: block.language, + text: text.trim().slice(0, 100), + }); + } + }); + } + + let total = 0; + for (const text of lines) total += occurrencesOn(text).length; + + return { fences: blocks.length, hits, outsideFences: total - hits.length }; +} + +/** + * The one scan. `main()`, `--list`, `--json` and the test suite all go through + * here, so the tests exercise the real code path rather than an imitation. + * + * @param {string} root Repository root to scan. + * @param {{ roots?: ReadonlyArray, fenceFloor?: number }} [options] + * `roots` overrides `SCAN_ROOTS` (fixtures declare their own); `fenceFloor` + * overrides `FENCE_FLOOR` -- pass 0 for a fixture tree, which is legitimately + * far below any repo floor. + */ +export function scan(root, { roots = SCAN_ROOTS, fenceFloor = FENCE_FLOOR } = {}) { + const perRoot = []; + const unresolved = []; + const hits = []; + let outsideFences = 0; + + for (const declared of roots) { + const resolved = resolveRoot(root, declared); + if (!resolved.ok) { + unresolved.push(resolved); + perRoot.push({ spec: declared.spec, files: 0, fences: 0, minFiles: declared.minFiles, resolved: false }); + continue; + } + + const documents = listDocuments(root, declared.spec, declared.kind); + let fences = 0; + for (const rel of documents) { + let source; + try { + source = readFileSync(join(root, rel), 'utf8'); + } catch { + continue; // symlink, gitlink, unreadable -- nothing to judge + } + const found = findResidue(source); + fences += found.fences; + outsideFences += found.outsideFences; + for (const hit of found.hits) hits.push({ file: rel, root: declared.spec, ...hit }); + } + + perRoot.push({ + spec: declared.spec, + files: documents.length, + fences, + minFiles: declared.minFiles, + resolved: true, + }); + } + + const census = { + roots: roots.length, + rootsResolved: perRoot.filter((r) => r.resolved).length, + files: perRoot.reduce((n, r) => n + r.files, 0), + fences: perRoot.reduce((n, r) => n + r.fences, 0), + perRoot, + outsideFences, + }; + + // The population, checked for collapse. See "GREEN AT REST" in the header. + const vacuous = []; + for (const r of perRoot) { + if (r.resolved && r.files < r.minFiles) { + vacuous.push({ what: `files under ${r.spec}`, value: r.files, floor: r.minFiles }); + } + } + if (census.fences < fenceFloor) { + vacuous.push({ what: 'fenced blocks examined', value: census.fences, floor: fenceFloor }); + } + + return { census, hits, unresolved, vacuous }; +} + +/** The census, as one line, for the verdict. */ +export function summarise({ census }) { + const per = census.perRoot + .map((r) => `${r.spec}: ${r.resolved ? `${r.files} file(s), ${r.fences} fence(s)` : 'UNRESOLVED'}`) + .join('; '); + return ( + `${census.rootsResolved}/${census.roots} root(s) resolved -- ${per}; ` + + `${census.files} file(s) and ${census.fences} fenced block(s) examined in total; ` + + `${census.outsideFences} occurrence(s) outside a fence (counted, not judged)` + ); +} + +function main() { + const result = scan(repoRoot()); + const { hits, unresolved, vacuous } = result; + + if (hits.length === 0 && unresolved.length === 0 && vacuous.length === 0) { + console.log(`✅ check-shell-escape-residue: OK (${summarise(result)}).`); + process.exit(0); + } + + if (hits.length > 0) { + const plural = hits.length === 1 ? 'fenced block carries' : 'fenced blocks carry'; + console.error(`❌ check-shell-escape-residue: ${hits.length} ${plural} machine-produced shell-escape residue\n`); + console.error(' Copied verbatim, an example carrying this run does not fail with a message --'); + console.error(' it HANGS on a heredoc terminator that never matches. A reader will not'); + console.error(' attribute that to the document (objectui#5150 / objectui#5151):\n'); + for (const hit of hits) { + const pattern = RESIDUE_PATTERNS.find((p) => p.id === hit.patternId); + console.error(` - ${hit.file}:${hit.line}:${hit.column} -- ${pattern.what}`); + console.error(` in the \`${hit.language || '(no info string)'}\` fence opened at line ${hit.fenceLine}:`); + console.error(` ${hit.text}`); + } + const remedies = [...new Set(hits.map((h) => RESIDUE_PATTERNS.find((p) => p.id === h.patternId).remedy))]; + console.error(`\n${remedies.map((r) => ` ${r}`).join('\n\n')}`); + console.error(` +⚠️ AGENTS.md, CLAUDE.md and skills/** are GOVERNED SURFACE. A finding in one of +those is for a human to fix in its own change -- report it, do not fold the fix +into an unrelated pull request. A finding under content/docs/** is an ordinary +docs fix. + +⛔ This gate checks an enumerated literal. It does NOT check that fenced shell +examples are executable -- nothing does. See this script's header.`); + } + + if (unresolved.length > 0) { + console.error('\n❌ check-shell-escape-residue: a declared scan root did not resolve\n'); + for (const u of unresolved) console.error(` - ${u.spec} (declared ${u.kind}) ${u.problem}`); + console.error(` +A root that is gone is reported rather than skipped, because a MISTYPED root and +a CLEAN root produce identical output otherwise -- and the mistyped one reads as +coverage for as long as nobody checks. If the file genuinely moved, move it in +\`SCAN_ROOTS\` in the same change.`); + } + + if (vacuous.length > 0) { + console.error('\n❌ check-shell-escape-residue: the population COLLAPSED -- this run proves nothing\n'); + for (const v of vacuous) console.error(` - ${v.what}: found ${v.value}, floor is ${v.floor}`); + console.error(` +This gate is GREEN AT REST -- there is nothing to find on an ordinary day -- so a +scan that silently examined nothing is indistinguishable from a passing one. That +is objectui#5151's own defect, one level up, so it is a FAILURE here instead. + +Something upstream of the judgement broke: a root moved, the document walk +stopped matching \`.md\`/\`.mdx\`, or the fence walker in +\`check-doc-fence-languages.mjs\` changed shape. Fix the walk. If a floor is +genuinely too high because the tree changed, move it in \`SCAN_ROOTS\` / +\`FENCE_FLOOR\` deliberately and say why -- never to make a red run green. + +Census: ${summarise(result)}`); + } + + process.exit(1); +} + +// Run only when invoked directly -- the test suite imports `scan` and friends +// and must not trigger a repo scan (or a `process.exit`) on import. +if (isEntrypoint(import.meta.url)) { + if (process.argv.includes('--json')) { + const result = scan(repoRoot()); + console.log(JSON.stringify({ census: result.census, hits: result.hits, unresolved: result.unresolved, vacuous: result.vacuous }, null, 2)); + } else if (process.argv.includes('--list')) { + const result = scan(repoRoot()); + for (const r of result.census.perRoot) { + console.log(`${r.resolved ? 'ok ' : 'UNRESOLVED'} ${r.spec.padEnd(16)} ${r.files} file(s), ${r.fences} fence(s)`); + } + for (const hit of result.hits) console.log(`RESIDUE ${hit.file}:${hit.line}:${hit.column} ${hit.patternId}`); + console.log(`\n${summarise(result)}`); + } else { + main(); + } +} diff --git a/scripts/dependabot-merge-gate.mjs b/scripts/dependabot-merge-gate.mjs index 5f5de3a605..756e31f378 100644 --- a/scripts/dependabot-merge-gate.mjs +++ b/scripts/dependabot-merge-gate.mjs @@ -128,6 +128,7 @@ import { isEntrypoint } from './invoked-as.mjs'; * doc-fence-languages.yml Doc Fence Language Check * pre-install-import-graph.yml Pre-Install Import Graph Check * vi-mock-specifiers.yml Inert vi.mock Specifier Check + * shell-escape-residue.yml Shell Escape Residue Scan * * The four shards are spelled out individually on purpose. A single `Test` * entry, or any pattern match, would be satisfied by whichever shard happened @@ -152,6 +153,7 @@ export const REQUIRED_CONTEXTS = Object.freeze([ 'Doc Fence Language Check', 'Pre-Install Import Graph Check', 'Inert vi.mock Specifier Check', + 'Shell Escape Residue Scan', ]); /**