From 478b817e6391f3325e0c0c32db3ed9920b737a44 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 24 Aug 2026 12:05:23 +0000 Subject: [PATCH] fix(spec): make check:skill-examples' marked-block extraction fence-aware The orphan-marker scan learned to ignore an `os:check` marker shown as example text inside a wrapping fence, so this gate's own convention could be documented in the roots it governs. The extraction loop was deliberately left out of that fix and kept recognising a bare ```ts / ```tsx / ```typescript fence-open line wherever it appeared, with no notion of sitting inside another fence. A marker ALONE nested in an illustration was therefore handled correctly, while a fully worked one -- marker AND a real ts fence, both example text inside a wrapper -- was extracted and handed to tsc as a genuine example: compiling by luck, or failing the whole gate against documentation prose. Both loops now read one `fenceOwners()` walk, which records per line which top-level fence owns it: `owners[i] >= 0` answers the orphan scan's question, `owners[i] === i` answers extraction's. The extraction loop's fence-open pattern is a strict subset of the CommonMark one the walk uses, so a genuine top-level block always owns itself and the guard cannot suppress one. No corpus occurrence trips this, so the counts are identical either side of the fix (257 marked examples across 99 files, three surfaces). The self-test is where the defect is measurable: a nested worked illustration with deliberately uncompilable payloads extracts nothing, while the identical payloads with the wrapper removed all extract -- and the same pair is pinned inside a JSDoc gutter-wrapped docblock, where ownership must be judged on stripped lines. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Rxnd8cyFnoU8V5y21PaTsy --- .../skill-examples-extraction-fence-aware.md | 27 ++ packages/spec/scripts/check-skill-examples.ts | 286 +++++++++++++++--- 2 files changed, 279 insertions(+), 34 deletions(-) create mode 100644 .changeset/skill-examples-extraction-fence-aware.md diff --git a/.changeset/skill-examples-extraction-fence-aware.md b/.changeset/skill-examples-extraction-fence-aware.md new file mode 100644 index 0000000000..82a6e173e2 --- /dev/null +++ b/.changeset/skill-examples-extraction-fence-aware.md @@ -0,0 +1,27 @@ +--- +'@objectstack/spec': patch +--- + +`check:skill-examples`: make the marked-block EXTRACTION loop fence-aware, not just the orphan scan + +The orphan-marker scan learned to ignore an `os:check` marker shown as example text inside a +wrapping fence, so this gate's own convention could be documented in the very roots it governs. +The extraction loop was deliberately left out of that fix, and kept recognising a bare ` ```ts ` / +` ```tsx ` / ` ```typescript ` fence-open line wherever it appeared, with no notion of sitting +inside another fence. A marker *alone* nested in an illustration was therefore handled correctly, +while a **fully worked** one — the marker AND a real ts fence, both written as example text inside +a wrapper — was extracted and handed to `tsc` as a genuine example: compiling by luck, or failing +the whole gate with a diagnostic pointing at documentation prose. + +Both loops now read one `fenceOwners()` walk. It records, per line, which top-level fence owns it, +so `owners[i] >= 0` answers the orphan scan's question ("is this marker inside a fence?") and +`owners[i] === i` answers extraction's ("does this ts fence open at top level?"). Because the +extraction loop's fence-open pattern is a strict subset of the CommonMark one the walk uses, a +genuine top-level block always owns itself and the new guard cannot suppress one. + +No occurrence in the corpus tripped this, so the counts are unchanged either side of the fix (257 +marked examples across 99 files, three surfaces). The self-test is where the defect is measurable: +a nested worked illustration whose payloads are deliberately uncompilable now extracts nothing, +while the identical payloads with the wrapper removed all extract — and the same pair is pinned +inside a JSDoc-gutter-wrapped docblock, where fence ownership has to be judged on the +gutter-stripped lines. diff --git a/packages/spec/scripts/check-skill-examples.ts b/packages/spec/scripts/check-skill-examples.ts index 863c3cddc3..9d1a66730b 100644 --- a/packages/spec/scripts/check-skill-examples.ts +++ b/packages/spec/scripts/check-skill-examples.ts @@ -166,17 +166,32 @@ * real consumer would, rather than relying on a same-package self-import * trick that may or may not resolve. * - * ── Fence-awareness in the orphan scan ──────────────────────────────────── - * The orphan-marker guard (`extractFromFile`) is fence-aware: a marker - * spelling shown as example text INSIDE some other fenced block (e.g. a - * ```md illustration showing what `` looks like) is not - * adjacent, at top level, to a real fence, so it claims nothing and must not - * be reported as a misplaced marker either — otherwise this gate's own - * convention could never be documented in the roots it governs. `fenceSpans()` - * tracks every top-level fence of ANY language (lifted from the #10533 - * fence-awareness shape in `scripts/check-role-word.mjs`); a marker at true - * top level that is merely not adjacent to its fence is unaffected and still - * fails loudly. + * ── Fence-awareness, in BOTH of extractFromFile's loops ─────────────────── + * `fenceOwners()` tracks every top-level fence of ANY language (lifted from + * the #10533 fence-awareness shape in `scripts/check-role-word.mjs`), and both + * loops read it, because this gate's own convention has to be documentable in + * the very roots it governs: + * + * - ORPHAN scan (#10791): a marker spelling shown as example text INSIDE + * some other fenced block (e.g. a ```md illustration showing what + * `` looks like) is not adjacent, at top level, to a real + * fence, so it claims nothing and must not be reported as a misplaced + * marker either. A marker at true top level that is merely not adjacent to + * its fence is unaffected and still fails loudly. + * - EXTRACTION (#11355): the same reasoning one step further in. #10791 + * deliberately left extraction alone (its card scoped the fix to the + * orphan-scan defect), so a bare ` ```ts ` / ` ```tsx ` / + * ` ```typescript ` fence-open line was still recognised wherever it + * appeared, with no notion of sitting inside a wrapping fence. A marker + * ALONE nested in an illustration was therefore handled correctly, while a + * FULLY worked one — marker and real ts fence, both written as example + * text inside the wrapper — was extracted and handed to `tsc` as a genuine + * example, to compile by luck or to fail the whole gate against + * documentation prose. No such occurrence existed in the corpus when this + * was found (extraction counts are identical either side of the fix), so + * the self-test's nested-illustration fixture is where the defect is + * measurable at all: nested, the worked illustration extracts nothing; + * un-nested, the identical payload extracts every block. * * Usage: * tsx scripts/check-skill-examples.ts # extract + type-check (CI) @@ -461,20 +476,46 @@ const JSDOC_GUTTER_RE = /^\s*\*\s?/; const ANY_FENCE_OPEN_RE = /^ {0,3}(`{3,})([^`]*)$/; /** - * Every line index that lies INSIDE some top-level fenced block — of ANY - * language, spanning both its opening and closing fence line. This gate's own - * `os:check` convention has to be documentable in the very roots it governs - * (e.g. a ```` ```md ```` illustration showing the marker's exact spelling), - * and a marker shown as example text inside such a fence claims nothing — it - * is not adjacent to a real fence the author intends to check, so it must not - * be flagged as a misplaced (orphan) marker either. The closing run length - * must match or exceed the opener's (a `````` fence wrapping a ```ts example - * closes on ITS OWN fence, not the inner one), and an unclosed fence runs to - * the end of the document (CommonMark), so it consumes the rest of the file - * rather than leaving the tail ambiguous. + * Which TOP-LEVEL fenced block owns each line — of ANY language, spanning both + * its opening and closing fence line. The value is the index of the line that + * OPENED the block, or `-1` for a line at true top level. A line that opens a + * top-level fence owns itself, which makes the array answer both questions this + * file asks of it, one per loop in `extractFromFile()`: + * + * - `owners[i] >= 0` — "line i is inside some fence". The ORPHAN scan's + * question (#10791): this gate's own `os:check` convention has to be + * documentable in the very roots it governs (e.g. a ```` ```md ```` + * illustration showing the marker's exact spelling), and a marker shown as + * example text inside such a fence claims nothing — it is not adjacent to a + * real fence the author intends to check, so it must not be flagged as a + * misplaced (orphan) marker either. + * - `owners[i] === i` — "line i opens a fence AT TOP LEVEL". The EXTRACTION + * loop's question (#11355): a ` ```ts ` fence-open line written as example + * text inside a wrapping fence is part of an illustration, not a block this + * gate should hand to `tsc`. Extraction was NOT fence-aware until #11355 — + * it matched a bare ts/tsx/typescript fence-open line wherever it appeared, + * so a FULLY worked nested illustration (the marker AND a real ts fence, + * both nested) would have been extracted and compiled as if it were a real + * example, failing the gate with a diagnostic pointing at documentation + * prose. (No occurrence existed in the corpus; the self-test is where the + * defect is measurable.) + * + * The two readings share ONE walk on purpose: two independent notions of "am I + * in a fence" in one file is how the halves drifted apart in the first place. + * + * `FENCE_OPEN_RE` is a strict subset of `ANY_FENCE_OPEN_RE` — three backticks + * plus an info string containing no backtick — so every line the extraction + * loop recognises is one this walk also opens a span on. A genuine top-level ts + * fence therefore always owns itself, and the `owners[i] === i` guard can never + * suppress one. + * + * The closing run length must match or exceed the opener's (a `````` fence + * wrapping a ```ts example closes on ITS OWN fence, not the inner one), and an + * unclosed fence runs to the end of the document (CommonMark), so it consumes + * the rest of the file rather than leaving the tail ambiguous. */ -function fenceSpans(lines: string[]): boolean[] { - const inFence = new Array(lines.length).fill(false); +function fenceOwners(lines: string[]): number[] { + const owners = new Array(lines.length).fill(-1); for (let i = 0; i < lines.length; i++) { const open = ANY_FENCE_OPEN_RE.exec(lines[i]); if (!open) continue; @@ -482,10 +523,10 @@ function fenceSpans(lines: string[]): boolean[] { const closeFence = new RegExp(`^ {0,3}\`{${run},}[ \\t]*$`); let end = i + 1; while (end < lines.length && !closeFence.test(lines[end])) end++; - for (let s = i; s < Math.min(end + 1, lines.length); s++) inFence[s] = true; + for (let s = i; s < Math.min(end + 1, lines.length); s++) owners[s] = i; i = end; } - return inFence; + return owners; } /** @@ -508,18 +549,24 @@ function logicalLines(rawLines: string[], root: SourceRoot): string[] { * fence is exactly the MARKER (ignoring surrounding whitespace) — after * gutter-stripping, for a `commentPrefixed` root. * + * BOTH halves are fence-aware, via one `fenceOwners()` walk. A ts fence-open + * line that is itself example text inside a wrapping fence opens nothing (a + * fully worked nested illustration — marker AND real ts fence — would + * otherwise be extracted and compiled as if it were a real example, #11355), + * and a marker shown inside such a fence claims nothing (#10791) and so is no + * orphan either. The two exclusions are the same fact read twice: the block is + * not, at top level, a real fence the author intends this gate to check. + * * Also reports `orphans`: top-level MARKER lines that are NOT directly above * a ts fence. A misplaced marker (a blank line slipped in between, or it * precedes a bash / json block) silently checks nothing — exactly the * failure mode this gate exists to prevent — so the caller treats an orphan - * as an error, not a no-op. A marker occurrence INSIDE some other fenced - * block (`fenceSpans()`) is example text illustrating the convention, not a - * claim, and is excluded from this scan for the same reason it is excluded - * from extraction above: it is not adjacent, at top level, to a real fence. + * as an error, not a no-op. */ function extractFromFile(source: string, root: SourceRoot): { examples: Example[]; orphans: number[] } { const rawLines = fs.readFileSync(source, 'utf-8').split('\n'); const lines = logicalLines(rawLines, root); + const owners = fenceOwners(lines); const examples: Example[] = []; const claimed = new Set(); // MARKER line indices that opened a real block let n = 0; @@ -527,6 +574,11 @@ function extractFromFile(source: string, root: SourceRoot): { examples: Example[ for (let i = 0; i < lines.length; i++) { const open = lines[i].match(FENCE_OPEN_RE); if (!open) continue; + // Top level only (#11355). A ts fence-open line INSIDE some other fence is + // part of an illustration of this very convention, not a block to compile — + // see `fenceOwners()`. A genuine top-level ts fence owns itself, so this + // guard cannot suppress one. + if (owners[i] !== i) continue; const marked = i > 0 && lines[i - 1].trim() === root.marker; // Find the matching close fence regardless of marking, so `i` advances past // this block and we never treat its body as top-level markdown. @@ -546,13 +598,12 @@ function extractFromFile(source: string, root: SourceRoot): { examples: Example[ i = close; // skip to the close fence } - const inFence = fenceSpans(lines); const orphans: number[] = []; for (let i = 0; i < lines.length; i++) { // Top level only. A marker shown INSIDE some other fenced block (e.g. a // ```md illustration of this very convention) is example text, not a - // claim — see `fenceSpans()`. - if (inFence[i]) continue; + // claim — see `fenceOwners()`. + if (owners[i] >= 0) continue; // Any marker spelling counts as an orphan claim — a wrong-format marker // checks nothing, which is precisely what this guard exists to catch. if (ALL_MARKERS.includes(lines[i].trim()) && !claimed.has(i)) orphans.push(i + 1); // 1-based @@ -1169,6 +1220,171 @@ function selfTest(): never { "silenced real orphan would weaken the gate's hard-error posture", ); + // ── Fence-awareness in EXTRACTION (#11355). The fixture above pins the + // ORPHAN half — a bare marker nested in an illustration. This one is + // the other half, and the one #10791 deliberately left open: a FULLY + // worked illustration, marker AND a real ```ts fence, all of it example + // text inside a wrapping fence. Before #11355 the extraction loop had no + // notion of being inside another fence, so it read the nested fence-open + // as a genuine block, extracted the payload and handed it to `tsc` — + // failing the gate against documentation prose (or, worse, compiling by + // luck and reporting coverage of a block nobody claimed). + // + // The payload is deliberately uncompilable, so a regression is not just + // a count that drifts: the poisoned text reaching an `Example.code` is + // itself the assertion, and downstream it would reach `tsc`. + // + // Both payloads sit AFTER an inner ``` close but before the wrapper's + // own ````` close, so a length-blind closer (one that ended the wrapper + // on the inner fence) surfaces here as a second extracted block rather + // than as silence. + const POISON_A = `const poison: number = 'not a number';`; + const POISON_B = `const alsoPoison: number = 'not a number either';`; + const REAL_CLAIM = 'const real: number = 1;'; + + const nestedWorked = path.join(dir, 'nested-worked.md'); + fs.writeFileSync( + nestedWorked, + [ + '# Worked nested illustration', // 1 + '', // 2 + 'Showing an author how to mark a block — marker and fence alike are', // 3 + 'example text here, not a claim:', // 4 + '', // 5 + '`````md', // 6 ← wrapping fence, five backticks + '', // 7 ← illustrated marker + '```ts', // 8 ← illustrated REAL ts fence-open + POISON_A, // 9 + '```', // 10 + '', // 11 + '', // 12 + '```ts', // 13 + POISON_B, // 14 + '```', // 15 + '`````', // 16 ← the wrapper's own close + '', // 17 + 'And the one real claim, at top level:', // 18 + '', // 19 + '', // 20 + '```ts', // 21 + REAL_CLAIM, // 22 + '```', // 23 + '', + ].join('\n'), + 'utf8', + ); + const nested = extractFromFile(nestedWorked, skillsRoot); + check( + nested.examples.length === 1, + `nested-illustration fixture: extracted ${nested.examples.length} block(s), expected 1 — a worked illustration ` + + 'nested inside a wrapping fence is example text; extracting it hands documentation prose to `tsc`', + ); + check( + nested.examples.every((e) => !e.code.includes('poison') && !e.code.includes('Poison')), + `nested-illustration fixture: an illustrated payload reached an extracted block — ` + + `${JSON.stringify(nested.examples.map((e) => e.code))}`, + ); + check( + nested.orphans.length === 0, + `nested-illustration fixture: reported ${nested.orphans.length} orphan marker(s), expected 0 — the illustrated ` + + 'markers are inside the wrapper, and skipping their fences must not turn them into orphans instead', + ); + if (nested.examples.length === 1) { + const ex = nested.examples[0]; + check( + ex.code === REAL_CLAIM, + `nested-illustration fixture: extracted ${JSON.stringify(ex.code)}, expected ${JSON.stringify(REAL_CLAIM)}`, + ); + check( + ex.bodyStartLine === 22, + `nested-illustration fixture: bodyStartLine ${ex.bodyStartLine}, expected 22 — skipping a wrapper must not ` + + 'shift the line a diagnostic is reported against', + ); + } + + // The CONTROL, and the load-bearing half of this pair: the identical three + // claims with the wrapper's two lines removed. Without it, a guard that + // simply stopped extracting anything would pass the assertions above — the + // #10533 (B3) shape, where the green has to be shown to come from the + // nesting rather than from the payload or the marking. + const unnestedWorked = path.join(dir, 'unnested-worked.md'); + fs.writeFileSync( + unnestedWorked, + [ + '', // 1 + '```ts', // 2 + POISON_A, // 3 + '```', // 4 + '', // 5 + '', // 6 + '```ts', // 7 + POISON_B, // 8 + '```', // 9 + '', // 10 + '', // 11 + '```ts', // 12 + REAL_CLAIM, // 13 + '```', // 14 + '', + ].join('\n'), + 'utf8', + ); + const unnested = extractFromFile(unnestedWorked, skillsRoot); + check( + unnested.examples.length === 3, + `un-nested control: extracted ${unnested.examples.length} block(s), expected 3 — the SAME payloads, minus the ` + + 'wrapping fence, must all extract, or the nested case above proves nothing about nesting', + ); + check( + JSON.stringify(unnested.examples.map((e) => e.bodyStartLine)) === JSON.stringify([3, 8, 13]), + `un-nested control: body start lines ${JSON.stringify(unnested.examples.map((e) => e.bodyStartLine))}, expected [3,8,13]`, + ); + check( + unnested.orphans.length === 0, + `un-nested control: reported ${unnested.orphans.length} orphan marker(s), expected 0`, + ); + + // The same nesting inside a `commentPrefixed` root. The guard reads the + // GUTTER-STRIPPED lines (`logicalLines()`), and nothing else here would + // catch it being computed over the raw ones instead: a raw-line walk never + // recognises ` * `````md ` as a fence, so every nested fence below reads as + // top level again and the defect returns on precisely the roots whose prose + // lives in docblocks. + const nestedDoc = path.join(dir, 'nested-doc.ts'); + fs.writeFileSync( + nestedDoc, + [ + '/**', // 1 + ' * How to mark an example, illustrated:', // 2 + ' *', // 3 + ' * `````md', // 4 + ' * ', // 5 + ' * ```ts', // 6 + ` * ${POISON_A}`, // 7 + ' * ```', // 8 + ' * `````', // 9 + ' *', // 10 + ' * @example', // 11 + ' * ', // 12 + ' * ```ts', // 13 + ` * ${REAL_CLAIM}`, // 14 + ' * ```', // 15 + ' */', // 16 + 'export const Documented = 1;', // 17 + ].join('\n'), + 'utf8', + ); + const nestedGutter = extractFromFile(nestedDoc, specSrcRoot); + check( + nestedGutter.examples.length === 1 && nestedGutter.examples[0].code === REAL_CLAIM, + `gutter-wrapped nested fixture: extracted ${JSON.stringify(nestedGutter.examples.map((e) => e.code))}, expected ` + + `only ${JSON.stringify([REAL_CLAIM])} — fence ownership must be judged on gutter-stripped lines`, + ); + check( + nestedGutter.orphans.length === 0, + `gutter-wrapped nested fixture: reported ${nestedGutter.orphans.length} orphan marker(s), expected 0`, + ); + // ── Build-dir distinctness (#10924). The REAL surfaces are asserted on // every run by `assertDistinctBuildDirs()`; these two fixtures pin the // predicate underneath it in both directions, because a guard that can @@ -1307,7 +1523,9 @@ function selfTest(): never { ' JSDoc-gutter-wrapped ```tsx block (client SDK surface) extracts, strips and maps lines\n' + ' identically, and a misplaced gutter-wrapped marker is still caught as an orphan; a marker\n' + ' shown as example text inside another fenced block is not an orphan, while a genuine\n' + - ' top-level misplaced marker still is; a gutter-wrapped ```ts block (spec-source\n' + + ' top-level misplaced marker still is; a FULLY worked nested illustration (marker AND a\n' + + ' real ```ts fence, wrapper-nested, gutter-wrapped or not) extracts NOTHING while the\n' + + ' identical payloads un-nested all extract; a gutter-wrapped ```ts block (spec-source\n' + ' surface) extracts with the right build extension, body and line mapping, its\n' + ' `.test.ts` sibling is skipped, two surfaces sharing one build dir are caught, a\n' + ' surface whose build dir is not covered by .gitignore is caught too, and a non-git\n' +