Skip to content

fix(spec): check:skill-examples refuses instead of degrading when a marked example does not parse - #12308

Merged
os-litant merged 3 commits into
mainfrom
claude/issue-12051-skill-examples-refuse-on-syntax
Aug 25, 2026
Merged

fix(spec): check:skill-examples refuses instead of degrading when a marked example does not parse#12308
os-litant merged 3 commits into
mainfrom
claude/issue-12051-skill-examples-refuse-on-syntax

Conversation

@os-litant

Copy link
Copy Markdown
Collaborator

Fixes#12051

check:skill-examples derives each block's build-file extension from its fence language (buildFileName), so a marked block whose body is JSX under a ```ts / ```typescript fence is written to a .ts file, where JSX is a syntax error. TypeScript reports syntactic diagnostics and then stops — the semantic pass never runs, for any file in the program. One such block therefore costs the whole surface's type-checking (227 blocks on skills+docs).

Measured correction to the card's headline

The card states the gate "still prints a green verdict" under this condition, and that the answer to "would this gate still be green if its semantic pass never ran?" is yes. Ablated on the real corpus before any change here — one JSX line forced into a marked ```ts block in content/docs/permissions/sso.mdx:

ablation[pre]: injected-marker hits on disk = 1
ablation[pre]: gate exit = 1
✗ [skills + docs (@objectstack/spec)] examples do not compile:
error TS1005: '>' expected.
error TS1161: Unterminated regular expression literal.

The verdict line did change. That green path was never reachable: tsc exits non-zero on a syntactic diagnostic and prints it, and the surface loop only skips a surface on code === 0 && diags.length === 0.

What was real, and is what this PR closes, is the degradation on either side of that red:

  • the failure text read as ordinary type drift — "Fix the example to match the current declarations" — while the truth was that the surface had not been type-checked at all;
  • the other 226 blocks' semantic pass had silently not run, and nothing in the output said so; an author who fixes the reported syntax errors is then meeting the surface's real diagnostics for the first time;
  • nothing enumerated the unmarked JSX-in-ts-fence blocks that become this failure the moment anyone marks them.

So the property closed is not "make a green run red" but: make the gate say which of the two happened, and never report an un-run semantic pass in the vocabulary of a semantic result.

What changed

1. REFUSE, with its own verdict token. Marked blocks are parsed before any build dir is written. If one does not parse, the surface gets a ⛔ REFUSE verdict naming it and the number of blocks left unchecked — distinct from ("produced a result, and it is bad"). Same exit code. When the body parses clean as .tsx and really contains JSX, the refusal names the one-word fix.

The predicate is program.getSyntacticDiagnostics() — precisely what tsc itself uses to decide whether to run the semantic pass — not a "TS1xxx means syntax" code-range heuristic, which would manufacture a REFUSE over the several 1xxx codes the checker reports. One program for the whole population, noLib + noResolve, so a parse-level verdict never depends on build state.

It runs before the bare-any guard, which walks a createSourceFile tree: that call degrades silently on input it cannot parse, so running it over an unparseable block was the dormant-checker shape its own docblock warns about.

2. The unmarked population, swept. Extraction now records every top-level ts/tsx/typescript fence, marked or not. A three-part, zero-false-positive predicate flags JSX under a plain-ts fence: fails to parse as .ts, parses clean as .tsx, and contains an actual JSX node. Prose fragments (defineStack({ ... }), a columns: [...] subtree) fail to parse both ways and are correctly left alone.

3. The green line is structural. A surface only counts as checked once it reaches tsc; the green line asserts that set covers every surface, and says what it means:

✅ 256 prose examples type-check across 3 surface(s) — every marked block parsed, so tsc ran the SEMANTIC pass on all of them

Population sweep — 5 blocks, all unmarked, all under content/docs/**

filefence linemarked?
content/docs/api/error-handling-client.mdx135unmarked
content/docs/api/error-handling-client.mdx307unmarked
content/docs/protocol/objectui/concept.mdx348unmarked
content/docs/protocol/objectui/concept.mdx647unmarked
content/docs/protocol/objectui/index.mdx318unmarked

All five fenced ```typescript, all genuine React/JSX, all retagged ```tsx. skills/** carries none — this PR touches no governed surface. (content/docs/api/client-sdk.mdx, the one instance PR #12045 repaired, is correctly absent.)

check:doc-authoring's FENCE_OPEN cannot carry the cheap prevention half

The card's third open question, answered here rather than forked to devx. scripts/check-doc-authoring.mjs is line-wise: FENCE_OPEN (^```(?:ts|typescript|tsx)\s*$) matches the fence-open line, and the body is then tested one line at a time against a literal regex (BARE.test(ln)). It never parses a block, so it cannot distinguish JSX from any other text, and widening it to carry this check would mean giving a devx-owned lint its own TypeScript parser. The check belongs where the fence language is consumed — where it decides a build-file extension — not where fences are merely recognised. No devx fork, and no widening of that gate.

Ablation — the card's discriminating test, post-fix

Run from the committed state; every leg proves the mutation landed on disk and proves the restore landed. No rebuild leg is claimed or needed: the mutated subject is prose read from disk and the gate runs from source via tsx; nothing here resolves through dist, and packages/spec/src is untouched.

legmutationverdict
Amarked block, JSX body under a ```ts fence⛔ REFUSE — [skills + docs (@objectstack/spec)] was NOT type-checked. · 226 marked block(s) on this surface were not type-checked either · ↳ This body is JSX and parses clean as .tsx … Retag the fence ```tsx
Bmarked block, ordinary syntax error, no JSXsame REFUSE, no tsx prescription — the discrimination holds
Cunmarked JSX under a ```typescript fence✗ JSX inside a ```ts / ```typescript fence · content/docs/permissions/sso.mdx:158 (```typescript, unmarked)
 injected-text hits on disk: 1 (must be >=1)
gate exit: 1
after restore, injected-text hits: 0 (must be 0)
═══ final tree state ═══
(empty above = tree fully restored)

Answer to "would this gate still be green if its semantic pass never ran?"no, and the verdict now says which of the two failures occurred.

Self-test

Six fixtures added, both directions pinned: a JSX body under a ```ts fence is flagged whether marked or not; the identical body under a ```tsx fence is not; a prose ellipsis fragment (parses as neither) is not; a marked block that does not parse REFUSES with its first error mapped to the right page line; a marked non-JSX parse failure refuses without recommending a tsx retag; and the bytes the refusal parses are the bytes the build dir is written with (buildFileText, now shared by both).

Verification — all at 516420e

Gate union re-derived from the actual diff with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack (no hand-built path list), run after the final commit.

Green: check:skill-examples (self-test + real corpus, 256/227 — unchanged from the pre-change baseline) · check:docs · check:doc-authoring · check:doc-anchors · check:doc-formula-expressions · check:doc-security-posture · check:docs-audit-scope · check:docs-redirects · check-doc-frontmatter · check-doc-route-spelling · check-docs-section-name · check-section-landing-index · check:role-word · check:page-declaration-shape · check:react-page-adapter-contract · check:cross-package-test-inputs (both spellings) · check-ci-filter-parity · check:merge-driver · check:published-files · check:published-readme-links · check:slot-lookup · check:test-source-alias · check:type-source-resolution · check-plugin-teardown-shape · check:empty-state · check:liveness · check:strictness-ledger · check:variant-docs · check:nul-bytes · check:agent-test-spelling · check:skill-compatibility · check:skill-frame-sync · check:pm-governed-merges · docs-audit/check-affected-docs · docs-audit/check-drift-comment · pnpm --filter @objectstack/spec typecheck · check:generated (14/14 up to date, after merging origin/main and rebuilding).

One red, attributed rather than asserted: check:dev-prereqs reports 34 of 67 workspace packages missing a dist/ entry point. It is diff-independent — the script reads disk only (it contains no merge-base, git diff or base-ref read), this branch changes 0 files under packages/apps, packages/plugins or packages/services, and the same dist/ directories are absent in this container's pristine main checkout, which carries none of this diff. It measures "is the workspace built", a container fact; CI builds the workspace before running it.

Changeset

skip-changeset. This PR releases nothing: packages/spec's published files array is dist, json-schema, liveness, prompts, llms.txt, README.md, src/**/*.zod.ts, CHANGELOG.md, api-surface, spec-changes.jsonscripts/ is not among them — and the other three files are docs-site prose. Same disposition as #12175 for the same script.


Draft, and stays draft until a human merges it — not because of a governed surface (there is none in this diff; skills/** is untouched) but because the sweep's completeness is a judgement about prose the maintainer may want to read.

Generated by Claude Code


Generated by Claude Code

… parse
check:skill-examples derives each block's build-file extension from its fence
language, so a marked block whose body is JSX under a ```ts / ```typescript
fence lands in a .ts file where JSX is a syntax error. TypeScript reports
syntactic diagnostics and then stops, so the semantic pass never runs -- for any
file in the program. The run did go red (measured), but in the vocabulary of a
semantic result, over a surface whose semantic pass had not run, and nothing in
the output said so.
Two guards separate those states. Marked blocks that do not parse now produce a
REFUSE verdict naming the surface and the number of blocks left unchecked, with
a tsx-retag prescription when the body really is JSX; the unmarked
JSX-under-a-ts-fence population is swept before anyone arms it. The green line
is printed only when every surface reached tsc, and says so.
Sweep repairs: five unmarked JSX blocks fenced ```typescript across
content/docs/** retagged ```tsx. skills/** carries none.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NDGG54XF5gbTLdQzCtnaVV
…iled check
`✗ ⛔ REFUSE` stacked two prefixes and read as one emphatic failure rather than
as a different KIND of failure. A refusal ("this gate produced no result") and a
failure ("this gate produced a result and it is bad") now print two tokens a
reader can tell apart and grep for. Same exit code: a refusal is still red.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NDGG54XF5gbTLdQzCtnaVV
@os-litantos-litant added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Aug 25, 2026 — with Claude
@github-actionsgithub-actionsBot added documentation Improvements or additions to documentation tooling labels Aug 25, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

Nothing in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 0 changed package(s)), so this run has no opinion about the docs.

What this run could not see
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 0 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 5ce5f8c12479f40d548b9f3ba9f8725c14c9bf9apackageMentionDocs.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/lskip-changesetPR has no user-facing published change; bypasses the changeset gatetooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[finding] One JSX block fenced typescript silently disables the SEMANTIC pass for all 227 marked examples — check:skill-examples still reports green

2 participants

@os-litant@claude