Uh oh!
There was an error while loading. Please reload this page.
fix(spec): regenerate drifted reference docs and gate them in CI - #3134
Merged
Conversation
…eSelfInfo, ADR-0097) content/docs/references/ is generated from packages/spec by `gen:schema && gen:docs`, but nothing in CI regenerates and diffs it, so the output has been drifting since #3014 (2026-07-16). Regenerating dirties 7 files; every change traces to a spec change that shipped without re-running the generator: - data/object.mdx RowCrudActionOverride missing entirely (#3076) — the type is exported from @objectstack/spec/data but undiscoverable in docs - api/discovery.mdx ServiceSelfInfo missing entirely - ui/dashboard.mdx ChartWidget.filterBindings, DashboardFilter.name missing - data/field.mdx readonly: server-enforced on UPDATE (#2948/#3003) - security/permission allowTransfer: enforced now, was "pending M2" (#3004) - integration/* connector ADR-0096 → ADR-0097, expanded providerConfig prose The ADR renumber is a correctness fix, not churn: 0096 is execution-surface identity admission, 0097 is declarative connector instances (#2977). The published docs have been pointing readers at the wrong ADR. Generated output only — no hand edits. Verified deterministic: two consecutive runs produce byte-identical output, so the drift is real rather than generator noise. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…enerated gates content/docs/references/** is generated from packages/spec and committed, but no CI job ever regenerated and diffed it — `grep -rlE "gen:docs|gen:schema" .github/workflows/` returned nothing. So the public reference docs drifted silently while main stayed green: #3076 added RowCrudActionOverride to the spec and the docs never learned the type existed. The parent commit regenerates; this one stops it recurring. build-docs.ts --check Follows the sibling convention (check:skill-docs / check:api-surface / …). Every write now goes through emit() and every wiped folder through manageDir(), so both modes run identical generation logic and differ only in the final disposition — write, or compare. That shared path is what makes the gate trustworthy: it cannot pass on output a real run would not produce. Verified output-identical to the previous generator (byte-for-byte across all 258 files). Catches all three drift classes, each proven to fail before being trusted: stale content (the #3076 case), a missing page, and a page left behind by a type removed from spec. The last two are why this is not `git diff --exit-code`: that misses untracked files. json-schema/ is gitignored, so `check:docs` runs gen:schema first and a run that finds no schemas errors rather than passing vacuously. Placement In lint.yml's "TypeScript Type Check", not ci.yml's "Build Docs". Build Docs is gated on a `docs` paths-filter that excludes packages/spec/**, so it skips exactly the spec-only PRs that cause this drift — #3076 was one. The typecheck job has no paths filter and is a required status check, so the gate cannot go dormant. It reads src/ + json-schema/ via tsx and needs no build, so it runs before the workspace build and fails in ~2s. Same hole found in the sibling job (fixed here, both currently in sync but unguarded): - check:spec-changes / check:upgrade-guide read the ADR-0087 registries and the protocol version, but ran under a filter listing only skills/** — a protocol bump or new migration could not trigger the job that verifies them. - the filter watched content/docs/guides/skills.mdx, a path #2584 moved; hand-edits to the real generated block in content/docs/ai/skills-reference.mdx went unchecked. Job renamed check-skill-docs → check-generated: it has guarded the protocol artifacts for a while, and a name that hid them is part of why their paths went missing from the filter. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
📓 Docs Drift CheckThis PR changes 1 package(s): 102 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:
|
Uh oh!
There was an error while loading. Please reload this page.
xuyushun441-sys pushed a commit
that referenced
this pull request
Jul 17, 2026
The 243 generated reference pages carried ~92 links to a page that never
existed (`[__schema0](./__schema0)`), 6 self-links to `(./#)`, and every
inline object rendered as an opaque `Object`. Root causes, all in
build-docs.ts:
- `formatType` turned `$ref` into `./<schemaName>`, but pages are named
after the *zod file* — so no `$ref` link could ever resolve. Named refs
now resolve through the schemaCategoryMap/schemaZodFileMap that
scanCategories() already built (dead code until now) to
`/docs/references/<category>/<file>#<schema>`, and fall back to plain
text rather than a 404 when no page exists.
- `__schemaN` refs are Zod-hoisted inline schemas with no page at all;
they are now expanded structurally, with a cycle guard (these schemas
are recursive — a node contains nodes — and naive inlining blows the
stack).
- `$ref: "#"` (self-reference) now links to the section's own anchor.
- Inline objects render their shape one level deep
(`{ label: string; value: string; color?: string; … }`) instead of
`Object`; const variants render their literal, so discriminated unions
read as `'a' | 'b'` rather than `Object | Object`.
- JSDoc `{@link ../automation/sync.zod.ts}` linked to a repo path that
404s on the site; source paths now rewrite to their docs route, and a
bare `@see` renders as "See also:" prose instead of a stray tag.
- Property type cells escape backslashes before pipes, matching the
neighbouring `desc` escaping (and its comment) — CodeQL's
js/incomplete-sanitization on the old order.
Counts after regenerating: __schemaN links 92 → 0, (./#) 6 → 0,
zod.ts links → 0. Passes the new `--check` gate from #3134.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>os-zhuang added a commit
that referenced
this pull request
Jul 17, 2026
) The 243 generated reference pages carried ~92 links to a page that never existed (`[__schema0](./__schema0)`), 6 self-links to `(./#)`, and every inline object rendered as an opaque `Object`. Root causes, all in build-docs.ts: - `formatType` turned `$ref` into `./<schemaName>`, but pages are named after the *zod file* — so no `$ref` link could ever resolve. Named refs now resolve through the schemaCategoryMap/schemaZodFileMap that scanCategories() already built (dead code until now) to `/docs/references/<category>/<file>#<schema>`, and fall back to plain text rather than a 404 when no page exists. - `__schemaN` refs are Zod-hoisted inline schemas with no page at all; they are now expanded structurally, with a cycle guard (these schemas are recursive — a node contains nodes — and naive inlining blows the stack). - `$ref: "#"` (self-reference) now links to the section's own anchor. - Inline objects render their shape one level deep (`{ label: string; value: string; color?: string; … }`) instead of `Object`; const variants render their literal, so discriminated unions read as `'a' | 'b'` rather than `Object | Object`. - JSDoc `{@link ../automation/sync.zod.ts}` linked to a repo path that 404s on the site; source paths now rewrite to their docs route, and a bare `@see` renders as "See also:" prose instead of a stray tag. - Property type cells escape backslashes before pipes, matching the neighbouring `desc` escaping (and its comment) — CodeQL's js/incomplete-sanitization on the old order. Counts after regenerating: __schemaN links 92 → 0, (./#) 6 → 0, zod.ts links → 0. Passes the new `--check` gate from #3134. Co-authored-by: Jack Zhuang <277994282+os-zhuang@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
xuyushun441-sys pushed a commit
that referenced
this pull request
Jul 17, 2026
`skills/` is generated from the spec by `gen:skill-refs` and `gen:react-blocks` and committed, but neither had a `check:` sibling and nothing in CI regenerated them. Both existing gates miss this surface: `check:docs` (#3134) covers only content/docs/references/, and `check:skill-docs` only SKILL.md frontmatter → README + skills-reference.mdx. So it drifted — 8 committed files were stale on main. This matters more than the content/docs case: `skills/` is published to third parties via `npx skills add objectstack-ai/framework --all`, so stale output ships to users. Fix the drift's root cause, not the output ------------------------------------------ Two published skills pointed agents at `data/dataset.zod.ts`, which #1620 renamed to `data/seed.zod.ts`. The generator warned and skipped, staying green, so objectstack-data and objectstack-platform silently lost their seed schema pointer and told agents to Read a path that no longer exists. SKILL_MAP now names seed.zod.ts, and a missing mapped file is a hard error instead of a warning — that half-rename is exactly what warn-and-skip is good at hiding. Add --check to both generators ------------------------------ Following the convention #3134 established in build-docs.ts: every write goes through emit(), every regenerated folder through manageDir(), and nothing touches the tree until flush() — so check and write share byte-for-byte identical generation logic and differ only in final disposition. Not `git diff --exit-code`, which sees only tracked files and would miss a new or leftover page. The sink is extracted to scripts/lib/generated-output.ts since both generators need it; manageDir() takes a predicate because a skill's references/ folder is a selective wipe, not a wholesale one — hand-written siblings (data-hooks.md, plugin-hooks.md) and the other generator's react-blocks.md live there and must survive. Wire into "TypeScript Type Check", not ESLint --------------------------------------------- main requires exactly 4 status checks (TypeScript Type Check, Build Core, Test Core, Dogfood Regression Gate). ESLint is not one of them, so a gate placed there runs but cannot block a merge. Same job as check:docs, same reason: no paths filter, required, and both read src/ via tsx with no build (~2s). Proven able to go red before shipping: tampered content, deleted file, leftover file, leftover in a sub-folder, stale SKILL_MAP entry, and both vacuous cases (empty SKILL_MAP / empty REACT_BLOCKS) each exit 1; a hand-written .md dropped in references/ stays green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
os-zhuang added a commit
that referenced
this pull request
Jul 17, 2026
…t, and gate them in CI (#3138) skills/*/references/_index.md and the objectstack-ui react-blocks contract are generated from packages/spec/src and committed, but nothing regenerated them in CI — no workflow referenced either generator, the spec `build` script does not run them, and neither had a `check:` mode. They drifted. Unlike the reference docs (#3134), these ship to third parties via `npx skills add`, so the drift is served straight to consumers' agents. An independent oracle over the committed indexes: 6 of 113 schema pointers named files that no longer exist — automation/workflow.zod.ts (deleted by #1398), system/masking.zod.ts (deleted), data/dataset.zod.ts (renamed to data/seed.zod.ts by #1620). After this change: 126 pointers, 0 broken. Regenerating alone was not enough. SKILL_MAP still named data/dataset.zod.ts, and the generator warned-and-skipped on the missing file, so a plain regen would have quietly dropped seed coverage from objectstack-data — whose own SKILL.md advertises defineSeed() authoring — and the new gate would then have locked that in as correct. Point the map at data/seed.zod.ts and make a dangling entry a hard failure in both modes. Gate mechanics follow build-docs.ts (#3134): every write goes through emit(), every wholesale-regenerated folder through manageDir(), nothing touches disk until flush() — so write and --check share all generation logic and differ only in disposition. Extracted to scripts/lib/generated-output.ts. manageDir() takes an ownership predicate because a skill's references/ folder is only partially generated: _index.md is ours, but react-blocks.md (sibling generator) and hand-written notes must survive. Placed in lint.yml's "TypeScript Type Check" (no paths filter + required), not ci.yml's check-generated: that job's filter lists specific spec paths but no schema dirs, so a PR touching src/data/** or src/ui/** never triggers it, and it is not required either. Both gates need no build, so they run before the workspace build and fail in seconds. Each gate was proven to go red before being trusted (13/13): stale content, missing file, stale untracked leftover, dangling SKILL_MAP entry, missing skill dir, and a spec change the contract had not absorbed. The stale-leftover case is why manageDir exists — git diff --exit-code returns 0 on it. Follow-up: #3139 (import regex is blind to multi-line imports). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Aug 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
content/docs/references/**is generated frompackages/specand committed, but no CI job ever regenerated and diffed it:So the public reference docs drift silently while
mainstays green. They have been drifting since #3014 (2026-07-16).The clearest symptom:
RowCrudActionOverrideis entirely missing from the docs. It is a real exported type (@objectstack/spec/data, added by #3076 on 2026-07-17), but it appears in neither the TypeScript-usage import line nor as its own properties section — users cannot discover it at all.1. Regenerate (
2b5ced2c9)gen:schema && gen:docsdirties 7 files. Every change traces to a spec change that shipped without re-running the generator:data/object.mdxRowCrudActionOverridemissing entirely (#3076)api/discovery.mdxServiceSelfInfomissing entirelyui/dashboard.mdxChartWidget.filterBindings,DashboardFilter.namemissingdata/field.mdxreadonly— now server-enforced on UPDATE (#2948/#3003)security/permission.mdxallowTransfer— enforced now, docs still said "pending M2" (#3004)integration/{connector,misc}.mdxproviderConfigproseThe ADR renumber is a correctness fix, not churn: ADR-0096 is execution-surface identity admission, ADR-0097 is declarative connector instances (#2977). Both ADRs exist and are distinct — the published docs have been pointing readers at the wrong one.
No hand edits. Verified deterministic: two consecutive runs produce byte-identical output, so the drift is real rather than generator noise (a nondeterministic generator would have made the gate below unusable).
2. Gate it (
5a4667236)build-docs.ts --check, following the sibling convention (check:skill-docs,check:api-surface, …).Every write now goes through
emit()and every wiped folder throughmanageDir(), so write and check run identical generation logic and differ only in the final disposition — write to disk, or compare against it. That shared path is what makes the gate trustworthy: it cannot pass on output a real run would not produce. Verified output-identical to the previous generator, byte-for-byte across all 258 files.Proven to fail before being trusted
A gate that cannot go red is worse than no gate, so each drift class was reproduced:
~ data/object.mdx (out of date)+ data/field.mdx (missing — spec adds it)- data/ghost-type.mdx (stale — …)json-schema/absentThe last two are why this is not
git diff --exit-codeas originally suggested: that misses untracked files, so a brand-new page would slip through.json-schema/is gitignored, socheck:docsrunsgen:schemafirst, and a run that finds no schemas errors rather than passing vacuously.Placement matters
In lint.yml's "TypeScript Type Check", not ci.yml's "Build Docs":
docspaths-filter that excludespackages/spec/**— it skips exactly the spec-only PRs that cause this drift (feat(spec): userActions.edit/delete accept per-record CEL predicates (objectui#2614) #3076 was one). Putting the gate there leaves it dormant.TypeScript Type Check,Build Core,Test Core,Dogfood Regression Gate) — so the gate both always fires and actually blocks merge. A new standalone job would not be required, and would be advisory only.src/+json-schema/via tsx), so it runs before the workspace build and fails in ~2s.3. Same hole in the sibling job
Both artifacts are currently in sync, but were unguarded on the PRs that can break them:
check:spec-changes/check:upgrade-guideread the ADR-0087 registries and protocol version, but ran under a filter listing onlyskills/**— a protocol bump or new migration could not trigger the job that verifies them.content/docs/guides/skills.mdx, a path docs(site): restructure IA into module-first single sidebar #2584 moved. Hand-edits to the real generated block incontent/docs/ai/skills-reference.mdxwent unchecked from then until now.Job renamed
check-skill-docs→check-generated(safe — not a required check, referenced only within ci.yml): it has guarded the protocol artifacts for a while, and a name that hid them is part of why their paths went missing from the filter.check:api-surfacewas audited and is fine — lint.yml has no paths filter, so it already runs on every PR.Verification
tsc --noEmit,check:docs, and eslint all green locallygeneratedpaths-filter block was parsed as YAML exactly the waydorny/paths-filterparses it, to confirm all 12 paths landpackages/spec, so the new step exercises itself — see "TypeScript Type Check" below🤖 Generated with Claude Code