Skip to content

docs(audits): measure zod to JSON Schema conversion fidelity for @object-ui/types - #6203

Merged
yinlianghui merged 3 commits into
mainfrom
claude/issue-5392-json-schema-fidelity-measurement
Aug 25, 2026
Merged

docs(audits): measure zod to JSON Schema conversion fidelity for @object-ui/types#6203
yinlianghui merged 3 commits into
mainfrom
claude/issue-5392-json-schema-fidelity-measurement

Conversation

@yinlianghui

Copy link
Copy Markdown
Collaborator

Part of #5392

Report-only. This PR adds one document under docs/audits/ and changes nothing
executable: no generator, no drift gate, no published artifact, no edit to
packages/types.

The 2026-08-22 ruling on #5392 ordered a measurement before the A / B / C fork is
decided. ⛔ This PR does not decide it — it reports what the numbers say.

Measured on

origin/main @ 50d0d6cc5, merged into this branch as 12dcdc110; the report commit
is b149c2b14. This base contains#6117 (105f3c55c, CRUDSchema retired), so the
census excludes that node type. An earlier run of this measurement was taken before that
landed; it is superseded and the delta is stated in the report.

Answers

QuestionAnswer
Converterz.toJSONSchema(), built in to zod 4.4.3 — already a dependency
PopulationAnyComponentSchema — 14 union members, 105 leaf node schemas
Faithful conversions0 / 105
Degraded105 / 105 (all open; 5 of 5 refinement predicates silently dropped)
Failed105 / 105 under default options, 0 / 105 lenient
Artifact size651,908 B pretty · 369,028 minified — +21.5 % / +12.2 % of published dist
Closed?No. 276 of 333 objects open; .strict() appears 0 times in the mirrors
Stricter than zod anywhere?No — 0 / 475 corpus files. It is a strict relaxation

The decisive result.BaseSchema is .passthrough(), so every one of the 105 node
types projects additionalProperties to an accept-anything schema. A misspelled real key
(classNmae for className) is accepted by the artifact exactly as it is by zod today.
The artifact would carry properties, required and a const on every type — it
reads closed and is not. The card's own hypothesis, measured rather than predicted.

The contrast that proves the cause is the source, not the converter.
@objectstack/spec — already a dependency — ships generated JSON Schema today
(json-schema/, 23.4 MB, 156 UI documents). Its sources are strictObject(...), and
118 of its 124 object-rooted UI documents close at the root. Same class of pipeline,
opposite closedness.

Which fork the measurement points to

B, with C untouched by these numbers and A blocked on #5155 rather than
refuted forever. safeValidateSchema already delivers everything the artifact would and
delivers it better: 97.1 % agreement over the corpus, never worse, better on 14 files.
C's value is completion, not rejection, and every degradation measured here leaves
suggestions intact. ⛔ The choice remains the maintainer's — see the report's four-way
write-up.

Verification

Numbers, not behaviour, so verification is that they are reproducible and non-vacuous.
Every count in the report carries the command that produced it, and every zero sits beside
a non-zero from the identical instrument (z.bigint( = 0 next to z.string( = 689;
{type:'module'} rejected next to {type:'button'} accepted; z.strictObject projecting
additionalProperties: false in the same run in which no ObjectUI node does). All five
refinement sites were isolated with controls that pass — none inferred. The report has its
own "what this measurement cannot see" section.

⛔ No ablation: there is no behaviour to ablate on a report-only diff. The measurement
scripts were written outside the repository and are deliberately not committed
committing one would be the generator this card is not ruled to build. git status is
clean apart from the report.

Gates

Report-only diff, so most of the farm has nothing to judge. ⛔ No green is claimed from a
gate that did not look.

GateResult
scripts/check-changeset-presence.mjsgreen, its own line: ✅ No source of a released package changed in this range, so no changeset is owed.
scripts/check-control-bytes.mjsgreen, its own line: ✅ check-control-bytes: OK (scanned 5122 tracked text file(s); skipped 85 binary).
pnpm --filter @object-ui/types buildexit 0 — run because a stale dist/*.d.ts would corrupt the measurement in both directions, not because this diff needs it
lint / typecheck / test farmnot applicable — no source file changed

No changeset: the gate above states none is owed. This repo has no skip-changeset
label, and none was created.

Out of scope, already filed

#6124 — 28 zod-mirror keys declared z.function(), which no JSON document can satisfy;
14 in-repo corpus files author them and are invalid today while sitting outside
objectui check's judgement surface. Filed unassigned, ⛔ not touched here.


Generated by Claude Code

…ect-ui/types
The measurement objectui#5392's 2026-08-22 ruling required before the A/B/C
fork is decided. No generator, no drift gate, no published artifact.
Converter: zod 4.4.3's built-in z.toJSONSchema (already a dependency).
Population: AnyComponentSchema — 14 union members, 106 leaf node schemas.
Headline: the generated artifact is a strict relaxation of the zod source.
Over 475 root-type JSON files it agrees with safeValidateSchema on 461
(97.1%), is weaker on 14, and is stricter on none. 0/106 node types convert
under default options; 106/106 convert lenient; 0/106 faithful. 285 of 342
objects are open because BaseSchema is .passthrough() and .strict() is
called zero times, so the artifact reads closed and is not. Size: 666,321
bytes pretty / 376,928 minified, or +21.8% / +12.3% of the published dist.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CSoz9uGhaaSgiq3hshtN7L
…base
The measurement objectui#5392's 2026-08-22 ruling required before the A/B/C
fork is decided. No generator, no drift gate, no published artifact.
Re-run on origin/main @ 50d0d6c, which contains objectui#6117 (CRUDSchema
retired). Converter: zod 4.4.3's built-in z.toJSONSchema (already a dependency).
Population: AnyComponentSchema — 14 union members, 105 leaf node schemas.
Headline: the generated artifact is a strict relaxation of the zod source. Over
475 root-type JSON files it agrees with safeValidateSchema on 461 (97.1%), is
weaker on 14, and is stricter on none. 0/105 node types convert under default
options; 105/105 convert lenient; 0/105 faithful. 276 of 333 objects are open
because BaseSchema is .passthrough() and .strict() is called zero times, so the
artifact reads closed and is not. Size: 651,908 bytes pretty / 369,028
minified, or +21.5% / +12.2% of the published dist.
New since the pre-#6117 run: all 5 refinement sites isolated with passing
controls (was 2 of 5); @objectstack/spec already ships generated JSON Schema
(23.4 MB, 156 UI documents, 118/124 closed at the root) as both a precedent and
the closedness contrast; the JSONC-only files were checked and carry no root
type, so the 475 census is complete against check.ts's own reader.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CSoz9uGhaaSgiq3hshtN7L
@github-actionsgithub-actionsBot added the documentation Improvements or additions to documentation label Aug 25, 2026
@yinlianghui
yinlianghui marked this pull request as ready for review August 25, 2026 01:36
@yinlianghui
yinlianghui added this pull request to the merge queueAug 25, 2026
Merged via the queue into main with commit c8995d1Aug 25, 2026
23 checks passed
@yinlianghui
yinlianghui deleted the claude/issue-5392-json-schema-fidelity-measurement branch August 25, 2026 01:47
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@yinlianghui@claude