Skip to content

ci(translations): enforce the output checklist as a validator - #74

Merged
os-zhuang merged 1 commit into
mainfrom
claude/issue-67-translation-output-validators
Aug 18, 2026
Merged

ci(translations): enforce the output checklist as a validator#74
os-zhuang merged 1 commit into
mainfrom
claude/issue-67-translation-output-validators

Conversation

@os-zhuang

Copy link
Copy Markdown
Contributor

Fixes#67

Turns the pre-PR checklist at the end of docs/TRANSLATION.md into
.github/scripts/check-translation-output.mjs — a third zero-dependency check
next to the freshness and ownership gates, wired into translations.yml.

Verified on 751312f.

Why a stamp was not enough

check-translations.mjs answers "was this derived from the current English?".
It cannot answer "is what came back actually a translation of that page?" The
gap is measurable: of the 84 files with fidelity findings on main, 83 are
not stale
— they carry a current source_sha and the freshness gate passes
them. Two of the findings:

FileFinding
resources/support.de.mdxsupport-ticket template reports ObjectOS version: 10.2.0; English says 14.7.0
resources/license.{de,es,fr,ja,ko}.mdxadded a sentence linking to /docs/reference/security, which the English license page never references

The version number is inside a fenced code block. That is the "helpfulness"
failure the card describes: the pass edited a code sample while translating, and
provenance had nothing to say about it.

Shape

Answers to the two questions left open in the card.

A third script, not a fold into check-translations.mjs. Different verdict
class (fidelity, not provenance), different remediation, and folding it in would
tangle the --stamp / --baseline / --worklist modes. It duplicates
locales() and walk(), which is what the two existing scripts already do to
stay self-contained.

It runs on every PR touching content/docs/, not only translation PRs. The
ownership check is inert until TRANSLATION_BOT_LOGIN is set (#68), so a
hand-edited locale file reaches main today; a validator scoped to translation
PRs would be blind to exactly the edits nothing else currently catches. Scanning
all 335 pages costs milliseconds, so nothing is bought by scoping the scan.

What is scoped is what blocks, mirroring the blocking/reported split
check-translations.mjs already makes:

  • unsafe blocks anywhere in the corpus, on English sources too. It is a
    security rule, a diff-scoped one is defeated by ordering, and the corpus is
    clean of it today (0 findings) so it can hold the line from day one.
  • fidelity rules block on the locale files the PR changed; the remaining 91
    pre-existing findings are reported. Fidelity is a relation between two
    files, so an English-only edit breaks it in six siblings the PR deliberately
    left stale — blocking those would contradict "stale is reported, not
    blocking" in AGENTS.md, and a gate born red on files nobody touched is a gate
    switched off in a week.

Two rules that needed calibrating rather than implementing literally

Length. Applied against 1.0 the ±40% band fails 77 of 256 translations —
because a correct Simplified Chinese page runs about 0.54x the character count
of its English source and a correct French one about 1.13x. Judged that way the
rule reports translating into Chinese as a defect. The band is applied to the
ratio normalized by each locale's measured expansion factor (LOCALE_EXPANSION,
recomputed by --calibrate); 12 findings remain, and the extremes are real —
configure/permissions/positions.de.mdx at 0.30x is missing an entire trailing
section. An uncalibrated locale is reported, never judged against 1.0.

URLs. Fragments are stripped before comparison. A heading id is derived from
the heading text and the heading text is translated, so
#the-open-source-alternative must become #开源替代方案 and
permission-sets#delegated-administration must become
permission-sets#委托管理delegated-administration; both are correct, and
comparing fragments demanded every translation link to headings it does not
have. Which page a link points at is still compared, which is what catches
the invented /docs/reference/security cross-reference above. Both false-
positive classes are pinned by fixtures.

Every rule is demonstrated able to fail

--self-test builds fixtures in a temp dir and asserts the exact set of
rules each one fires, so a fixture that goes red for the wrong reason fails.
Written that way it immediately caught a defect in itself: the clean pair
tripped length, because the fixture was being judged against a different
locale's band.

✓ clean pair fired [—]
✓ fewer links than English (subset is allowed) fired [—]
✓ code fence edited fired [fence]
✓ code fence info string translated fired [fence]
✓ code fence dropped fired [fence]
✓ link the English page does not have fired [url]
✓ cross-page link with a translated fragment fired [—]
✓ link to a page the English page never references fired [url]
✓ same-page anchor translated with its heading fired [—]
✓ URL closed by full-width punctuation (no space) fired [—]
✓ frontmatter key added fired [frontmatter]
✓ frontmatter key dropped fired [frontmatter]
✓ MDX component renamed fired [component]
✓ MDX prop dropped fired [component]
✓ script tag in the translation fired [unsafe]
✓ javascript: URL in the translation fired [unsafe]
✓ on*= handler in the translation fired [unsafe]
✓ script tag in the ENGLISH source fired [unsafe]
✓ body truncated to half fired [length]
✓ body padded with invented content fired [length]
✓ self-test: 20 case(s) on locale "zh-Hans", every rule demonstrated able to fail

The self-test runs as its own workflow step, before the corpus, so a rule that
stops being able to fail fails the job. It is the only executable guarantee
available here: pnpm turbo run test runs zero tasks in this repo.

Bounded in-place fix: both gate steps were swallowing their exit status

Not the card's subject, but the same defect class (a gate that cannot go red) in
a file already in this card's surface, and I could not add a fourth piped step
next to it in good conscience.

The default shell for a run: step is bash -e {0} with no pipefail
(workflow syntax),
so in node ... | tee -a "$GITHUB_STEP_SUMMARY" the step takes tee's status.
Measured with a real gate script from this repo:

$ node .github/scripts/check-translation-output.mjs >/dev/null 2>&1; echo $?
1
$ bash -e -c 'node .github/scripts/check-translation-output.mjs 2>/dev/null | tee /dev/null >/dev/null; echo $?'
0

The Freshness step has this shape on main, so its blocking verdicts
(unstamped, orphan) exit 1 locally and pass the job in CI. Fix is shell: bash
on each piped step, which is bash --noprofile --norc -eo pipefail {0}.

Verification

Run on 751312f, the commit this PR is opened at.

$ node .github/scripts/check-translations.mjs
English pages: 79 · translations: 256 · guide rev 1
✓ translations gate passed EXIT=0
$ node .github/scripts/check-translation-ownership.mjs --actor test --files changed.txt
⚠ TRANSLATION_BOT_LOGIN is not set — ownership is not enforced yet.
This PR touches 0 translation artifact(s) and 3 other file(s). EXIT=0
$ TRANSLATION_BOT_LOGIN=objectos-translator ... --actor objectos-translator ...
✗ translation PRs may only touch translation artifacts. EXIT=1
$ node .github/scripts/check-translation-output.mjs --files changed.txt
Scanned 335 page(s) · blocking on 0 changed translation(s)
✓ gate passed (91 pre-existing finding(s) reported) EXIT=0
$ pnpm turbo run type-check --force 1 successful, EXIT=0
$ pnpm turbo run build --force 1 successful, EXIT=0
$ pnpm turbo run test 0 tasks — no test task exists, EXIT=0

Both existing checks pass unchanged, and the ownership check still fails in the
direction it should. Scoping verified in both directions: a diff naming
resources/support.de.mdx exits 1 with exactly that file's fence finding, and
a diff naming a clean locale file exits 0 despite the 91 corpus findings.

origin/main had not moved when this was pushed (a132376), so the declared
read-coupling with #61 on apps/docs/lib/i18n.ts did not fire — #61 lands
second and should re-run these three checks.
The locale list is still read
from i18n.ts, never hardcoded; the locale list and the LOCALE_EXPANSION
table are separate, and a locale present in i18n.ts but absent from the table
is reported uncalibrated rather than judged.

Not done here, deliberately: GUIDE_REV is not bumped. This PR changes how
the guide is enforced, not what to translate, and bumping it would mark all 256
files for retranslation. The 91 pre-existing findings are filed separately.


Generated by Claude Code

The pre-PR checklist in docs/TRANSLATION.md was prose: code fences
byte-identical to English, URL set a subset, frontmatter keys matching,
MDX components unchanged, no <script / javascript: / on*=, length within
±40%. A prose list is checked by the same agent that just decided the
link was broken.
Adds .github/scripts/check-translation-output.mjs, a third zero-dependency
check alongside the freshness and ownership gates. It answers the question
a provenance stamp cannot: a file can carry a current source_sha and still
have had its code samples repaired or half its body dropped. Measured on
main: 83 of the 84 files with fidelity findings are NOT stale.
Scope: the whole corpus is always scanned; what is scoped to the PR diff
is what blocks, mirroring the blocking/reported split check-translations.mjs
already makes. `unsafe` blocks anywhere, on English sources too — it is a
security rule and the corpus is clean of it today.
Also fixes both gate steps swallowing their exit status: the default `run:`
shell is `bash -e {0}` with no pipefail, so `node ... | tee` reported tee's
status and the freshness gate could not fail the job.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CJPxtTxoxTUnjNdTbiEaRa
@os-zhuang
os-zhuang marked this pull request as ready for review August 18, 2026 09:54
@os-zhuang
os-zhuang merged commit 722e842 into mainAug 18, 2026
2 checks passed
@os-zhuangClaude

Copy link
Copy Markdown
ContributorAuthor

Merged as 722e8420 under a direct maintainer instruction, quoted verbatim for the audit trail: 「ci 绿了就合并」.

That instruction lifts the hold recorded in the ACCEPT above, which was not about this PR's content — the path surface is clean of the governed face (.github/** and docs/TRANSLATION.md are not in it) — but about objectos having no merge queue, which is open as objectstack-ai/objectstack#9551.

Merged first of the three reviewed PRs deliberately: it is the one that repairs the swallowed exit status, so landing it ahead of #69 and #73 means those two are re-checked by a gate that can actually fail, rather than by my assertion that they would pass. Both have been updated from main for exactly that reason.


Generated by Claude Code

os-warren pushed a commit that referenced this pull request Aug 19, 2026
The three Node floor declarations — root `engines.node`, `apps/docs`
`engines.node`, and `.node-version` — are read by nothing in the install
path. `.npmrc` sets no `engine-strict`, pnpm does not enforce `engines` by
default, and all three workflows pin `node-version: 22` explicitly rather
than consulting them. A wrong floor was therefore structurally silent: this
repo's stayed wrong across the whole Node 20 era and was caught by someone
reading a file, not by a red job.
`check-node-floor.mjs` reduces every `engines.node` range in
`pnpm-lock.yaml` to the lowest Node version satisfying it, takes the maximum
across the tree, and requires each declared floor to clear it. The direction
is the rule: a dependency asking for less than the declaration is the normal
case and stays green. It also asserts the three declarations agree with each
other. Zero-dependency, so it runs without an install.
Per PR #74, `--self-test` builds fixtures and asserts each rule is
demonstrably able to fail, plus that the silence-bearing rules can stay
silent. It ran before the gate ever did and caught two real parser bugs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DXBoKN4MauvPdbMemPMqpr
os-warren added a commit that referenced this pull request Aug 19, 2026
* ci: gate the declared Node floor against the dependency tree
The three Node floor declarations — root `engines.node`, `apps/docs`
`engines.node`, and `.node-version` — are read by nothing in the install
path. `.npmrc` sets no `engine-strict`, pnpm does not enforce `engines` by
default, and all three workflows pin `node-version: 22` explicitly rather
than consulting them. A wrong floor was therefore structurally silent: this
repo's stayed wrong across the whole Node 20 era and was caught by someone
reading a file, not by a red job.
`check-node-floor.mjs` reduces every `engines.node` range in
`pnpm-lock.yaml` to the lowest Node version satisfying it, takes the maximum
across the tree, and requires each declared floor to clear it. The direction
is the rule: a dependency asking for less than the declaration is the normal
case and stays green. It also asserts the three declarations agree with each
other. Zero-dependency, so it runs without an install.
Per PR #74, `--self-test` builds fixtures and asserts each rule is
demonstrably able to fail, plus that the silence-bearing rules can stay
silent. It ran before the gate ever did and caught two real parser bugs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DXBoKN4MauvPdbMemPMqpr
* ci: register the node floor self-test, report ungoverned declarations
Two things the repo's own guards surfaced after the first commit.
`tools/ci-scripts/run-self-tests.mjs` fails when a script under
`.github/scripts/` declares a `--self-test` dispatch but is not listed in
`SELF_TESTED` — so the new gate turned the required `build` job red until it
was registered. Measured: the runner exits 1 naming the file, and 0 with the
script listed. That guard is exactly what it was written for.
The workspace also holds a FOURTH Node declaration, not the three the card
names: `tools/ci-scripts/package.json` declares `engines.node ">=20.0.0"`,
disagreeing with the root and `apps/docs`. An explicit `DECLARATION_FILES`
list that silently covers two of three declarations is the same
structurally-silent shape this gate exists to end, so the gate now discovers
every workspace `package.json` and REPORTS the ones it does not govern.
The `ungoverned` rule is advisory, never blocking. "Not governed" is not a
claim the value is wrong: `tools/ci-scripts` has no dependencies of its own,
so `>=20.0.0` may be correct in isolation. Whether the workspace holds one
floor or several is a decision for the seat, not something this script
should force by going red.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DXBoKN4MauvPdbMemPMqpr
---------
Co-authored-by: Claude <noreply@anthropic.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enforce the translation output checklist as an automated check, not a prose list

2 participants

@os-zhuang@claude