Skip to content

docs: generated terms.md with sync check - #262

Merged
chaksaray merged 1 commit into
developfrom
docs/terms-md-generated
Sep 5, 2026
Merged

docs: generated terms.md with sync check#262
chaksaray merged 1 commit into
developfrom
docs/terms-md-generated

Conversation

@chaksaray

Copy link
Copy Markdown
Contributor

Closes #261. Glossary generated from live schema descriptions, never
hand-edited. check_terms_sync.py is a hard CI failure if terms.md and
the schema disagree, unlike this project's other soft-warning checks,
since a wrong glossary entry is worse than a missing one.

What's here

  • scripts/generate_terms.py — the only thing that writes docs/terms.md's per-field section, from schema/ave-record-1.1.0.schema.json's own properties[name].description values (confirmed all 46 have a real, non-empty description before writing this).
  • docs/terms-relationships.md — hand-written notes on the confusable field pairs, checked in separately, re-emitted verbatim by the generator rather than regenerated.
  • scripts/check_terms_sync.py — fails if docs/terms.md doesn't match what the generator would produce right now. Hard failure (exit 1), not a soft warning — a deliberate, explicit break from this project's usual pattern (check_confidence_signal.py, check_framework_sources.py), since a stale glossary entry looks as authoritative as a correct one.
  • tests/test_terms.py — 7 tests, each mutation-checked by hand (dropping sorted(), breaking the empty-description fallback, breaking crash-tolerance on an empty relationships string, breaking load_field_descriptions(), disabling the sync check's own comparison — each correctly turned exactly the test(s) naming that behavior red).
  • Wired into .github/workflows/tests.yml as its own named step, and both commands added to CONTRIBUTING.md's local check list plus a dedicated line under Schema changes: regenerate after any field description change, before opening a PR.

Verified, not copied from the draft as given

Every relationship note was checked against the live schema and this project's actual history before writing it:

Validated

  • python scripts/validate_records.py: 80/80 valid
  • python scripts/check_fixtures.py: all pass
  • python scripts/check_terms_sync.py: passes against the live schema
  • pytest tests/ -x -q: 470 passed (463 existing + 7 new)

Closes #261.

Per-field definitions are generated from the schema's own descriptions
and never hand-edited; check_terms_sync.py fails CI if the two
disagree. Relationship notes for the known confusable field pairs
(confidence_baseline/verification_basis, evidence_vantage/
evidence_method, entry_class/detection_layer/detection_stage,
security_boundary/missing_control, the framework-mapping fields versus
framework_sources) are hand-written and checked in separately in
docs/terms-relationships.md.

Every relationship note was verified against the live schema and this
project's history before writing it, not copied from a draft as given:

- entry_class's real enum (14 values: content, server_card,
  registry_metadata, runtime, transport, tool_response, tool_schema,
  server_card_document, model_generated, memory, retrieved_document,
  user_input, operator_config, skill_file) is broader than a first
  pass assumed, and its own schema description says to reuse
  detection_layer's value directly when the class is layer-scoped --
  the two fields are designed to share values in the common case, not
  to always diverge.
- verification_basis's substrate_reconstructed combination was
  confirmed as a real enum member and a real derivation output in
  scripts/write_verification_basis.py, not an invented example.
- The framework_sources note's coverage claim was checked against the
  live corpus rather than stated generically: owasp_mcp and owasp_asi
  are fully backfilled (80/80, 69/69), mitre_atlas is 40 of 50, and
  nist_ai_rmf has none -- following the recent backfill work
  (#255, #256, #259). An earlier draft's "most of the current 80 do
  [lack provenance]" would already have been wrong by the time this
  merged.
- issue #98 and PR #224 were both fetched and read directly to confirm
  the confidence_baseline and security_boundary/missing_control notes
  before writing them, rather than assumed from memory of prior work
  in this same project.

check_terms_sync.py is a hard CI failure, not a soft warning -- a
deliberate, explicit break from this project's usual pattern
(check_confidence_signal.py, check_framework_sources.py), since a
stale glossary entry looks as authoritative as a correct one and has
no safe warn-and-continue default.

All 46 schema properties confirmed to carry a real, non-empty
description before the generator was written against them.

Every new test in tests/test_terms.py mutation-checked by hand:
dropping sorted() in render(), breaking the empty-description
fallback, breaking render()'s tolerance of an empty relationships
string, breaking load_field_descriptions(), and disabling the sync
check's own comparison each correctly turned exactly the test(s)
naming that behavior red, confirmed then reverted.

Validated: python scripts/validate_records.py (80/80 valid), python
scripts/check_fixtures.py (all pass), python scripts/check_terms_sync.py
(passes against the live schema), pytest tests/ -x -q (470 passed, 463
existing + 7 new). check_terms_sync.py wired into
.github/workflows/tests.yml as its own named step, and both commands
added to CONTRIBUTING.md's local check list plus a dedicated note under
the Schema changes section stating plainly: regenerate after any field
description change, before opening a PR.
@chaksaray chaksaray linked an issue Sep 4, 2026 that may be closed by this pull request
@chaksaray
chaksaray merged commit 92bb011 into develop Sep 5, 2026
6 checks passed
@chaksaray
chaksaray deleted the docs/terms-md-generated branch September 5, 2026 00:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add generated terms.md with a hard sync check

1 participant