docs: generated terms.md with sync check - #262
Merged
Merged
Conversation
Closes #261. Per-field definitions are generated from the schema's own descriptions and never hand-edited; check_terms_sync.py fails CI if the two disagree. Relationship notes for the known confusable field pairs (confidence_baseline/verification_basis, evidence_vantage/ evidence_method, entry_class/detection_layer/detection_stage, security_boundary/missing_control, the framework-mapping fields versus framework_sources) are hand-written and checked in separately in docs/terms-relationships.md. Every relationship note was verified against the live schema and this project's history before writing it, not copied from a draft as given: - entry_class's real enum (14 values: content, server_card, registry_metadata, runtime, transport, tool_response, tool_schema, server_card_document, model_generated, memory, retrieved_document, user_input, operator_config, skill_file) is broader than a first pass assumed, and its own schema description says to reuse detection_layer's value directly when the class is layer-scoped -- the two fields are designed to share values in the common case, not to always diverge. - verification_basis's substrate_reconstructed combination was confirmed as a real enum member and a real derivation output in scripts/write_verification_basis.py, not an invented example. - The framework_sources note's coverage claim was checked against the live corpus rather than stated generically: owasp_mcp and owasp_asi are fully backfilled (80/80, 69/69), mitre_atlas is 40 of 50, and nist_ai_rmf has none -- following the recent backfill work (#255, #256, #259). An earlier draft's "most of the current 80 do [lack provenance]" would already have been wrong by the time this merged. - issue #98 and PR #224 were both fetched and read directly to confirm the confidence_baseline and security_boundary/missing_control notes before writing them, rather than assumed from memory of prior work in this same project. check_terms_sync.py is a hard CI failure, not a soft warning -- a deliberate, explicit break from this project's usual pattern (check_confidence_signal.py, check_framework_sources.py), since a stale glossary entry looks as authoritative as a correct one and has no safe warn-and-continue default. All 46 schema properties confirmed to carry a real, non-empty description before the generator was written against them. Every new test in tests/test_terms.py mutation-checked by hand: dropping sorted() in render(), breaking the empty-description fallback, breaking render()'s tolerance of an empty relationships string, breaking load_field_descriptions(), and disabling the sync check's own comparison each correctly turned exactly the test(s) naming that behavior red, confirmed then reverted. Validated: python scripts/validate_records.py (80/80 valid), python scripts/check_fixtures.py (all pass), python scripts/check_terms_sync.py (passes against the live schema), pytest tests/ -x -q (470 passed, 463 existing + 7 new). check_terms_sync.py wired into .github/workflows/tests.yml as its own named step, and both commands added to CONTRIBUTING.md's local check list plus a dedicated note under the Schema changes section stating plainly: regenerate after any field description change, before opening a PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #261. Glossary generated from live schema descriptions, never
hand-edited. check_terms_sync.py is a hard CI failure if terms.md and
the schema disagree, unlike this project's other soft-warning checks,
since a wrong glossary entry is worse than a missing one.
What's here
scripts/generate_terms.py— the only thing that writesdocs/terms.md's per-field section, fromschema/ave-record-1.1.0.schema.json's ownproperties[name].descriptionvalues (confirmed all 46 have a real, non-empty description before writing this).docs/terms-relationships.md— hand-written notes on the confusable field pairs, checked in separately, re-emitted verbatim by the generator rather than regenerated.scripts/check_terms_sync.py— fails ifdocs/terms.mddoesn't match what the generator would produce right now. Hard failure (exit 1), not a soft warning — a deliberate, explicit break from this project's usual pattern (check_confidence_signal.py,check_framework_sources.py), since a stale glossary entry looks as authoritative as a correct one.tests/test_terms.py— 7 tests, each mutation-checked by hand (droppingsorted(), breaking the empty-description fallback, breaking crash-tolerance on an empty relationships string, breakingload_field_descriptions(), disabling the sync check's own comparison — each correctly turned exactly the test(s) naming that behavior red)..github/workflows/tests.ymlas its own named step, and both commands added toCONTRIBUTING.md's local check list plus a dedicated line under Schema changes: regenerate after any field description change, before opening a PR.Verified, not copied from the draft as given
Every relationship note was checked against the live schema and this project's actual history before writing it:
detection_layer's value directly when the class is layer-scoped, so the note says the two are designed to share values in the common case, not to always diverge.substrate_reconstructedcombination is confirmed as a real enum member and a real output ofscripts/write_verification_basis.py's derivation logic, not an invented illustration.owasp_mcpandowasp_asiare now fully backfilled (80/80, 69/69),mitre_atlasis 40 of 50,nist_ai_rmfhas none — reflecting the recent backfill work (framework_sources backfill, and a real gap found: pin_status has no 'unknown' state #255, feat: framework_sources backfill (owasp_asi, mitre_atlas), and a schema fix it needed first #256, fix: owasp_mcp audit corrections (39 records) and framework_sources backfill #259). A generic "most records lack provenance" claim would already be stale.confidence_baseline/verification_basisandsecurity_boundary/missing_controlnotes, rather than assumed from memory.Validated
python scripts/validate_records.py: 80/80 validpython scripts/check_fixtures.py: all passpython scripts/check_terms_sync.py: passes against the live schemapytest tests/ -x -q: 470 passed (463 existing + 7 new)