Add the interface engine - verify a workflow's contract by name (#374, PR-B) - #375
Conversation
… PR-B) Consume fidelity: interface. The audit now checks that a carried workflow honors its fixed contract without touching the owned body. - audit.py: split_jobs slices a workflow into job blocks by indent (no YAML parser); check_interface verifies required job keys, the ruleset-bound check name, the release-asset-<branch>-<target> handoff, and per-job required (pattern:/merge-multiple:) and forbidden (artifact-ids:) tokens. All findings are DRIFT - the body is owned and a rename is a hint to verify. An interface unit's absence is DRIFT too, not a hard letter, since a workflow's naming varies more than a carried config. - files.json: two workflow interface entries - test-pull-request.yml (universal: the check-workflow-status job and its ruleset-bound name) and build-release-task.yml (build types: the get-version/github-release jobs, the artifact handoff, and the no-artifact-ids fork rule). - audit.py --selftest exercises the engine on synthetic fixtures (7 cases: conformant, missing job+name, renamed check, artifact-ids fork, missing merge-multiple, an owned extra leaf job, and split_jobs). - AUDIT.md notes the new check. Also fix pre-existing clause-joining semicolons in this file's comments and docstrings so they do not fester. Verified: --selftest 7/7 pass, and the hub's own canonical test-pull-request.yml and build-release-task.yml are conformant to their contracts (0 findings). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds an interface-fidelity audit path so workflow baseline entries can be verified by their contracted wiring (job keys, required job name:, artifact handoff conventions, and forbidden download patterns) without requiring verbatim workflow bodies. This fits the repo’s broader “carry model” governance by making workflow conformance checkable in a structured, low-false-positive way.
Changes:
- Extend
spec/audit.pywith a lightweight workflow “jobs” slicer (split_jobs) and an interface-contract checker (check_interface), plus a--selftestfixture suite. - Add
fidelity: interfaceentries tospec/files.jsonfortest-pull-request.ymlandbuild-release-task.ymlwith targeted contract fields. - Update
AUDIT.mdto document the new interface-conformance check and its DRIFT-only semantics.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| spec/files.json | Adds two fidelity: interface baseline entries with workflow-specific contract fields. |
| spec/audit.py | Implements the interface engine (split_jobs, check_interface) and wires it into the audit with --selftest. |
| AUDIT.md | Documents the new workflow interface conformance check as part of the mechanized audit. |
Uh oh!
There was an error while loading. Please reload this page.
Copilot round-1: "Three recursive trees calls" reads wrong; correct to "tree calls", consistent with "any tree is truncated" in the same sentence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Copilot round-2 findings: - split_jobs: a dedented comment after the jobs children no longer scans into a job block or ends the mapping early - it is skipped, so a trailing or between-jobs top-level comment is not mis-attributed. Added a split_jobs test with a trailing top-level comment. - check_interface: skip the require/forbid token checks for a job that is absent - the requiredJobKeys check already reports the absence, so no redundant "job missing required token" for every token it cannot contain. Added a self-test: an absent required job reports exactly once. Verified: --selftest 8/8, hub canonical workflows still conformant. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Copilot round-3 findings: check_interface used a raw substring search over the whole workflow text and job blocks, so a token mentioned only in a comment counted as signal. A carried task file documents its own contract in comments (build-release-task.yml names release-asset- and artifact-ids: in prose), so this could false-pass a missing handoff and false-flag a forbidden token that appears only in a comment. Scan a code view (comment-only lines dropped) for requiredCheckName, artifactNameToken, and the per-job require/forbid tokens. Added self-tests: a forbidden token only in a comment is ignored, and a required token only in a comment does not count. Verified: --selftest 10/10, hub canonical workflows still conformant. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
spec/audit.py:205
- check_interface() treats requiredCheckName as a raw substring match across the whole workflow (minus comment-only lines). That can false-pass if the string appears in some other field (including the workflow-level
name:) and does not actually appear as a jobname:in the required job. Since the contract is about the ruleset-bound job check name, match it as aname:line (and prefer searching within the required job block when there is exactly one required job).
name = contract.get("requiredCheckName")
if name and name not in code:
findings.append(("DRIFT", f"interface: {path} missing the ruleset-bound check name '{name}'"))
tok = contract.get("artifactNameToken")
if tok and tok not in code:
findings.append(("DRIFT", f"interface: {path} missing the '{tok}<branch>-<target>' artifact handoff"))
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Copilot round-4: - split_jobs: _JOB_KEY now matches a key with an inline value, and the key line is included in its block, so an inline-mapping job (rare, but valid) is captured with its content. Added inline-mapping and trailing-comment split_jobs self-tests. - check_interface: match requiredCheckName as a `name:` field value (not a raw substring), so the ruleset-bound check name appearing in some other field (a run step, the workflow-level name) does not false-pass. Added a self-test for the check name present only in a run step. Verified: --selftest 13/13, hub canonical workflows still conformant. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
spec/audit.py:205
- check_interface()'s requiredCheckName scan matches any
name:line anywhere in the workflow (including step-levelname:). That can false-pass a workflow where the ruleset-bound job name is missing but a step happens to use the same name. Since the contract is explicitly about the ruleset-bound job name, restrict the match to job-levelname:keys (the first-level keys inside each job block) and optionally add a selftest case where a stepname:equals requiredCheckName but the job-level name is absent.
name = contract.get("requiredCheckName")
if name and not re.search(r"^\s*name:\s*['\"]?" + re.escape(name) + r"['\"]?\s*$", code, re.M):
findings.append(("DRIFT", f"interface: {path} missing the ruleset-bound check name '{name}' as a job name"))
tok = contract.get("artifactNameToken")
if tok and tok not in code:
findings.append(("DRIFT", f"interface: {path} missing the '{tok}<branch>-<target>' artifact handoff"))
Copilot round-5 low-confidence finding (correct): the requiredCheckName scan
matched any name: line, so a step named the same as the ruleset-bound check
could false-pass while the job name was absent. Add job_level_names (a job's
own direct-child name key, at the shallowest block indent, excluding step names
and - name: list items) and match requiredCheckName against it. Added
self-tests for the check name appearing only as a step name or only in a run
step.
Verified: --selftest 14 cases pass, and the hub's test-pull-request.yml job
names ('Check pull request workflow status job', 'Validate sources job') are
extracted at job level so it stays conformant.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Second of the fidelity series (#374). Consumes
fidelity: interface: the audit now verifies a carried workflow honors its fixed contract by name and wiring, never the owned body — the virtual-function signature check.Change
spec/audit.pysplit_jobs(text)— slices a workflow into{job_key: block}by the jobs-level indent, no YAML parser (stdlib), mirroringheading_texts.check_interface(path, contract, text)— required job keys, the ruleset-bound checkname:, therelease-asset-<branch>-<target>handoff, and per-job required (pattern:/merge-multiple:) and forbidden (artifact-ids:) tokens. All findings DRIFT.interfaceunit's absence is DRIFT too (not a hard LETTER) — a workflow's naming varies more than a carried config, so everything here is advisory.--selftest(no network) — 7 fixture cases.spec/files.json— two workflowinterfaceentries:test-pull-request.yml(appliesTo: "*", the universalcheck-workflow-statusjob + its ruleset-bound name) andbuild-release-task.yml(build types, theget-version/github-releasejobs + artifact handoff + no-artifact-ids:rule).AUDIT.md— notes the new check, and fixes pre-existing clause-joining semicolons in this file's comments/docstrings so they don't fester.Verified
python3 spec/audit.py --selftest→ 7/7 (conformant, missing job+name, renamed check, artifact-ids fork, missing merge-multiple, owned extra-leaf-job no-finding, split_jobs).test-pull-request.ymlandbuild-release-task.ymlare conformant (0 findings).validate.pygreen (contracts pass PR-A's value-type guards), lints clean.Scoping of
build-release-task.ymlis best-effort (excludes source-only/docs to avoid false "absent" DRIFTs); any mis-scope only yields advisory DRIFTs to triage. It refines as downstreams are audited. Held ondevelop. PR-C adds the verbatim engine.