From dbfcc7d2a798f7c18c923ac892f48afd602b2f8a Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Mon, 10 Aug 2026 08:50:54 -0700 Subject: [PATCH 1/5] Compare the Ruleset Fields the Fleet Actually Manages The AUDIT.md section 6 ruleset comparison projected bypass_actors on both sides. No committed payload declares that key, so the committed side did not merely mis-compare, it exited 5 on "Cannot iterate over null". Process substitution then handed diff an empty left side and every ruleset on every repo reported DRIFT. Blog, which is in sync, reported both branches drifted against the whole live payload. spec/audit.py already excludes the field, and its comment records that including it once made every repo report a ruleset DEFECT. configure.sh is the authority the two must agree with: apply writes the live list back unchanged and check reports it without asserting, because who may bypass a ruleset is a human decision taken in the UI and no payload declares one. Two further surfaces stated the old policy and are swept with it. The payload regeneration snippet in docs/repo-config-carry.md captured bypass_actors into the committed file, which would bake one repo's bypass list into the canonical every other repo diffs against. repo-config/README.md listed the field as part of the subset the audit compares. Co-Authored-By: Claude Opus 5 (1M context) --- AUDIT.md | 7 ++++--- docs/repo-config-carry.md | 3 ++- repo-config/README.md | 2 +- 3 files changed, 7 insertions(+), 5 deletions(-) diff --git a/AUDIT.md b/AUDIT.md index 0f22bb0a..8c42fd24 100644 --- a/AUDIT.md +++ b/AUDIT.md @@ -74,10 +74,11 @@ Run [`WORKFLOW.md`][workflow]'s methodology against the repo's **own** Actions: && echo "settings: in sync" || echo "settings: DRIFT" ``` -- **Rulesets** - diff each live ruleset against the committed expected payload with a normalized comparison (sort the order-insensitive `rules[]` and `bypass_actors[]` before diffing so a reordered but equivalent ruleset does not read as drift): +- **Rulesets** - diff each live ruleset against the committed expected payload with a normalized comparison (sort the order-insensitive `rules[]` before diffing so a reordered but equivalent ruleset does not read as drift). The compared subset is `name`, `target`, `enforcement`, `conditions` and `rules`, and `bypass_actors` sits deliberately outside it, which is the same subset [`spec/audit.py`][audit-runner] compares. Who may bypass a ruleset is a per-repository human decision taken in the UI, no payload declares one, and [`repo-config/configure.sh`][repo-config] treats it that way in both modes, writing the live list back unchanged on `apply` and reporting it without asserting on `check`. Comparing it here would contradict that and report a ruleset finding against every repository that has any bypass actor, which is the field's normal state rather than a deviation: ```sh - norm='{name,target,enforcement,bypass_actors,conditions,rules} | .rules|=sort_by(.type) | .bypass_actors|=sort_by(.actor_id)' + # bypass_actors stays outside the projection, since no payload declares one and jq cannot sort the null that leaves. + norm='{name,target,enforcement,conditions,rules} | .rules|=sort_by(.type)' # Model-aware expected payload: an operational repo's develop ruleset diffs against # operational/develop.json (registry workflowModel; the same selection audit.py makes). model=$(jq -r --arg n "" '(.repos[] | select(.name==$n) | .workflowModel) // .defaults.workflowModel // "release"' registry/repos.json) @@ -92,7 +93,7 @@ Run [`WORKFLOW.md`][workflow]'s methodology against the repo's **own** Actions: [ "$count" -eq 1 ] || { echo "$b: expected exactly 1 ruleset, found $count (defect/drift)"; continue; } id=$(jq --arg n "$b" '.[] | select(.name==$n) | .id' <<<"$rulesets") diff <(jq -S "$norm" "$file") \ - <(gh api "repos///rulesets/$id" --jq '{name,target,enforcement,bypass_actors,conditions,rules}' | jq -S "$norm") \ + <(gh api "repos///rulesets/$id" --jq '{name,target,enforcement,conditions,rules}' | jq -S "$norm") \ && echo "$b: in sync" || echo "$b: DRIFT" done ``` diff --git a/docs/repo-config-carry.md b/docs/repo-config-carry.md index 472661d4..6f743807 100644 --- a/docs/repo-config-carry.md +++ b/docs/repo-config-carry.md @@ -41,8 +41,9 @@ for name in develop main; do count=$(jq --arg n "$name" '[.[] | select(.name==$n)] | length' <<<"$rulesets") [ "$count" -eq 1 ] || { echo "expected exactly 1 ruleset named $name, found $count (drift)" >&2; exit 1; } id=$(jq --arg n "$name" '.[] | select(.name==$n) | .id' <<<"$rulesets") + # bypass_actors is left out, since a payload that declared one would assert this repo's bypass list against every repo diffed on it. gh api "repos/$repo/rulesets/$id" \ - --jq '{name, target, enforcement, bypass_actors, conditions, rules}' \ + --jq '{name, target, enforcement, conditions, rules}' \ | jq -S --indent 4 '.' > "$out" done ``` diff --git a/repo-config/README.md b/repo-config/README.md index b0af33a1..dc67d969 100644 --- a/repo-config/README.md +++ b/repo-config/README.md @@ -2,7 +2,7 @@ Repository and branch configuration held as committed files, kept out of `.github/` (which holds the GitHub-consumed configuration: workflows, Dependabot). -- `main.json` plus one `develop` variant: the branch rulesets as the writable API subset (`name`, `target`, `enforcement`, `bypass_actors`, `conditions`, `rules`). The `develop` payload is `develop.json` (`release` repos) or `operational/develop.json` (`operational` repos). These are the canonical expected payloads that the self-audit (`AUDIT.md`) diffs the live rulesets against. +- `main.json` plus one `develop` variant: the branch rulesets as the managed part of the writable API subset (`name`, `target`, `enforcement`, `conditions`, `rules`). The `develop` payload is `develop.json` (`release` repos) or `operational/develop.json` (`operational` repos). These are the canonical expected payloads that the self-audit (`AUDIT.md`) diffs the live rulesets against. `bypass_actors` is writable and deliberately unmanaged, so no payload declares one and nothing diffs it: who may bypass a ruleset is a human decision taken in the UI, which `configure.sh` preserves on `apply` and reports without asserting on `check`. - `operational/develop.json`: the `develop` ruleset for **operational** repos (registry `workflowModel: operational`), taking direct signed pushes with no PR gate. Present in operational repos only, since a `release` repo does not have it. See "Rulesets" below. - `configure.sh`: **hosted in the hub and run from a hub checkout, not carried here**, per [GOVERNANCE.md "Hub-Hosted Tooling"][governance-hub-hosted-tooling]. The payloads above are what this repo is audited against and stay with it. The script holds nothing per-repo and is one copy for the fleet, and it resolves every payload path against its own directory rather than against the target repo, so a run from a hub checkout compares this repo against the hub's payloads and never reads the copies above. Name the target repository explicitly, since the command defaults to whichever repository the shell is sitting in. Two modes over the GitHub API. `configure.sh apply owner/repo release|operational` creates-or-updates the settings, the Dependabot security features, and the rulesets idempotently (a full-payload update). `configure.sh check owner/repo release|operational` is the read-only inverse and exits non-zero on any drift, with the ruleset and settings assertions driven by the committed payloads so they stay repo-agnostic (rule presence, merge methods, and required checks, not a byte diff, so a GitHub-normalized stored ruleset does not false-positive). The command defaults to `apply`, the repo to the current one, and the model to the registry `workflowModel` lookup, falling back to inference from the carried `develop` payload where there is no registry to consult and aborting rather than guessing on an ambiguous layout. Pass the model rather than leaving it to the lookup. A hub checkout has the registry, so a repo the registry does not yet name resolves through `defaults.workflowModel` to `release`, which applies the release `develop` ruleset to an operational repo. Both arguments are optional to the parser rather than to the caller, which is the hazard: a lone positional naming a model is read as the model, so `configure.sh check operational` resolves against whichever repository the shell is in rather than erroring on a missing target. From eeca1b343d5dc530988f63bc5a9bd16ad7ea5eb4 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Mon, 10 Aug 2026 09:13:51 -0700 Subject: [PATCH 2/5] Sort Ruleset Rules on Content, the Key audit.py Uses The snippet says it compares the same subset as spec/audit.py, and it sorted rules by .type where audit.py sorts by each rule's whole content. Two rules sharing a type stay in input order under a .type sort, so a reordered pair reads as drift on a ruleset that matches. Verified order-insensitive by normalizing main.json against a copy with its rules array reversed, which now compares identical, and re-run against four repos across both workflow models. Co-Authored-By: Claude Opus 5 (1M context) --- AUDIT.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/AUDIT.md b/AUDIT.md index 8c42fd24..fae34ac8 100644 --- a/AUDIT.md +++ b/AUDIT.md @@ -74,11 +74,13 @@ Run [`WORKFLOW.md`][workflow]'s methodology against the repo's **own** Actions: && echo "settings: in sync" || echo "settings: DRIFT" ``` -- **Rulesets** - diff each live ruleset against the committed expected payload with a normalized comparison (sort the order-insensitive `rules[]` before diffing so a reordered but equivalent ruleset does not read as drift). The compared subset is `name`, `target`, `enforcement`, `conditions` and `rules`, and `bypass_actors` sits deliberately outside it, which is the same subset [`spec/audit.py`][audit-runner] compares. Who may bypass a ruleset is a per-repository human decision taken in the UI, no payload declares one, and [`repo-config/configure.sh`][repo-config] treats it that way in both modes, writing the live list back unchanged on `apply` and reporting it without asserting on `check`. Comparing it here would contradict that and report a ruleset finding against every repository that has any bypass actor, which is the field's normal state rather than a deviation: +- **Rulesets** - diff each live ruleset against the committed expected payload with a normalized comparison (sort the order-insensitive `rules[]` on each rule's whole content before diffing, so a reordered but equivalent ruleset does not read as drift). The compared subset is `name`, `target`, `enforcement`, `conditions` and `rules`, and `bypass_actors` sits deliberately outside it, which is the same subset and the same sort key [`spec/audit.py`][audit-runner] uses. Who may bypass a ruleset is a per-repository human decision taken in the UI, no payload declares one, and [`repo-config/configure.sh`][repo-config] treats it that way in both modes, writing the live list back unchanged on `apply` and reporting it without asserting on `check`. Comparing it here would contradict that and report a ruleset finding against every repository that has any bypass actor, which is the field's normal state rather than a deviation: ```sh # bypass_actors stays outside the projection, since no payload declares one and jq cannot sort the null that leaves. - norm='{name,target,enforcement,conditions,rules} | .rules|=sort_by(.type)' + # Rules sort on their whole content rather than on .type, which is the key audit.py sorts by. + # Sorting on .type alone leaves two rules of one type in input order, so a reordered pair would read as drift. + norm='{name,target,enforcement,conditions,rules} | .rules|=sort_by(tojson)' # Model-aware expected payload: an operational repo's develop ruleset diffs against # operational/develop.json (registry workflowModel; the same selection audit.py makes). model=$(jq -r --arg n "" '(.repos[] | select(.name==$n) | .workflowModel) // .defaults.workflowModel // "release"' registry/repos.json) From 30e40d4db363c141996fb0742c46be3d2f84516c Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Mon, 10 Aug 2026 09:22:32 -0700 Subject: [PATCH 3/5] Say Which Sort Key Belongs to Which Tool The comment read "on their whole content rather than on .type, which is the key audit.py sorts by", where the relative clause attaches to .type and so states the opposite of the line above it. Name the function instead, which leaves no antecedent to pick wrong. Co-Authored-By: Claude Opus 5 (1M context) --- AUDIT.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/AUDIT.md b/AUDIT.md index fae34ac8..68350a67 100644 --- a/AUDIT.md +++ b/AUDIT.md @@ -78,7 +78,7 @@ Run [`WORKFLOW.md`][workflow]'s methodology against the repo's **own** Actions: ```sh # bypass_actors stays outside the projection, since no payload declares one and jq cannot sort the null that leaves. - # Rules sort on their whole content rather than on .type, which is the key audit.py sorts by. + # Rules sort on each rule's whole content, matching the key normalize_ruleset in audit.py sorts by. # Sorting on .type alone leaves two rules of one type in input order, so a reordered pair would read as drift. norm='{name,target,enforcement,conditions,rules} | .rules|=sort_by(tojson)' # Model-aware expected payload: an operational repo's develop ruleset diffs against From a96b2b8f50ae90460fe8b08dbe1465edcf96b1df Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Mon, 10 Aug 2026 09:33:23 -0700 Subject: [PATCH 4/5] Canonicalize Rule Keys Before Serializing the Sort Key sort_by(tojson) serializes each rule in its own key order, and the two sides do not share one. The committed payload is written key-sorted by jq -S, and the API returns its own order, so the same rule yields two different sort keys and the arrays sort into two different orders. Measured: the live main ruleset returns type before parameters where the committed file has parameters before type, and sorting on those strings puts pull_request and required_status_checks in different positions. The snippet nonetheless reported in sync, which is the part worth recording. The live side reaches jq through `gh api --jq`, and that is gojq, which emits object keys sorted. So the projection was silently canonicalizing the input and the filter depended on it. Rewriting the same pipeline with raw `gh api` piped to real jq reports drift on a ruleset that matches. canon sorts keys at every depth before tojson, so the sort key no longer depends on which jq implementation produced the input. Verified three ways: the shipped snippet against four repos across both workflow models, the raw-jq path that exposed the defect, and a rules array reversed to confirm order-insensitivity. Co-Authored-By: Claude Opus 5 (1M context) --- AUDIT.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/AUDIT.md b/AUDIT.md index 68350a67..67560522 100644 --- a/AUDIT.md +++ b/AUDIT.md @@ -80,7 +80,9 @@ Run [`WORKFLOW.md`][workflow]'s methodology against the repo's **own** Actions: # bypass_actors stays outside the projection, since no payload declares one and jq cannot sort the null that leaves. # Rules sort on each rule's whole content, matching the key normalize_ruleset in audit.py sorts by. # Sorting on .type alone leaves two rules of one type in input order, so a reordered pair would read as drift. - norm='{name,target,enforcement,conditions,rules} | .rules|=sort_by(tojson)' + # canon sorts keys at every depth before serializing, because the committed payload is written key-sorted and the API returns its own order, so a bare tojson gives the same rule two different sort keys. + canon='def canon: walk(if type == "object" then to_entries | sort_by(.key) | from_entries else . end);' + norm="$canon"'{name,target,enforcement,conditions,rules} | .rules|=sort_by(canon|tojson)' # Model-aware expected payload: an operational repo's develop ruleset diffs against # operational/develop.json (registry workflowModel; the same selection audit.py makes). model=$(jq -r --arg n "" '(.repos[] | select(.name==$n) | .workflowModel) // .defaults.workflowModel // "release"' registry/repos.json) From ff623147c8555c6bc1c882ee7d4a216616b68d00 Mon Sep 17 00:00:00 2001 From: Pieter Viljoen Date: Mon, 10 Aug 2026 09:48:19 -0700 Subject: [PATCH 5/5] Define the Key Sort Without walk, Matching configure.sh canon called walk/1, which arrived in jq 1.6, and spec/host-tools.json declares no jq floor at all. A host on jq 1.5 does not degrade on walk, it fails to compile the filter, so the diff reports drift on every ruleset it never compared. repo-config/configure.sh already defines its own recursion for exactly this reason and says so, so calling walk here reintroduced the two-tools-two- policies defect the rest of this branch removes. The recursion inserts keys in sorted order, so tojson emits them sorted at every depth. Verified identical to the walk version's output on a live payload, then re-run as the shipped snippet against four repos across both workflow models. Co-Authored-By: Claude Opus 5 (1M context) --- AUDIT.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/AUDIT.md b/AUDIT.md index 67560522..cd97cac7 100644 --- a/AUDIT.md +++ b/AUDIT.md @@ -81,7 +81,9 @@ Run [`WORKFLOW.md`][workflow]'s methodology against the repo's **own** Actions: # Rules sort on each rule's whole content, matching the key normalize_ruleset in audit.py sorts by. # Sorting on .type alone leaves two rules of one type in input order, so a reordered pair would read as drift. # canon sorts keys at every depth before serializing, because the committed payload is written key-sorted and the API returns its own order, so a bare tojson gives the same rule two different sort keys. - canon='def canon: walk(if type == "object" then to_entries | sort_by(.key) | from_entries else . end);' + # It recurses rather than calling walk/1, which arrived in jq 1.6, and no declared floor puts a host above that. + # A host on jq 1.5 would not degrade on walk, it would fail to compile the filter and report drift on every ruleset it never compared, which is what repo-config/configure.sh defines its own recursion to avoid. + canon='def canon: . as $in | if type == "object" then reduce (keys_unsorted|sort)[] as $k ({}; . + { ($k): ($in[$k]|canon) }) elif type == "array" then map(canon) else . end;' norm="$canon"'{name,target,enforcement,conditions,rules} | .rules|=sort_by(canon|tojson)' # Model-aware expected payload: an operational repo's develop ruleset diffs against # operational/develop.json (registry workflowModel; the same selection audit.py makes).