Skip to content

fix: reconcile explicit admin deliveries - #3132

Merged
stranske merged 3 commits into
mainfrom
codex/admin-delivery-reconciliation
Aug 16, 2026
Merged

stranske merged 3 commits into
mainfrom
codex/admin-delivery-reconciliation

Conversation

@stranske

Copy link
Copy Markdown
Owner

Summary

  • preserve the default Collab-Admin exclusion from fleet candidate and campaign evidence
  • allow a deliberate one-repository Maint 71 request to reconcile its generated delivery
  • document and test the bounded manual exception

Root cause

Maint 68 generates a stable Collab-Admin delivery with the staging seal contract, but Maint 71 removed Collab-Admin even from an explicit repos request. Gate therefore correctly refused the unsealed PR while its only authorized sealer reported success after processing zero repositories.

Validation

  • 66 Node contract tests passed
  • 25 focused Python workflow tests passed
  • targeted regression test passed after a deliberate return to the old exclusion produced the expected failure
  • Actionlint passed
  • git diff --check passed

The PR is ready for review.

@stranske stranske added agents Agent-related issues automation Automation and workflow automation labels Aug 16, 2026
@cursor

cursor Bot commented Aug 16, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@stranske

Copy link
Copy Markdown
Owner Author

@codex review

@coderabbitai

coderabbitai Bot commented Aug 16, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 49 minutes

Limit details: You’ve used all 1 included review currently available under your plan. You completed 107 included PR reviews in the past 7 days; at that activity level, included reviews refill at 1 review per hour.

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 6c0d2607-468b-4df4-9f55-95df5a495066

📥 Commits

Reviewing files that changed from the base of the PR and between 6e9f703 and 704e46b.

📒 Files selected for processing (6)
  • .github/scripts/__tests__/sync_pr_merge_contract.test.js
  • .github/scripts/maint71_merge_sync_prs.js
  • .github/workflows/maint-71-merge-sync-prs.yml
  • docs/ops/CONSUMER_REPO_MAINTENANCE.md
  • docs/ops/SYNC_DEPENDENCY_CAMPAIGN.md
  • tests/workflows/test_sync_delivery_liveness.py

Comment @coderabbitai help to get the list of available commands.

@stranske-keepalive

Copy link
Copy Markdown
Contributor

Workflow source needed

PR #3132 needs either a linked GitHub issue or one valid non-issue Workflow Source before PR metadata automation can manage it safely.

Please do one of:

  • Add <!-- meta:issue:123 --> or a normal Closes #123 / Related to #123 line.
  • Check one Workflow Source option in the PR body.
  • Add a hidden marker such as <!-- workflow-source:local_request -->, <!-- workflow-source:manual_remote -->, <!-- workflow-source:review_followup -->, <!-- workflow-source:sync_campaign -->, or <!-- workflow-source:dependabot -->.
  • Add a workflow source label such as workflow:source-direct-pr, workflow:source-local-request, workflow:source-review-followup, workflow:source-sync, or workflow:no-automation.

Once a valid source is present, this warning will not be reposted.

@stranske-keepalive

stranske-keepalive Bot commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Automated Status Summary

Head SHA: f7afdbd
Latest Runs: ⏳ pending — Gate
Required contexts: summary
Required: core tests (3.12): ⏳ pending, core tests (3.13): ⏳ pending, docker smoke: ⏳ pending, gate: ⏳ pending

Workflow / Job Result Logs
(no jobs reported) ⏳ pending

Coverage Overview

  • Coverage history entries: 1

Coverage Trend

Metric Value
Current 76.26%
Baseline 85.00%
Delta -8.74%
Minimum 70.00%
Status ✅ Pass

Top Coverage Hotspots (lowest coverage)

File Coverage Missing
scripts/issue_dedup_smoke.py 0.0% 4
scripts/runner_lib/__main__.py 0.0% 3
scripts/validate_template_sync.py 0.0% 81
scripts/langchain/topic_splitter.py 19.1% 57
tools/codex_log_analyzer.py 19.6% 140
scripts/repo_review_round2_runner.py 25.1% 344
scripts/prune_agent_stubs.py 39.7% 26
scripts/repo_review_round1_runner.py 40.7% 133
tools/ensure_workflow_timeout_variables.py 42.1% 74
scripts/sync_label_docs.py 42.9% 64
tools/discover_model_catalog.py 44.8% 55
scripts/repo_review_backlog_scan.py 45.3% 116
scripts/repo_review_body_writer.py 46.5% 86
tools/codex_session_analyzer.py 47.9% 59
scripts/create_verifier_labels.py 48.3% 58

Low Coverage Files (<50.0%)

File Coverage Missing
scripts/issue_dedup_smoke.py 0.0% 4
scripts/runner_lib/__main__.py 0.0% 3
scripts/validate_template_sync.py 0.0% 81
scripts/langchain/topic_splitter.py 19.1% 57
tools/codex_log_analyzer.py 19.6% 140
scripts/repo_review_round2_runner.py 25.1% 344
scripts/prune_agent_stubs.py 39.7% 26
scripts/repo_review_round1_runner.py 40.7% 133
tools/ensure_workflow_timeout_variables.py 42.1% 74
scripts/sync_label_docs.py 42.9% 64
tools/discover_model_catalog.py 44.8% 55
scripts/repo_review_backlog_scan.py 45.3% 116
scripts/repo_review_body_writer.py 46.5% 86
tools/codex_session_analyzer.py 47.9% 59
scripts/create_verifier_labels.py 48.3% 58

Updated automatically; will refresh on subsequent CI/Docker completions.


Keepalive checklist

Scope

No scope information available

Tasks

  • No tasks defined

Acceptance criteria

  • No acceptance criteria defined

@stranske
stranske temporarily deployed to agent-high-privilege August 16, 2026 21:39 — with GitHub Actions Inactive

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 40c3081700

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/scripts/maint71_merge_sync_prs.js Outdated
@stranske

Copy link
Copy Markdown
Owner Author

@codex review

@stranske
stranske temporarily deployed to agent-high-privilege August 16, 2026 22:02 — with GitHub Actions Inactive

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0d949f14f9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docs/ops/SYNC_DEPENDENCY_CAMPAIGN.md Outdated
@stranske

Copy link
Copy Markdown
Owner Author

@codex review

@stranske
stranske temporarily deployed to agent-high-privilege August 16, 2026 22:09 — with GitHub Actions Inactive
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: 704e46b2c0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@stranske
stranske merged commit f7afdbd into main Aug 16, 2026
51 checks passed
@stranske
stranske deleted the codex/admin-delivery-reconciliation branch August 16, 2026 22:25
@stranske stranske added the verify:compare Compare multiple LLM evaluations label Aug 16, 2026
@stranske
stranske temporarily deployed to agent-high-privilege August 16, 2026 22:25 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown
Contributor

Provider Comparison Report

Provider Summary

Provider Model Verdict Confidence Summary
openai gpt-5.6-terra PASS 94% The change cleanly centralizes target selection in selectReconciliationTargets and implements the bounded exception correctly: Collab-Admin remains excluded from normal explicit selections and flee...
anthropic claude-sonnet-5 PASS 75% The PR extracts the target-repo selection logic into a new selectReconciliationTargets function that preserves the default Collab-Admin (and other admin-repo) exclusion for fleet/candidate runs whi...
📋 Full Provider Details (click to expand)

openai

  • Model: gpt-5.6-terra
  • Verdict: PASS
  • Confidence: 94%
  • Scores:
    • Correctness: 9.0/10
    • Completeness: 9.0/10
    • Quality: 9.0/10
    • Testing: 9.0/10
    • Risks: 9.0/10
  • Summary: The change cleanly centralizes target selection in selectReconciliationTargets and implements the bounded exception correctly: Collab-Admin remains excluded from normal explicit selections and fleet/candidate behavior, while a workflow_dispatch delivery request targeting exactly one allowlisted manual-reconciliation repository is permitted. Candidate processing remains derived from the canonical canary list, and multi-repository requests cannot use the exception. The workflow configuration, operational documentation, Node contract coverage, and focused Python workflow coverage were updated. The added contract test covers normal fleet delivery, the permitted single-repository delivery case, candidate behavior, non-manual event/hash combinations, and mixed explicit requests. No meaningful security, compatibility, or maintainability risks are apparent; the exception is constrained by event type, delivery mode, single-target selection, and an explicit allowlist.

anthropic

  • Model: claude-sonnet-5
  • Verdict: PASS
  • Confidence: 75%
  • Scores:
    • Correctness: 8.0/10
    • Completeness: 8.0/10
    • Quality: 8.0/10
    • Testing: 8.0/10
    • Risks: 7.0/10
  • Summary: The PR extracts the target-repo selection logic into a new selectReconciliationTargets function that preserves the default Collab-Admin (and other admin-repo) exclusion for fleet/candidate runs while allowing a narrowly-scoped manual exception: only a single-repo workflow_dispatch request with requestedSyncHash === 'delivery' and the repo explicitly listed in manuallyReconcilableRepos bypasses the exclusion. This matches the stated tasks (preserve default exclusion, allow bounded one-repo override, document/test it). The added Node test suite thoroughly exercises the new function across the 'all' case, single-repo manual override, candidate sync hash (which should never honor the override), various event/sync-hash combinations that must NOT trigger the exception, and a multi-repo request containing the admin repo (which must still be filtered). A Python test and documentation updates were also added. CI checks referenced (JS/Python tests, lint, actionlint) reported success on the merge commit. The main limitation is that the workflow YAML change and full doc diffs were truncated from the review context, so the environment-variable wiring could not be fully confirmed, but the core logic change is sound, well-tested, and appropriately scoped.
  • Concerns:
    • The workflow YAML diff (+3/-0) that presumably wires the new MANUAL_RECONCILIATION_REPOS_INPUT env var was truncated from the review context, so it could not be directly verified that the workflow_dispatch input is properly plumbed through to the script.
    • Acceptance-criteria checkboxes in the PR body remain unchecked, though CI results and code changes suggest the work was completed; this is likely just template formatting rather than an actual gap.
    • The 'targeted regression test passed after a deliberate return to the old exclusion produced the expected failure' criterion is a manual verification step not directly evidenced in the diff, though the new selectReconciliationTargets tests functionally cover the same regression scenario (multi-repo and non-explicit-delivery cases still get excluded).

Agreement

  • Verdict: PASS (all providers)
  • Correctness: scores within 1 point (avg 8.5/10, range 8.0-9.0)
  • Completeness: scores within 1 point (avg 8.5/10, range 8.0-9.0)
  • Quality: scores within 1 point (avg 8.5/10, range 8.0-9.0)
  • Testing: scores within 1 point (avg 8.5/10, range 8.0-9.0)

Disagreement

Dimension openai anthropic
Risks 9.0/10 7.0/10

Unique Insights

  • openai: The change cleanly centralizes target selection in selectReconciliationTargets and implements the bounded exception correctly: Collab-Admin remains excluded from normal explicit selections and fleet/candidate behavior, while a workflow_dispatch delivery request targeting exactly one allowlisted m...
  • anthropic: The workflow YAML diff (+3/-0) that presumably wires the new MANUAL_RECONCILIATION_REPOS_INPUT env var was truncated from the review context, so it could not be directly verified that the workflow_dispatch input is properly plumbed through to the script.; Acceptance-criteria checkboxes in the PR body remain unchecked, though CI results and code changes suggest the work was completed; this is likely just template formatting rather than an actual gap.; The 'targeted regression test passed after a deliberate return to the old exclusion produced the expected failure' criterion is a manual verification step not directly evidenced in the diff, though the new selectReconciliationTargets tests functionally cover the same regression scenario (multi-repo and non-explicit-delivery cases still get excluded).

🔍 LangSmith Traces

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent-related issues automation Automation and workflow automation verify:compare Compare multiple LLM evaluations

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants