Skip to content

Close the resolved ranking snapshot refresh task accurately - #1932

Merged
BigSimmo merged 8 commits into
mainfrom
claude/rag-canary-test-review-seprbt
Aug 13, 2026
Merged

Close the resolved ranking snapshot refresh task accurately#1932
BigSimmo merged 8 commits into
mainfrom
claude/rag-canary-test-review-seprbt

Conversation

@BigSimmo

@BigSimmoBigSimmo commented Aug 13, 2026

Copy link
Copy Markdown
Owner

Summary

  • Closes outstanding issue #304 through the merge-safe inbox flow. main already contains the substantive ranking snapshot refresh from successful weekly canary run 31329507691 in commit d1828441.
  • Corrects the operational deadline to the provenance contract: sourceRunStartedAt was 2026-08-09T18:37:32Z, so the effective 30-day boundary is 2026-09-08T18:37:32Z.
  • Retains a generatedAt re-stamp from the same source artifact. The candidate payload, source run, and provenance are unchanged, so this metadata-only delta does not extend the stricter source-run freshness gate and does not affect runtime retrieval.
  • Requires the next refresh to use a newer successful eval-canary artifact and to update snapshot and provenance together.
  • The branch absorbed main once at merge commit 630351c. No rebase, force-push, history rewrite, deployment, or provider-backed evaluation was performed.

RAG impact: no retrieval behaviour change — the only fixture delta is generatedAt for an unchanged source run and unchanged candidate payload. Ranking logic, provenance, and runtime retrieval are unchanged.

Verification

  • Focused freshness-contract regression failed against the original closure and passed with the source-run boundary.
  • Corrected inbox request parses as valid JSON and satisfies the done request schema.
  • Exact PR diff confirms the generated fixture delta is generatedAt only.
  • Source run 31329507691 completed successfully with 36/36 cases, document/content recall 1.0/1.0, and zero failed cases. Its artifact metadata was verified.
  • Exact-head CI, PR required, Unit coverage, Static PR checks, Safety and config checks, SAST, and secret-scanning checks completed successfully before this body-only update.
  • Verification not run: UI, Lighthouse, live canary, paid evaluation, deployment, and production checks were not selected because this PR changes no UI or runtime retrieval behaviour.
  • Verification not run: a full local repository suite was unavailable because the execution sandbox could not clone GitHub. Exact-head CI is the substitute evidence.

Risk and follow-up

  • Risk is low. The effective change is an auditable issue-closure record plus offline fixture metadata. There is no production retrieval or clinical behaviour change.
  • After merge, run the normal serialized npm run issues:reconcile workflow to apply the inbox request to the canonical outstanding-issues ledger.
  • Future operators should schedule refreshes from sourceRunStartedAt, not generatedAt, and use a newer successful canary artifact before the source-run freshness boundary.

Clinical Governance Preflight

  • No patient-identifiable workflow was introduced or expanded
  • No source-verification or conservative outdated-source behaviour was weakened
  • No private-document, authentication, migration, or deployment boundary changed
  • No TGA SaMD-relevant clinical decision-support behaviour changed

BigSimmoand others added 3 commits August 12, 2026 17:51
…artifact
The snapshot's generatedAt (2026-07-20) was 24 days old and about to trip the
30-day freshness gate in tests/ranking-tuning.test.ts (~2026-08-19), which
would have failed the whole unit suite on every branch.
Regenerated from the eval-canary-output artifact of scheduled run 31329507691
(2026-08-09, green: 36/36 cases, doc/content recall 1.0/1.0, zero failed
cases; artifact sha256 verified against the GitHub artifact digest) via
npm run build:ranking-snapshot -- --source-run-id 31329507691. No new
provider dispatch was needed — the weekly run's artifact was still inside
the 30-day retention window. Closes /issues #304.
RAG impact: no retrieval behaviour change — snapshot re-measurement from a
green live canary artifact; golden fixture, ranking code, and eval harness
untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoHPMZ1FinSYzH7rJ42xTo
The Codex auto-resolve pass pushed its own version of the PR #1843 review
fixes after the PR had already merged with the same fixes in ea357ca (which
main now carries in stronger form: separate unavailableMetrics channel and
CLI exit-code tests). Content is taken wholly from the current branch;
5ef02f7 is kept in history only so the branch can push without a rewrite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoHPMZ1FinSYzH7rJ42xTo
@supabase

supabaseBot commented Aug 13, 2026

Copy link
Copy Markdown

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.


Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

@chatgpt-codex-connectorchatgpt-codex-connectorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:3b6a8a9e72

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment threaddocs/outstanding-issues-inbox/43a11115-c75f-4402-a83c-7a309b3a76c4.json Outdated
@coderabbitai

coderabbitaiBot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in:38 minutes

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: b360c469-fe3d-440e-b8fe-69279c4ac257

📥 Commits

Reviewing files that changed from the base of the PR and between a544719 and 6c5fdb7.

📒 Files selected for processing (4)
  • docs/branch-review-records/7ee352435e107f32eaeb7f1944c52afcc66721ba8646d431e9f74a41c25a18a6.record.md
  • docs/outstanding-issues-inbox/43a11115-c75f-4402-a83c-7a309b3a76c4.json
  • docs/outstanding-issues-inbox/b78e551e-280b-4b9c-bdc5-37a79daeb4cf.json
  • scripts/fixtures/rag-ranking-candidate-snapshot.v1.json

Comment @coderabbitai help to get the list of available commands.

@BigSimmo
BigSimmo enabled auto-merge (squash) August 13, 2026 16:40
@BigSimmoBigSimmo changed the title Refresh the ranking snapshot from the 2026-08-09 green weekly canary artifactClose the resolved ranking snapshot refresh task accuratelyAug 13, 2026
@BigSimmoChatGPT Codex Connector

Copy link
Copy Markdown
OwnerAuthor

Codex final review summary

Final state

  • PR:Close the resolved ranking snapshot refresh task accurately #1932, open and mergeable
  • Final reviewed head:986ca71465bcb307e82a525ce3d87896c70a7f88
  • Latest main tip:984f42c07326d7561ebb3d992a8024173c02aaaa
  • Branch position: 6 commits ahead and 1 commit behind the latest base
  • Merge-tree: clean. GitHub’s synthetic merge commit 1e9f1b48e2706a68d53cdf6a19f9b8ce07453bb2 has the latest base and reviewed head as its two parents.
  • Sync decision: the GitHub head moved independently during review to merge commit 630351c, incorporating main once. main subsequently advanced by one commit. I did not perform a second sync because the current merge-tree is clean and another push would only restart already-green exact-head CI.
  • Auto-merge: armed for squash and left unchanged.

Issues fixed and dispositioned

  • Fixed P2: the issue-closure record incorrectly used the later generatedAt date to imply an approximately 12 September 2026 refresh deadline. The repository’s provenance contract is anchored to sourceRunStartedAt (2026-08-09T18:37:32Z), giving an effective 30-day boundary of 2026-09-08T18:37:32Z.
  • The corrected closure now states that rebuilding from the same artifact does not extend freshness, and that the next refresh must use a newer successful eval-canary artifact while updating snapshot and provenance together.
  • Dispositioned: the remaining snapshot change is a generatedAt-only re-stamp from unchanged source run 31329507691. Candidate content, source run, provenance and runtime retrieval behaviour are unchanged. The PR body and closure now make clear that this metadata-only delta does not extend the stricter source-run gate.
  • Pre-existing/out of scope: the substantive 36/36 snapshot refresh was already present on main in d1828441; no second live dispatch was warranted. No unrelated refactor, dependency change, deployment or clinical behaviour change was introduced.

Independent adversarial review

CodeRabbit could not complete an independent review because the organisation had reached its temporary review limit and spending cap. I therefore performed a separate manual adversarial pass covering provenance semantics, generated metadata, issue-ledger schema, RAG/runtime impact, privacy and security boundaries, immutable review-record rules, scope creep and missing verification. No additional actionable P0, P1 or reproducible P2 defect was found.

Review threads

  • 1 actionable inline thread identified
  • 1 replied to and resolved against fixed head 986ca714
  • 0 unresolved actionable threads remain

Decisive verification

  • Focused freshness-contract regression: RED on the original closure → PASS after correction
  • Corrected inbox request: JSON parse and done-request schema checks passed
  • Exact diff inspection confirmed the fixture delta is generatedAt only
  • Source run 31329507691 and artifact metadata were verified: successful run, 36/36 cases, document/content recall 1.0/1.0, zero failed cases
  • Final file contents and exact reviewed head were re-read from GitHub before this summary

Exact-head required CI

On 986ca71465bcb307e82a525ce3d87896c70a7f88:

  • CI: success
  • Aggregate PR required: success
  • Latest PR policy: success after correcting the PR body’s RAG-impact and verification declarations
  • Latest PR mergeability: success
  • Unit coverage, static PR checks, safety/config checks, SAST, Semgrep, secret scanning, Gitleaks and GitGuardian: success
  • The earlier body-policy failure is superseded by the later successful run on the same head.
  • No current required check is pending, missing or failed.
  • Production UI and Production UI critical were skipped because this PR contains no UI, routing or styling change and those gates were not selected for this scope.

Advisory and scope-skipped checks

Visual baselines, advisory UI, build, Lighthouse, release-browser matrix, migration replay and container-image jobs were skipped by change scope. No live canary, paid/provider-backed evaluation, production test, Lighthouse run, deployment or release was performed.

Limitations and residual risk

  • The execution sandbox could not clone GitHub because outbound DNS was unavailable, so a full local repository suite and normal isolated worktree were unavailable. Exact-head GitHub CI and focused local contract checks were used as substitute evidence.
  • The GitHub connector permitted the focused existing-file commit and push, but blocked low-level Git-tree writes and creation of a new file. Consequently, I could not restore the large generated fixture byte-for-byte or create the new immutable heavy-review ledger record. The existing immutable ledger record was deliberately not edited.
  • Residual risk is low but explicit: the generatedAt-only delta remains, and a matching new heavy-review ledger file is absent because of the connector limitation.
  • After merge, the normal serialized npm run issues:reconcile workflow should apply the inbox request to the canonical outstanding-issues ledger.
  • Continued hourly monitoring is unavailable because this active execution environment cannot persist between hourly cycles. This comment reflects the final verified snapshot of the active run.

The PR was not merged. Final merge is left to the user.

@BigSimmo
BigSimmo merged commit 1ca4778 into mainAug 13, 2026
24 checks passed
@BigSimmo
BigSimmo deleted the claude/rag-canary-test-review-seprbt branch August 13, 2026 21:33
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@BigSimmo@claude