docs(rag): document the eval-canary pair protocol's dispatch-only trigger and two canary-reading lessons - #2128
Conversation
…trigger and two bisection lessons eval-canary.yml fires only on the eval-canary repository dispatch and the Sunday 18:00 UTC cron, always loads the default branch, and has no workflow_dispatch or ref input — so a canary pair means the latest green run on main before a merge compared against a post-merge dispatch. Document the exact dispatch command, comparison gates, and two precedents for reading a canary result: a red first post-run can be an unrelated non-golden case (S1b/S1d, runs 32038751592 -> 32039841070), while a deterministic extractive-path flip is a real regression requiring bisection and single-commit revert (#2065 -> #2088, run 32097916649). Cross-link the RAG improvement programme's sequencing summary and verification commands sections to this protocol and to two already-satisfied prerequisites (#231 Phase 5.2, #316 Phase 1.2). Closes ledger row #TYJ0XP. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
This pull request has been ignored for the connected project Preview Branches by Supabase. |
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in:12 minutes Limit details: You’ve used all 1 included review currently available under your plan. You completed 101 included PR reviews in the past 7 days; at that activity level, included reviews refill at 1 review per hour. Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
Comment |
Uh oh!
There was an error while loading. Please reload this page.
Summary
eval-canary.ymltrigger mechanics indocs/rag-behaviour/safeguards.md: it fires only on theeval-canaryrepository dispatch and the Sunday 18:00 UTC cron, always loads the default branch, and has noworkflow_dispatch/refinput — so a "canary pair" means the latest green run onmainbefore a merge compared against a post-merge dispatch (gh api repos/BigSimmo/Database/dispatches -f event_type=eval-canary), diffed withnpm run eval:retrieval:compare -- <baseline.json> <post.json> --fail-on-regressionon the twoeval-canary-outputartifacts.32038751592->32039841070, root-caused by a live re-run/probe), versus a deterministic extractive-path flip being a real regression that needs bisection and a single-commit revert (#2065->#2088precedent, interim red run32097916649).docs/rag-behaviour/README.mdagainst the updated protocol text for contradictions — found none; left unchanged.docs/rag-improvement/README.md§5 (Sequencing summary) and §6 (Verification commands): §5 notes#231/#316's already-satisfied cross-links (Phase 5.2 satisfied by S1's healthy-latency probes 2026-08-17; Phase 1.2 found the RPC divergence attribute-only), §6 points at the safeguards.md protocol for trigger mechanics and the comparison command.#TYJ0XP; records the review and queues the closure request (npm run issues:reconcileapplies it after merge).Verification
npm run verify:pr-local— docs-only scope selectedformat:changed,sitemap:check,docs:check-index,docs:check-inventory,docs:check-scripts,docs:check-links,check:branch-review-ledger,check:outstanding-issues,check:ledger-write-discipline; all passed. Decisive lines:docs link check passed: 1903 repo path references resolve.and summaryfailed: (none).npm run verify:ui— not run: no UI, routing, styling, or browser behavior changed.npm run verify:release— not run: not a release/handoff confidence claim.npm run eval:retrieval:quality— not run: no retrieval, ranking, selection, chunking, or scoring code changed; this PR only documents the existing protocol.npm run eval:rag -- --limit 15/npm run eval:quality -- --rag-only— not run: no answer generation, synthesis prompt, or answer post-processing changed.npm run check:production-readiness— not run: no clinical workflow, privacy, environment, Supabase, source governance, or deployment behavior changed.npm run check:deployment-readiness— not run: no deployment startup, hosting, or rollout behavior changed.RAG impact
RAG impact: no retrieval behaviour change — documentation only. This PR edits
docs/rag-behaviour/safeguards.mdanddocs/rag-improvement/README.mdto record how the existing, unmodifiedeval-canary.ymlworkflow already triggers and how two already-landed canary results were correctly interpreted; no code undersrc/lib/rag/**, the eval harness, the golden fixture, or the workflow file itself changed.Risk and rollout
Clinical Governance Preflight
Not applicable — this diff does not touch ingestion, answer generation, search/ranking, source rendering, document access, privacy, production env, or clinical output; it documents the existing eval-canary protocol and cross-links two already-resolved issue rows.
Notes
#TYJ0XPclosure is queued vianpm run issues:done(request file committed on this branch);npm run issues:reconcilefrom a dedicated ledger branch applies it to the canonical ledger after this PR lands.npm run ledger:appendat commit3e9905d0ac9d88f22dc88d41639399ae69237cf0(recorddocs/branch-review-records/65462d1d993e393eb03e0fc2cefc1b6b50174027a14f1831edb0d266829567aa.record.md).