diff --git a/docs/branch-review-records/65462d1d993e393eb03e0fc2cefc1b6b50174027a14f1831edb0d266829567aa.record.md b/docs/branch-review-records/65462d1d993e393eb03e0fc2cefc1b6b50174027a14f1831edb0d266829567aa.record.md new file mode 100644 index 0000000000..a2044281a0 --- /dev/null +++ b/docs/branch-review-records/65462d1d993e393eb03e0fc2cefc1b6b50174027a14f1831edb0d266829567aa.record.md @@ -0,0 +1 @@ +| 2026-08-18 | claude/eval-canary-protocol-docs-01bb49 | 3e9905d0ac9d88f22dc88d41639399ae69237cf0 | docs/rag-behaviour/safeguards.md, docs/rag-improvement/README.md — eval-canary pair protocol trigger mechanics and bisection lessons (ledger #TYJ0XP) | approved — docs-only, no code/behaviour change | verify:pr-local (docs-scoped: format:changed, sitemap:check, docs:check-index, docs:check-inventory, docs:check-scripts, docs:check-links, check:branch-review-ledger, check:outstanding-issues, check:ledger-write-discipline) all passed | diff --git a/docs/outstanding-issues-inbox/5899581b-dc23-486d-a6da-e456c5c5f3bc.json b/docs/outstanding-issues-inbox/5899581b-dc23-486d-a6da-e456c5c5f3bc.json new file mode 100644 index 0000000000..ad45d9116a --- /dev/null +++ b/docs/outstanding-issues-inbox/5899581b-dc23-486d-a6da-e456c5c5f3bc.json @@ -0,0 +1,11 @@ +{ + "version": 2, + "id": "5899581b-dc23-486d-a6da-e456c5c5f3bc", + "createdOn": "2026-08-18", + "action": "done", + "payload": { + "id": "#TYJ0XP", + "outcome": "Recorded in docs/rag-behaviour/safeguards.md (eval-canary dispatch/cron-only trigger, gh api dispatch command, denominator caveat, S1b/S1d + #2065/#2088 lessons) and docs/rag-improvement/README.md §5/§6 cross-links; PR opened from claude/eval-canary-protocol-docs-01bb49 at 3e9905d0a.", + "baseRowFingerprint": "27208bbb65830782e093bb6677d42aa887b9bf46066d810a83eee48af67138c2" + } +} diff --git a/docs/rag-behaviour/safeguards.md b/docs/rag-behaviour/safeguards.md index af209c44d7..ac84c3d817 100644 --- a/docs/rag-behaviour/safeguards.md +++ b/docs/rag-behaviour/safeguards.md @@ -63,24 +63,46 @@ keys above relevance, read this folder first. **Also:** do not add `review_due` `unknownCurrentness` / governance-metadata ranking penalties or boosts — that shape is refuted (`refuted-approaches.md` § Refutation 3; ledger `#032`). The eval-canary pair protocol: -1. Baseline: latest green canary on current main (or one dispatch). -2. Change merges to main. (Post runs are post-merge only: the workflow accepts only the - `eval-canary` repository dispatch and always loads code from the default branch — no - `workflow_dispatch`, no `ref` input; both are asserted absent by - `tests/eval-canary-workflow.test.ts`.) -3. Post: one dispatch; gates = recall 1.0/1.0, zero per-case rr regressions. Compare the - pair's `--json-out` artifacts — the command diffs per-case `rr@10`/`contentRR@10` and - exits non-zero on any per-case regression, a non-identical case set, or an unavailable - rank metric: +`.github/workflows/eval-canary.yml` fires only on the `eval-canary` repository dispatch and the +Sunday 18:00 UTC cron (`schedule: cron: "0 18 * * 0"`) — it has no `workflow_dispatch` and no +`ref` input, both asserted absent by `tests/eval-canary-workflow.test.ts`. Every run, scheduled +or dispatched, always loads code from the default branch, so a canary can only ever measure +`main` — there is no way to canary an open branch or PR head directly. + +1. Baseline: the latest green canary on current `main` **before the merge** (a scheduled run or + a prior dispatch). +2. Change merges to `main`. +3. Post: one repository dispatch **after the merge** — + `gh api repos/BigSimmo/Database/dispatches -f event_type=eval-canary` — then compare the + pair's `--json-out` artifacts, both named `eval-canary-output`, one per run: ```bash npm run eval:retrieval:compare -- --fail-on-regression ``` + The command diffs per-case `rr@10`/`contentRR@10` and exits non-zero on any per-case + regression, a non-identical case set, or an unavailable rank metric. Gates: document/content + recall 1.0/1.0, zero per-case reciprocal-rank regressions, and the answer-quality gate clean + — read the denominator from the run's own report rather than assuming a fixed case count; it + has moved as `rag-eval-cases.ts` grew. + 4. Regression → immediate single-commit revert + one confirmation dispatch. Provider-backed dispatches always need explicit user approval (~$1–2 each). +**Reading a canary result:** + +- A red first post-run may be an unrelated non-golden case, not a diff regression — root-cause + with a live probe before blaming the diff. Precedent: the first post-S1b canary (run + `32038751592`) failed one answer-quality case; a re-run (`32039841070`) plus cache-bypassed + live probes showed the fast model's phrasing had taken a different, still-correct branch, not + a retrieval or code fault — retrieval stayed 36/36 identical across both runs. +- A deterministic extractive-path flip is a real regression — bisect by probe and revert in a + single commit. Precedent: interim post run `32097916649` went red on + `agitation-im-po-route-short-terms`; live bisection isolated it to PR #2065 (S1c's + condition-first `for`/`in` binding regex), which was reverted by PR #2088, with confirmation + run `32100681177` restoring green. + ## Rollback proof (2026-07-20) The full cycle was exercised live: merge (#1004) → regression detected by canary #55 within diff --git a/docs/rag-improvement/README.md b/docs/rag-improvement/README.md index 5e08d4a41e..7f46eee116 100644 --- a/docs/rag-improvement/README.md +++ b/docs/rag-improvement/README.md @@ -373,6 +373,10 @@ work packets, the live status table, checklists, and paste-ready prompts. | 9 | B5/B6 Ragas/reranker | conditional | No until separately promoted | | 10 | B7 DSPy | ≥100 labelled cases | No until separately promoted | +Cross-links: `#231` cross-links to `docs/database-remediation-plan.md` Phase 5.2 (re-test after +the trigram-index restore) — satisfied by S1's healthy-latency probes 2026-08-17; `#316` Phase +1.2 found the RPC divergence attribute-only. + ## 6. Verification commands (per PR, smallest first) ```bash @@ -387,7 +391,9 @@ npm run check:production-readiness # domain changes (env flags, answer path) ``` Live canary pairs fire only via the `eval-canary` repository dispatch with explicit owner -approval per run; regression → single-commit revert + confirmation run. +approval per run; regression → single-commit revert + confirmation run. See +`docs/rag-behaviour/safeguards.md` § the eval-canary pair protocol for the exact trigger +mechanics (dispatch/cron-only, no `ref` input) and the comparison command. ## 7. Rollback map