Skip to content

docs(rag): settle the Gate E §2a procedure to what was actually captured#2257

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
BigSimmo merged 3 commits into from
Aug 21, 2026
+14 −8
Merged

docs(rag): settle the Gate E §2a procedure to what was actually captured #2257

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ba294b5
docs(rag): settle the Gate E §2a procedure to what was actually captured
claude Aug 21, 2026
e4613b5
Merge branch 'main' into claude/gate-e-verdict-record-docs-pttz4g
claude Aug 21, 2026
f0cf64b
Merge branch 'main' into claude/gate-e-verdict-record-docs-pttz4g
BigSimmo Aug 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 14 additions & 8 deletions docs/rag-improvement/HANDOVER.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -119,14 +119,20 @@ it can verify from GitHub/git state. Keep rows one line.
> v18 `4ea310e48` vs v19 `cdfcbaccd` comparison; the verdict is recorded in the §2 S2 and
> Gate E rows. This section is kept as the reusable procedure for any **future** blinded
> read — it is not outstanding work. Steps 3 and 4 are provider-backed and paid, so
> re-running them requires fresh owner approval.

Compares prompt v18 (commit `4ea310e48`, the S2 baseline canary half) against v19 (current
`main`) on the 30 `answerQualityEvalCases` plus up to ~10 owner-chosen live questions.
`scripts/eval-answer-quality.ts` and `scripts/eval-utils.ts` are byte-identical between
`4ea310e48` and current `main` and every new dump field is defensive, so the updated script
runs unmodified in a v18 worktree. Steps 3 and 4 are **provider-backed and paid** (OpenAI +
Supabase, ~80 cache-bypassed answers, est ~$4–8); everything else is offline.
> re-running them is a fresh capture: it needs new owner approval and produces a new
> capture pair, recorded on its own terms rather than folded into the 2026-08-21 result.

The executed run compared prompt v18 (commit `4ea310e48`, the S2 baseline canary half)
against v19 (commit `cdfcbaccd`) on the 30 `answerQualityEvalCases`; no owner-chosen live
questions were added, so 30 pairs were built with 0 unpaired. A future run may still add up
to ~10 live questions through `--extra-cases`. `scripts/eval-answer-quality.ts` and
Comment on lines +125 to +128

@chatgpt-codex-connectorchatgpt-codex-connectorBotAug 21, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reconcile the closed run with the numbered procedure

The new claim that no owner questions were added makes the preceding assertion that “all eight steps” ran impossible: steps 1–4 require an --extra-cases file, while parseExtraCaptureCases rejects an empty questions array (also pinned in tests/eval-answer-quality.test.ts). Anyone auditing or replaying this paid capture therefore cannot tell what command actually produced the 30-pair result. Document the executed commands without --extra-cases, or distinguish the omitted steps from the future reusable recipe.

Useful? React with 👍 / 👎.

`scripts/eval-utils.ts` were byte-identical between `4ea310e48` and `cdfcbaccd` and every new
dump field is defensive, so the updated script ran unmodified in a v18 worktree — re-confirm
that before reusing the shortcut against a newer after-half. Steps 3 and 4 are
**provider-backed and paid** (OpenAI + Supabase, cache-bypassed answers); the pre-run
estimate was ~$4–8 for ~80 answers, and the executed 30-case run came in at ~$0.66 — itself
an ESTIMATE derived from token counts because `estimated_cost_usd` is null in both dumps,
never a measured cost. Everything else is offline.

1. _(offline)_ Author `.local/gate-e/extra-cases.json` in the main checkout:
`{"questions":[{"id":"live-01","question":"..."}, ...]}` — ids `live-*`; the tool rejects
Expand Down
Loading