Uh oh!
There was an error while loading. Please reload this page.
VERA 2.0: Multi-target, Multi-turn Evaluator - #182
Draft
luca-belli wants to merge 67 commits into
Draft
Conversation
…gration phasing - Introduce u/c/j entity vocabulary (user/chatbot/judge), replacing the ambiguous p_ prefix and "patient"/"provider" terminology throughout - Formalize config.json/state.json traceability (immutable config + sha256 sidecar, mutable resume state) and the orthogonal generation/judging config split - Add use cases 5 (pool) and 6 (resume), resolve open questions from the PR thread (multi-folder judge output, multi-rubric layout, folder naming, provider-vs-model identifiers) - Split migration into 4 phases: cosmetic wrapper, scoring split, multi-rubric support, substantial refactor - Remove stale "why async judges are not async?" TODO
… policy - Phase 1 is no longer "cosmetic" -- it fully replaces the top-level scripts with the new CLI/config surface (-u/-j/--sample/--config) from day one, using an informal config.json shape until Phase 3 formalizes it as a stable interface - Split scoring-split and multi-rubric-support into separate phases (5 total); multi-rubric no longer depends on artifacts that don't exist until a later phase - Add per-phase "Done when" acceptance bars, incremental import-linter enforcement instead of all-at-once, explicit legacy-output-data and rollback stances - Introduce "stable interfaces" concept: ABCs/Role/naming module/config schema require a design doc to change, enforced via new CODEOWNERS entries -- codebase is optimized for agent coding, so a small set of rarely-changed files get higher scrutiny than the rest
- Phase 0's rubric-loading fix moves to a reusable library helper instead of judge.py's soon-to-be-deleted main(), so Phase 1 reuses it rather than throwing it away - Introduce the rubric bundle manifest format (rubric/prompt files + informational persona links) -- previews the real judging.rubrics[] config shape instead of a throwaway CLI convention - judging.rubrics is a list from day one at every phase (not just Phase 3), closing the same schema-break risk one phase earlier - vera judge explicitly exposes --rubric in Phase 1's CLI surface - Phase 1 requires a parity/regression test suite against the scripts it replaces, and deletes them outright at phase end (no lingering deprecation-stub period, which also removes a MUST-NOT contradiction) - Phase 4's acceptance bar now also verifies single-rubric backward compatibility, not just the new multi-rubric case - Every phase's Done-when now implicitly covers keeping README/AGENTS docs current, stated once instead of per-phase
vera generate drives a multi-turn, non-deterministic LLM conversation -- exact-output parity testing against generate.py isn't meaningful the way it might be for more structural code, and the non-determinism compounds turn over turn. Split Phase 1's testing bar into structural parity (automated, via the existing mock_llm.py harness) and live-LLM manual spot-checks (explicitly not something pytest can certify), so this doesn't get silently treated as fully covered by a single test suite.
…ongs Phase 1 only replaces the CLI front-end -- generate_conversations/judge internals are untouched, so structural parity testing is sufficient and the risk is low. Phase 5's workers/ unification is what actually rewrites the execution engine (both runners move off independent hand-rolled asyncio queues onto the shared workers/ queue/dispatch), which is where concurrency/timing regressions could actually appear. The two-part testing approach (structural parity + manual live-LLM spot-check) now attaches to Phase 5's Done-when instead.
…nfig overlap, provider concurrency, ESCALATE governance - scripts/pool_vera_scores.py's disposition was silently dropped from the original 8-phase plan when it collapsed to 6 phases -- checked again in Phase 5 - Explicitly acknowledge that the Phase 3 naming-scheme swap breaks anything outside vera.py that parses the old p_*/j_* pattern directly - State the separation of concerns between the rubric bundle manifest (what a rubric is) and config.json (how to run it), since they now overlap in subject matter - Flag per-provider concurrency limits as a real Phase 5 requirement -- workers/ enables genuine parallel fan-out with no cap yet - Clarify that documenting a phase here does not pre-clear its ESCALATE requirement; every phase still needs fresh sign-off before it starts
- generate_conversations/ -> generate/, scoring/ -> score/ (judge/ was already consistent) across architecture.md and vera-cli-use-cases.md - Remove docs/architecture-plan.md (superseded by architecture.md, removal was its own stated intent once the target doc was done) - Remove TODO (its items are now resolved or tracked in the migration plan's open items)
…tries Fixes findings from a code review of PR #170 plus two open Copilot comments: the naming-module contradiction in docs/architecture.md, CODEOWNERS entries pointing at nonexistent files, a dead architecture-invariants.md stub, duplicated/drifted ESCALATE guidance in AGENTS.md, ambiguous "until migration" phrasing, and the generate.py/judge.py vs vera.py framing contradiction in README.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…tries Fixes findings from a code review of PR #170 plus two open Copilot comments: the naming-module contradiction in docs/architecture.md, CODEOWNERS entries pointing at nonexistent files, duplicated/drifted ESCALATE guidance in AGENTS.md, ambiguous "until migration" phrasing, and the generate.py/judge.py vs vera.py framing contradiction in README.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…tion) New storage/ package with a StorageBackend ABC (write/read/exists on raw bytes+keys) plus LocalFilesystemStorage as the default -- same interface-with-implementations idiom already used by llm_clients/ and workers/, so a future non-local backend (S3) is a new implementation, not a rewrite. Explicitly orthogonal to phases 0-5: no dependency on, and nothing depends on it, so it can be pulled earlier or run in parallel if a concrete need shows up. Also fixed a stale generate_conversations/ reference left in CODEOWNERS from the earlier package rename.
rubric-navigation-in-code, non-developer data/ rationale - config.json.sha256 is one canonical computation shared by the run-id folder name and the sidecar -- never two independent values, and never embedded in config.json's own filename - vera pipeline --target <name> is the one deliberate exception where a rubric bundle manifest's personas become authoritative for both generation and judging in one shot; every other invocation keeps generation/judging orthogonal - Rubric navigation (which question comes next) is code-owned (QuestionNavigator), never inferred by the judge LLM from prompt text - Rubric/persona content in data/ is explicitly required to stay outside code, since CLEO must be usable by non-developers
…on, run nicknames, phase testing Addresses Emily's 2026-07-25 re-review round and the earlier RFC comments: add the missing -c/--chatbot selection (CLI, config.json, Domain model), anchor config.json paths to $ROOT instead of CWD (mirroring the manifest's own-folder convention), mirror --target into the input config with its mutual-exclusivity rule, add human-readable run nicknames alongside the sha, split Phase 5's live-LLM testing into an automated tolerance-based smoke tier and a qualitative manual tier, clarify that new rubrics only need Phase 0-4, spell out the design-doc enforcement mechanism (CODEOWNERS + CI gate), and note that judge/score still work against old-layout data via explicit paths even though resume does not.
Fixes two stale/wrong entries (run-id naming lost its nickname segment, AD-23 overclaimed that old-layout data can't be read at all) and adds two new ADs the spine was missing entirely: chatbot selection (AD-27) and config vs. manifest path resolution (AD-28). Extends AD-15 with the concrete design-doc enforcement mechanism and AD-21 with the target field now mirrored into config.json, so every prose rule added to architecture.md and vera-cli-use-cases.md today has a citable AD.
Today's additions (chatbot flag, $ROOT path rule, nicknames, phase testing split) are still design-stage decisions. Flag both docs as subject to revision once implementation surfaces things the design discussion alone couldn't.
Fills the gap between the mechanical live smoke test (did it run, same shape) and the manual spot-check (does it read well): embed old-engine vs new-engine conversations for matching (persona, config) pairs and compare cosine similarity against a calibrated old-vs-old baseline, so content drift is caught statistically without requiring exact wording match or relying on manual review alone.
…guity generation has two competing LLM roles (chatbot, user); a bare "models" field didn't say which. judging keeps "models" since only one LLM role exists there, so the name is already unambiguous in that block.
Links to architecture.md from vera-cli-use-cases.md used ../ instead of ./, since both files live under docs/. Also clarify that -c/-u/-j in the new CLI intentionally reuse letters that mean something different in today's generate.py/judge.py, with no coexistence window since those scripts are deleted in Phase 1.
…de too Phase 0 previously only wired judge.py's --rubrics flag to the new bundle-manifest helper, leaving a manifest's personas field completely unused until Phase 1's vera.py --target shorthand. Since a rubric bundle manifest is meant to attach a rubric and its intended personas as one unit, add generate.py --rubric-manifest as Phase 0's stopgap generation-side counterpart, so the attachment is proven out on both sides of the pipeline before the CLI unification in Phase 1.
RubricConfig.load_bundle() inlined manifest JSON parsing/validation, which only judge.py could reach. Per the updated Phase 0 scope in docs/architecture.md, generate.py needs to read the same manifest's personas list, so the reading/validation logic moves to utils/ (leaf layer) instead -- generate/ and judge/ must never import each other. load_bundle() now delegates to utils.rubric_manifest.load_manifest(); behavior is unchanged, covered by the existing load_bundle tests plus new direct tests for the extracted module.
load_manifest_personas() returned personas entries verbatim instead of
resolving them relative to the manifest's own folder, contradicting
docs/architecture.md's stated rule for manifest paths (the same rule
rubric_file/etc. already follow via RubricConfig.load()). Update
data/rubric_manifest.json's personas entry to the correct
manifest-relative form ("personas.tsv", not "data/personas.tsv") now
that resolution actually happens, and cover both relative and
absolute-path entries with tests.
Caught while reorganizing data/ into per-rubric subfolders on a
downstream branch -- fixing it here instead, since this PR hasn't
merged yet and is where the bug was introduced.Same shape as load_manifest_personas(): another optional field a rubric bundle manifest can carry, read by utils/ so both generate/ and judge/ can reach it without importing each other. Unused until a later PR wires a schema-specific persona context template into generation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
data/rubric_manifest.json now points at data/persona_context_template.txt via the new persona_context_template_file field (accessor added in the prior commit), matching how the personas field was wired into this same manifest ahead of generate.py's --rubric-manifest flag landing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
judge.py --rubrics already loads a rubric bundle manifest's rubric half; generate.py had no way to load the same manifest's personas half, leaving a manifest's rubric+personas attachment unverified on the generation side (docs/architecture.md's updated Phase 0 scope). Add --rubric-manifest to generate.py: resolves the manifest's personas list via utils.rubric_manifest.load_manifest_personas() and threads it through ConversationRunner as persona_prompt_path (a new, previously-hardcoded-to-data/personas.tsv parameter) instead of the fixed default. Mirrors judge.py's own "first entry wins, warn on extras" handling for manifests listing more than one persona file. Also wires --rubric-manifest through run_pipeline.py's generation step, alongside its existing --rubrics (judging) flag.
VERA 2.0 Architecture
The shared data/persona_prompt_template.txt hardcoded the SI backstory
block, so a second rubric schema (e.g. PHQ9) couldn't reuse it with its
own persona fields. Split it into a shared template (stylistic/behavioral
instructions, with a {persona_context} placeholder) and a schema-specific
context template (data/persona_context_template.txt, selected via the
manifest's persona_context_template_file field added in the prior PR).
generate_conversations/utils.py's load_prompts_from_csv() now requires a
persona_context_template_path, formats it against the TSV row first, and
validates upfront that the context template's placeholders all exist as
CSV columns (previously a missing column silently skipped the row with a
printed warning). generate.py resolves this path from --rubric-manifest
the same way it already resolves persona_prompt_path, defaulting to
data/persona_context_template.txt when no manifest is given.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>Everything currently in data/ (rubric.tsv, rubric_prompt_beginning.txt, question_prompt.txt, rubric_manifest.json, personas.tsv) is specific to the SI rubric -- once a second rubric type (e.g. PHQ9) exists, data/ needs a subfolder per rubric rather than one flat, implicitly-SI set of files. Move all five into data/SI/ and update every hardcoded default/reference across generate.py, judge.py, run_pipeline.py, judge/score.py, judge/score_utils.py, scripts/, README.md, docs, and tests.
--rubrics/--rubric-manifest require a full manifest path; there's no way to select a rubric by bare name (e.g. typing "SI" anywhere on the command line) until --target lands on the future vera.py CLI. Also documents generate.py --rubric-manifest, which wasn't in the README at all.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…elper feat: add RubricConfig.load_bundle() for rubric bundle manifests
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
feat: wire --rubrics flag to RubricConfig.load_bundle()
…allback judge/score_utils.py loaded the rubric at import time from a cwd-relative path, so importing the module from outside the repo root raised FileNotFoundError. Resolve it from __file__ instead. scripts/pool_vera_scores.py's programmatic fallback still pointed at the pre-move data/personas.tsv. Since the call site guards on is_file(), risk level analysis silently skipped for callers that did not pass personas_tsv. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…o-si-folder refactor: move the SI rubric bundle into data/SI/
Add note about including SHA of config files to avoid silent failures.
…factor/decouple-si-rubric
…ERA-MH into refactor/decouple-si-rubric
Refactor: decouple rubric logic from SI structure
fix: preserve scores on final assign end
docs: define design record conventions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.