VERA 2.0: Multi-target, Multi-turn Evaluator

.github/CODEOWNERS

@@ -0,0 +1,18 @@
+# Architecture and domain packages require maintainer review.
+/docs/architecture.md @SpringCare/vera-mh-maintainers
+/docs/vera-cli-use-cases.md @SpringCare/vera-mh-maintainers
+/llm_clients/ @SpringCare/vera-mh-maintainers
+/judge/ @SpringCare/vera-mh-maintainers
+/generate/ @SpringCare/vera-mh-maintainers
+/run_pipeline.py @SpringCare/vera-mh-maintainers
+/openspec/ @SpringCare/vera-mh-maintainers
+/pyproject.toml @SpringCare/vera-mh-maintainers
+
+# Stable interfaces -- changing these requires a design doc (see docs/architecture.md#stable-interfaces-agent-coding-optimization)
+/llm_clients/llm_interface.py @SpringCare/vera-mh-maintainers
+/utils/role.py @SpringCare/vera-mh-maintainers
+/utils/naming.py @SpringCare/vera-mh-maintainers
+/utils/conversation_layout.py @SpringCare/vera-mh-maintainers
+# /workers/queue.py, /utils/config_schema.py, and /storage/storage_backend.py will be
+# added here once each file exists (Phase 5, Phase 3, and Phase S respectively --
+# see docs/architecture.md#migration-from-current-layout)

AGENTS.md

@@ -29,19 +29,30 @@ cp .env.example .env # Add API keys (ANTHROPIC_API_KEY, OPENAI_API_KEY, et
 
 | Area | Key paths | When to edit |
 |------|-----------|--------------|
-| **Generation** | `generate.py`, `generate_conversations/` | Conversation simulation, turns, personas |
-| **Judging** | `judge.py`, `judge/` | Rubric scoring, TSV output, question navigation |
+| **CLI** | `vera.py` | Subcommands: generate, judge, score, pool, pipeline |
+| **Generation** | `generate_conversations/` | Conversation simulation, turns, personas |
+| **Judging** | `judge/` | Rubric scoring, TSV output, question navigation |
 | **LLM providers** | `llm_clients/`, `llm_clients/llm_factory.py` | New models, custom HTTP/API providers |
-| **Pipeline** | `run_pipeline.py`, `scripts/` | End-to-end generate → judge → score workflows |
+| **Pipeline helpers** | `scripts/` | Pooling and automation until absorbed into `vera pool` |
 | **Data** | `data/` (personas, rubrics) | Evaluation inputs (committed) |
 | **Output** | `output/` (gitignored) | Generated transcripts, evaluations, logs |
 | **Config** | `utils/model_config_loader.py`, `llm_clients/config.py` | Model name resolution, API keys |
 | **Shared utils** | `utils/` | Naming, logging, conversation layout |
 
-**Entry points:** `generate.py` (simulate), `judge.py` (evaluate), `run_pipeline.py` (full workflow), `judge/score.py` (scoring/visualization).
+**Entry point (target):** `vera.py` subcommands only. Legacy scripts (`generate.py`, `judge.py`, `run_pipeline.py`) are deleted entirely at the end of Phase 1 of the migration (not the full 6-phase migration) — see [docs/architecture.md](docs/architecture.md#migration-from-current-layout).
 
 **Temporary experiments:** `tmp_tests/` (not committed). **Permanent tests:** `tests/`.
 
+## Architecture compliance
+
+Read [docs/architecture.md](docs/architecture.md) before structural changes. See its [ESCALATE section](docs/architecture.md#escalate-stop-and-ask) for when to stop and ask, and [Enforcement](docs/architecture.md#enforcement) for the pre-push verification commands.
+
+When an architecture escalation requires a design record, create it under
+[`docs/design/`](docs/design/) and follow the required format and lifecycle in
+[`docs/design/README.md`](docs/design/README.md). Keep current normative rules in
+the architecture documentation; design records preserve rationale and
+consequences without duplicating that contract.
+
 ## Testing
 
 The project uses [pytest](https://docs.pytest.org/) with unit and integration tests under `tests/`. Coverage is enforced (`--cov-fail-under=30` in `pyproject.toml`).
@@ -76,28 +87,39 @@ uv run pytest tests/integration/
 
 ## Key Commands
 
+Target CLI (`vera.py` — not implemented yet; use legacy commands below until Phase 1 of the migration completes):
+
 ```bash
-# End-to-end pipeline (preferred for full workflows)
-uv run python run_pipeline.py \
+# End-to-end pipeline (target)
+uv run python vera.py pipeline \
  --user-agent claude-sonnet-4-5-20250929 \
  --provider-agent gpt-4o \
  --runs 1 \
  --turns 10 \
  --judge-model claude-sonnet-4-5-20250929 \
  --max-personas 5
 
-# Generate conversations only
-uv run python generate.py \
- -u claude-sonnet-4-5-20250929 \
- -p gpt-4o \
- -t 6 -r 1
+# Generate / judge / score (target)
+uv run python vera.py generate -u claude-sonnet-4-5-20250929 -p gpt-4o -t 6 -r 1
+uv run python vera.py judge -f output/{YOUR_P_RUN}/ -j claude-sonnet-4-5-20250929
+uv run python vera.py score -r output/{YOUR_P_RUN}/evaluations/{YOUR_J_RUN}/results.csv
+```
+
+Legacy (current implementation):
+
+```bash
+uv run python run_pipeline.py \
+ --user-agent claude-sonnet-4-5-20250929 \
+ --provider-agent gpt-4o \
+ --runs 1 \
+ --turns 10 \
+ --judge-model claude-sonnet-4-5-20250929 \
+ --max-personas 5
 
-# Judge/evaluate an existing generation run
-uv run python judge.py \
- -f output/{YOUR_P_RUN}/ \
- -j claude-sonnet-4-5-20250929
+uv run python generate.py -u claude-sonnet-4-5-20250929 -p gpt-4o -t 6 -r 1
+uv run python judge.py -f output/{YOUR_P_RUN}/ -j claude-sonnet-4-5-20250929
 
-# Recommended published-score profile (scripted)
+# Recommended published-score profile (scripted; legacy)
 ./scripts/run_recommended_vera_pipeline.sh <provider-agent-model>
 
 # Development
@@ -119,7 +141,7 @@ Use dated model IDs (e.g. `claude-sonnet-4-5-20250929`) as in README; shorthand
 - **Formatting:** `uv run ruff format .`
 - **Linting:** `uv run ruff check .`
 - **Type checking:** `uv run pyright` (basic mode)
-- **Pre-commit:** `pre-commit install` — see `docs/pre-commit-hooks.md`
+- **Pre-commit:** `pre-commit install` — see [docs/pre-commit-hooks.md](docs/pre-commit-hooks.md)
 - Configuration: `pyproject.toml`
 
 ## Git Conventions
@@ -151,7 +173,9 @@ One canonical home per concern — cross-link, don't copy paragraphs.
 
 | Doc | Audience | Use for |
 |-----|----------|---------|
-| [README.md](./README.md) | Humans | Setup, CLI usage, output layout, detailed architecture |
+| [README.md](./README.md) | Humans | Setup, CLI usage, output layout |
+| [docs/architecture.md](./docs/architecture.md) | Humans and agents | Target architecture, invariants, layer model |
+| [docs/design/](./docs/design/) | Humans and agents | Historical design decisions, rationale, and compatibility consequences |
 | **AGENTS.md** (this file) | All coding agents | Style, architecture map, testing, key commands, git conventions |
 | [CLAUDE.md](./CLAUDE.md) | Claude Code only | Slash commands, `.claude/` maintenance |
 | [docs/](./docs/) | Humans and agents | Topic deep dives (see links below) |
@@ -162,6 +186,8 @@ One canonical home per concern — cross-link, don't copy paragraphs.
 
 ### Links
 
+- **Architecture:** [docs/architecture.md](docs/architecture.md)
+- **Design decision records:** [docs/design/README.md](docs/design/README.md)
 - **Setup, pipeline, output layout:** [README.md](./README.md)
 - **Custom LLM providers:** [docs/evaluating.md](./docs/evaluating.md)
 - **Judge behavior:** [docs/judge.md](./docs/judge.md)

README.md

@@ -77,7 +77,7 @@ Use this when the **provider** you want to evaluate (the mental-health chatbot u
 Use this profile when you want a **reliable VERA-MH score comparable to the published VERA-MH v1.1 scores**:
 
 - **Personas**
- - Use all **100** rows in [`data/personas.tsv`](data/personas.tsv).
+ - Use all **100** rows in [`data/SI/personas.tsv`](data/SI/personas.tsv).
  - Persona mix covers presenting concerns, SI risk, disclosure, and modifiers.
  - Full set probes safety more thoroughly than small persona slices.
  - Full set also tends to reduce score variability vs. smaller persona sets.
@@ -103,7 +103,7 @@ For the [recommended settings](#recommended-settings) (dual user agents, 30 turn
 
 Use the same **provider** model id you would pass to `run_pipeline.py` as `--provider-agent` (the system under evaluation). The script:
 
-- Runs `run_pipeline.py` **twice**: once with **GPT 5.2** as the user agent (`gpt-5.2`) and once with **Claude Opus 4.5** (`claude-opus-4-5-20251101`), each with **30** turns and **1** conversation per persona (all personas in `data/personas.tsv` unless you cap the count).
+- Runs `run_pipeline.py` **twice**: once with **GPT 5.2** as the user agent (`gpt-5.2`) and once with **Claude Opus 4.5** (`claude-opus-4-5-20251101`), each with **30** turns and **1** conversation per persona (all personas in `data/SI/personas.tsv` unless you cap the count).
 - Judges each batch with **GPT 5.4** (`gpt-5.4`).
 - Merges both evaluation runs via `scripts/pool_vera_scores.py` into a **single pooled** folder `j_<judge>__p_.../` (e.g. `j_gpt-5.4x1__p_gpt_5_2+claude_opus_4_5__a_.../`, next to your `p_*` runs by default) containing merged `results.csv`, `pool_metadata.json`, `scores/scores.json`, and the usual score / risk visualizations. Use that pooled folder for headline VERA-MH numbers across both user-agent suites.
 
@@ -191,7 +191,8 @@ uv run python run_pipeline.py --help
 | `-c` | `--max-concurrent` | Maximum number of concurrent conversations (defaults to None (no limit); use this if the provider you're testing times out) |
 | `-w` | `--max-total-words` | Optional maximum total words across all responses in a conversation |
 | `-i` | `--run-id` | Run ID for the conversations (if not provided, a default will be generated) |
-| `-mp` | `--max-personas` | Maximum number of personas to use (limits personas loaded from [data/personas.tsv](data/personas.tsv)) |
+| `-mp` | `--max-personas` | Maximum number of personas to use (limits personas loaded from [data/SI/personas.tsv](data/SI/personas.tsv)) |
+| | `--rubric-manifest` | Rubric bundle manifest to load personas from (e.g. `data/SI/rubric_manifest.json`), instead of the default `data/SI/personas.tsv`. No default -- must be given explicitly. Requires a full manifest path; there is no `--target SI`-style shorthand yet (see note below). |
 | `-psf` | `--provider-speaks-first` | Provider speaks first (default: persona speaks first). max_turns is adjusted so provider speaks last. |
 | `-pfm` | `--provider-first-message` | Static first message from provider (no LLM call for first turn). E.g. `"How are you today?"` Used on turn 0 when `--provider-speaks-first` is set. |
 | `-psp` | `--provider-start-prompt` | Prompt sent to provider LLM when starting the conversation (first turn). Used on turn 0 when `--provider-speaks-first` is set. Default: `"Start the conversation based on the system prompt"` |
@@ -223,14 +224,16 @@ This will generate conversations under `output/<p_* run>/conversations/` by defa
 | `-c` | `--conversation` | Path to a single conversation file to judge (mutually exclusive with `--folder`) |
 | `-j` | `--judge-model` | Model(s) to use for judging (required). Format: `model` or `model:count` for multiple instances. Can specify multiple: `--judge-model model1 model2:3`. Examples: `claude-sonnet-4-5-20250929`, `claude-sonnet-4-5-20250929:3`, `claude-sonnet-4-5-20250929:2 gpt-4o:1` |
 | `-jep` | `--judge-model-extra-params` | Extra parameters for the judge model (optional). Examples: `temperature=0.7,max_tokens=1000`. Default: `temperature=0` (unless overridden) |
-| `-r` | `--rubrics` | Rubric file(s) to use (default: `data/rubric.tsv`) |
+| `-r` | `--rubrics` | Rubric bundle manifest(s) to use (default: `data/SI/rubric_manifest.json`). Only the first is used; multi-rubric support is not yet implemented |
 | `-l` | `--limit` | Limit number of conversations to judge (for debugging) |
 | `-o` | `--output` | Without `--resume`: parent directory where a new `j_*__*` evaluation folder is created. Default: `<gen_run>/evaluations/` when `-f` is a nested generation run with `conversations/`; otherwise `evaluations/` at the repo root (a notice is printed). With `--resume`: the existing `j_*` evaluation folder itself. |
 | | `--resume` | Continue batch judging in an existing evaluation folder: use with `-f` and `-o` pointing at that folder. Skips `(conversation, judge, instance)` jobs whose `.tsv` already exists, then rebuilds `results.csv` from all TSVs there. Not supported with `-c` / `--conversation`. |
 | `-m` | `--max-concurrent` | Maximum number of concurrent workers (default: None (no limit)). Set to a high number or omit for unlimited concurrency |
 | `-pj` | `--per-judge` | If set, `--max-concurrent` applies per judge model. Otherwise, it applies to total workers across all judges. Example: `-m 4 -pj` with two judge models runs up to 4 workers per model (8 total) |
 | `-vw` | `--verbose-workers` | Enable verbose worker logging to show concurrency behavior |
 
+**No `SI`-style shorthand yet:** `--rubrics`/`--rubric-manifest` both require a full path to a rubric bundle manifest (e.g. `data/SI/rubric_manifest.json`) -- typing a bare rubric name like `SI` anywhere on the command line does **not** get expanded to that path. Symbolic-name resolution (`--target SI`) is planned for the future `vera.py` CLI, not these scripts. In the meantime, `data/SI/rubric_manifest.json` is simply the current default for `judge.py`/`run_pipeline.py`'s `--rubrics`, so omitting the flag already gets you SI; to select a *different* rubric folder (once one exists), pass its manifest path explicitly.
+
 **Output from `judge.py`:**
 
 When judge.py is run in batch mode, it writes a `j_*__*` folder (by default under `<generation run>/evaluations/` when using the nested layout). Per-conversation judge logs live in `logs/` inside that run folder.
@@ -297,12 +300,12 @@ The output from this script goes to the `score_comparisons` folder by default.
  ```bash
  uv run python3 scripts/summarize_results.py \
  --results output/{YOUR_P_RUN}/evaluations/{YOUR_J_RUN}/results.csv \
- --rubric data/rubric.tsv \
+ --rubric data/SI/rubric.tsv \
  --out-stats output/{YOUR_J_RUN}/improvement_stats.json \
  --out-md output/{YOUR_J_RUN}/improvement_report.md
  ```
 
-After scoring, use `scripts/summarize_results.py` to turn a judge **`results.csv`** into a structured breakdown of where a provider failed and which rubric questions drove those failures. The script reads per-dimension outcome columns plus `*_yes_question_id` / `*_yes_reasoning` (the rubric branch that triggered a Suboptimal or High Potential for Harm rating), joins question text from **`data/rubric.tsv`**, and emits:
+After scoring, use `scripts/summarize_results.py` to turn a judge **`results.csv`** into a structured breakdown of where a provider failed and which rubric questions drove those failures. The script reads per-dimension outcome columns plus `*_yes_question_id` / `*_yes_reasoning` (the rubric branch that triggered a Suboptimal or High Potential for Harm rating), joins question text from **`data/SI/rubric.tsv`**, and emits:
 
 * **`--out-stats`** — JSON with dimension scores, global failure modes, and per-dimension counts broken down by outcome band and rubric question (including optional judge reasoning exemplars).
 * **`--out-md`** — Markdown **improvement report** with a TL;DR grouped by dimension (High Potential for Harm before Suboptimal), then detailed per-dimension sections with percentages, rubric question text, and sample judge reasoning.
@@ -314,7 +317,7 @@ If you omit both output paths, the script prints a short JSON meta summary and a
 | Flag | Description |
 |------|-------------|
 | `--results` | Path to judge **`results.csv`** (required) |
-| `--rubric` | Rubric TSV for question text and severity (default: `data/rubric.tsv`) |
+| `--rubric` | Rubric TSV for question text and severity (default: `data/SI/rubric.tsv`) |
 | `--out-stats` | Write structured JSON stats here |
 | `--out-md` | Write Markdown improvement report here |
 | `--top-questions` | Max rubric questions listed per outcome band per dimension (default: `12`) |
@@ -429,7 +432,9 @@ VERA-MH simulates realistic conversations between Large Language Models (LLMs) f
 
 ## Architecture
 
-### Core Components
+See **[docs/architecture.md](docs/architecture.md)** for the target layer model, invariants, and single CLI orchestrator (`vera.py`). Below is a quick module reference; CLI usage and output layout are in the sections above.
+
+### Core Components (current implementation)
 
 - **`generate.py`**: Main entry point for conversation generation with configurable parameters
 - **`judge.py`**: Main entry point for evaluating conversations using LLM judges
@@ -460,18 +465,19 @@ VERA-MH simulates realistic conversations between Large Language Models (LLMs) f
  - **`conversation_utils.py`**: Conversation formatting and file operations
  - **`logging_utils.py`**: Comprehensive logging for conversations
 - **`data/`**: Persona and configuration data
- - **`personas.tsv`**: TSV file containing patient persona data
- - **`persona_prompt_template.txt`**: Template for generating persona prompts
- - **`rubric.tsv`**: Clinical rubric for conversation evaluation
- - **`rubric_prompt_beginning.txt`**: System prompt for the judge
- - **`question_prompt.txt`**: Prompt template for asking rubric questions
+ - **`SI/personas.tsv`**: TSV file containing patient persona data
+ - **`persona_prompt_template.txt`**: Shared template for persona behavior
+ - **`SI/persona_context_template.txt`**: SI-specific persona context fields
+ - **`SI/rubric.tsv`**: Clinical rubric for conversation evaluation
+ - **`SI/rubric_prompt_beginning.txt`**: System prompt for the judge
+ - **`SI/question_prompt.txt`**: Prompt template for asking rubric questions
  - **`model_config.json`**: Model assignments for different prompt types
 
 ### Persona System
 
 The system uses a TSV-based approach for managing mental health patient personas:
 
-#### Persona Data Structure (`data/personas.tsv`)
+#### Persona Data Structure (`data/SI/personas.tsv`)
 Each persona includes:
 - **Demographics**: Name, Age, Gender, Background
 - **Mental Health Context**: Current mental health situation
@@ -481,7 +487,9 @@ Each persona includes:
 - **Sample Prompt**: Example of what they might say
 
 #### Prompt Templating (`data/persona_prompt_template.txt`)
-Uses Python string formatting to inject persona data into a consistent prompt template, ensuring realistic and consistent behavior across conversations.
+The manifest selects a schema-specific context template, such as
+`data/SI/persona_context_template.txt`. Persona values are formatted into that
+context, then inserted into the shared `{persona_context}` placeholder.
 
 ### Structured Output System
 
@@ -572,7 +580,7 @@ uv run python generate.py
 ```
 
 The script will:
-1. Load personas from `data/personas.tsv`
+1. Load personas from `data/SI/personas.tsv`
 2. Generate conversations between each persona and the agent
 3. Run multiple iterations per persona (configurable)
 4. Save conversations and logs to timestamped folders
@@ -581,7 +589,7 @@ The script will:
 
 ### Custom Personas and Prompts
 
-#### 1. Add New Personas (`data/personas.tsv`)
+#### 1. Add New Personas (`data/SI/personas.tsv`)
 Add new rows to the TSV file with the required fields:
 - Name
 - Age
@@ -603,8 +611,10 @@ Add new rows to the TSV file with the required fields:
 - Recent Triggers and Stressors
 - Final Seed Phrase
 
-#### 2. Modify Prompt Template (`data/persona_prompt_template.txt`)
-Update the template to include new fields or modify behavior patterns.
+#### 2. Modify Prompt Templates
+Update `data/SI/persona_context_template.txt` to select or arrange TSV fields.
+Update `data/persona_prompt_template.txt` only for behavior shared by all persona
+schemas.
 
 #### 3. Configure Models (`model_config.json`)
 Assign models to different prompt types in the JSON configuration.