docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotion (#401)#445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AndresL230 merged 5 commits into from
Jul 28, 2026
Merged

docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotion (#401) #445

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e4a054f
docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotio…
AndresL230 Jul 28, 2026
befba6f
docs(e2e): stop citing a gitignored planning file from the runbook
AndresL230 Jul 28, 2026
3c65b97
docs(e2e): fix awkward line wrap in runbook
AndresL230 Jul 28, 2026
56511af
docs(e2e): fix runbook per code-review — seam artifact vs real bug, t…
AndresL230 Jul 28, 2026
df2672b
docs(e2e): rewrap code span so the module path doesn't split mid-toke…
AndresL230 Jul 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
313 changes: 313 additions & 0 deletions docs/e2e-exploration.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,313 @@
# Chapter 2: exploratory testing

This is the runbook for Sapling's Chapter 2 — LLM-driven exploratory testing
of the running app. Read this before your first `make explore`.

## 1. What this is

Sapling has two E2E chapters, and they answer different questions.

| | Chapter 1 | Chapter 2 |
|---|---|---|
| What it is | Scripted, deterministic journeys | LLM-driven exploration |
| Where it lives | `frontend/e2e/*.spec.ts` (Playwright) | `scripts/explore.sh`, `.claude/skills/explore/`, `backend/e2e_oracles/` |
| Gates CI? | Yes — `.github/workflows/e2e.yml`, main-only | Never |
| Runs when | Every push to main | On demand, local-only (`make explore` / `/explore`) |
| Judge | Fixed assertions + `support/db.ts` raw SQL | The e2e oracles (`backend/e2e_oracles`) or a reproducible captured failure |

Chapter 2 exists because a scripted suite only ever re-checks what someone
already thought to assert. An LLM with a real browser will click things,
double-submit forms, hit Back mid-flow, and generally behave like a curious
student — surfacing bugs nobody wrote a journey for yet.

The core rule, verbatim from the epic (#403), governs everything below:

> The LLM explores; only oracles (or a reproducible captured failure) make
> something a finding.

That's why every exploration session ends with a deterministic oracle pass,
and why the triage protocol (§7) insists on reproduction before anything
becomes a GitHub issue. The explorer's own prose is a lead, not a verdict.

## 2. Prerequisites

Everything Chapter 1's local stack needs (see
[local-supabase.md](local-supabase.md)), plus two things specific to Chapter 2:

- Supabase CLI on `PATH`
- Rootless Podman (documented) or Docker (auto-detected fallback)
- `backend/venv` (`python -m venv backend/venv && backend/venv/bin/pip install -r backend/requirements.txt`)
- `frontend/node_modules` (`cd frontend && npm ci`)
- `backend/.env` with `APP_ENV=local` (or `test`) — without it, `POST
/api/auth/test-login` 404s and the harness can't mint a session
- The `claude` CLI on `PATH` (drives the headless explorer via `claude -p`)
- Network access for the first run — `npx` downloads and pins
`@playwright/mcp@0.0.78` the first time it's invoked; afterward it's cached

You do **not** need to set a real `GEMINI_API_KEY` or `SEED_RICH` yourself —
`scripts/explore.sh` forces both. A real key in your shell is not
used and never billed by this harness.

## 3. Kick-off: headless

```bash
make explore
```

This runs the full pipeline: boot the stack, mint a session, hand the
explorer prompt to a Playwright-MCP-armed `claude -p`, run the oracle final
pass, then tear everything down. It takes the machine-singleton stack lock
for the whole run (see §10). With warm caches (a repo you've already run
`make e2e-up` or `make explore` on before), the whole thing typically
finishes in a few minutes end-to-end; the first-ever run on a machine can
take closer to ~10 minutes (see §9's cost note — it's the `npx
@playwright/mcp` download plus a cold test-profile Next build, not the
exploration itself).

Env knobs (all optional, read by `scripts/explore.sh`):

| Knob | Default | Meaning |
|---|---|---|
| `EXPLORE_MAX_TURNS` | `40` | Turn budget passed to `claude -p --max-turns`. Bounds both wall-clock and token cost. |
| `EXPLORE_MODEL` | `sonnet` | Model alias passed to `claude -p --model`. |
| `EXPLORE_HEADED` | `0` | `1` = watch the browser drive itself (non-headless); `0` = headless. |
| `EXPLORE_USER` | `rich-user-active` | Which seeded `rich-*` user to sign in as. One of `rich-user-active`, `rich-user-second`, `rich-user-new`, `rich-user-pending`, `rich-user-admin`. |

Example: a slower, watchable run as a different seeded user:

```bash
EXPLORE_HEADED=1 EXPLORE_USER=rich-user-admin EXPLORE_MAX_TURNS=60 make explore
```

Two things worth internalizing before your first run:

- **`.explore/` is wiped at `up`, but only after the lock is acquired.** If
you want to keep a previous run's findings, triage them (§7) before
re-running — a second `make explore` deletes the prior artifacts (except
the in-flight `lock.ok`, which is bookkeeping, not a finding).
- **A crashed run is recoverable.** If a prior `make explore` died mid-run
(killed session, machine sleep, whatever), `make explore-down` (same as
`scripts/explore.sh down`) runs the oracle final pass against whatever is
still up, appends it to `findings.md`, tears the stack down, and releases
the lock — safe to run even with nothing up.

## 4. Kick-off: interactive

```
/explore
```

Run this as a slash command in a Claude Code session opened on this repo.
Unlike `make explore`, you watch (and can steer) the exploration live:

1. The skill runs `scripts/explore.sh up` itself (this takes minutes —
Supabase, migrations, seed, backend, a test-profile Next build — and takes
the same stack lock `make explore` does; if another session holds it, the
skill stops and tells you).
2. It opens a real browser tab (Playwright MCP if configured for the session,
else the Claude-in-Chrome tools) at `http://localhost:3000` and signs in as
`rich-user-active` by minting a session directly from the page.
3. It reads `scripts/explore/explorer-prompt.md` as its own mission briefing
and starts exploring — narrating what it's trying so you can redirect it
at any point. Same persona, same break-things mandate, same
report-never-fix rule, same oracle cadence, same findings format as the
headless run.
4. It finishes with `scripts/explore.sh down` — same oracle final pass,
same teardown, same lock release.

The output is identical in shape to a headless run: `.explore/findings.md`
with explorer-written entries plus the harness-appended oracle pass. The only
difference is you're in the loop instead of reading a transcript after the
fact.

## 5. Outputs

Everything lands in `.explore/` at the repo root (gitignored in full):

| Path | What it is |
|---|---|
| `findings.md` | The running human-readable findings log — explorer-written `### F<N>:` entries plus a harness-appended `## Oracle final pass` section at teardown. Pre-created by the harness with a header so the explorer always has an existing file to append to. |
| `session.log` | The full headless-run transcript: `claude -p --output-format stream-json --verbose`, reformatted by `jq` into readable `[assistant]` / `[tool_use]` / `[tool_result]` / `[result]` lines (each truncated to 500 chars). |
| `session.stderr.log` | `claude -p`'s stderr, kept in its own file so it can never interleave with (and corrupt) the JSON stream `session.log` is parsed from. |
| `oracle-final.txt` / `oracle-final.json` | The final `python -m e2e_oracles` pass, run at teardown while the stack is still up, in both text and JSON form. |
| `traces/` | `@playwright/mcp` session-recording artifacts (page snapshots plus an MCP session directory) — these are MCP session recordings, not Playwright Trace Viewer `.zip` files. |
| `storageState.json`, `mcp.json` | Plumbing, not findings: the minted Playwright storage state and the `--mcp-config` fed to `claude -p`. Useful for debugging why the explorer couldn't reach a page, otherwise ignorable. |
| `lock.ok` | Internal lock-holder bookkeeping (holder-written, self-identifying — its content is the holder process's own PID). Not a finding artifact. |

## 6. The oracles, standalone

The explorer runs the oracles as it goes, and the harness runs one final pass
at teardown — but you can also run them yourself against a stack that's
already up (e.g. mid-`/explore`, or after driving the app by hand):

```bash
cd backend && venv/bin/python -m e2e_oracles [--json] [--check ciphertext|counts|graph|logscan|orphans] [--user ID]
```

- `--check` is repeatable; omit it to run all five (default, sorted order).
- `--json` emits structured `Finding` records instead of the text report —
useful when quoting evidence into a GitHub issue.
- `--user` selects which seeded user's data to check (default
`rich-user-active`).
- Exit codes: `0` clean, `1` findings, `2` infra error (a check crashed, or
itself returned an `oracle-error` finding — treat the whole run's other
results as untrustworthy when this happens, not just the failing check).
- Needs the stack up (`make e2e-up` or `scripts/explore.sh up`) — it hits the
local Postgres directly and the backend's `/api/health`-adjacent surface.
- The `logscan` check allowlists known #439 RAG-indexing log noise — an
allowlisted hit still counts toward "N suppressed" in the summary line, but
doesn't produce a `Finding`.

## 7. Triage protocol

For each entry in `findings.md` (explorer-written or oracle-appended):

1. **Reproduce it.** By hand in a browser, or by re-running the relevant
oracle `--check`. Not reproducible → drop it, with a one-line note in your
triage record of why. Part of reproducing it is asking whether the
"wrong" output is actually the function-mode seam working as designed
(see the worked example below) — that's a drop too, just a different
reason than "couldn't reproduce."
2. **Reproducible and real** → `gh issue create` with the finding block
(steps/expected/actual), the oracle JSON if there is any, and the
`.explore/traces/` path attached. Label it as sourced from exploration so
it's traceable back to a session.
3. **Already-known** (currently: #355, #430, #435, #436, #439, #441) → don't
file a new issue. Add the new evidence as a comment on the existing issue
only if it's actually new evidence (a different surface, a clearer repro,
fresh oracle output) — otherwise it's just noise on an already-tracked bug.
4. **Noise the oracle wrongly flagged** → that's a bug in the oracle itself
(a missing allowlist entry, or a judge that's too strict/wrong). Fix it as
a normal PR against `backend/e2e_oracles/` — don't just delete the finding
from `findings.md` and move on.

Two worked examples from this harness's own acceptance testing show the
difference between a genuine finding and noise the *harness itself*
(not the oracle) produced:

**Genuine, promotable (category 2).** Resuming a tutor session in one
acceptance run produced a real `500` on `POST
/api/graph/<user>/concept-description`, traced through the browser console →
network tab → backend log to `LookupError: SAPLING_MODEL_MODE=function but
no handler is registered for task 'concept_describe'` — `concept_describe`
genuinely has no handler in `backend/agents/function_handlers_e2e.py`. The
`logscan` oracle independently caught the same `500` and traceback in the
backend log, so this has both a UI-observed repro and oracle corroboration.
It isn't on the known-bugs list — exactly the kind of thing this chapter
exists to surface, and a strong promotion candidate (§8).

**Seam artifact — drop it, then improve the harness, not the app (still
category 1).** A different run's explorer flagged what looked like a
"wrong-data" bug: uploading a real syllabus PDF came back with a summary
about gradient descent, category "Lecture notes", and concepts "Gradient
Descent"/"Learning Rate" — content belonging to an entirely different,
already-uploaded document. Alarming-looking, but not a product bug: Chapter
2 always runs the document pipeline in function mode
(`SAPLING_MODEL_MODE=function` /
`SAPLING_FUNCTION_HANDLERS=agents.function_handlers_e2e`, forced by
`scripts/explore.sh`), and
`agents/function_handlers_e2e.py`'s classifier/summary/concepts handlers are
*designed* to return the same fixed constants
(`E2E_DOC_CATEGORY`/`E2E_DOC_ABSTRACT`/`E2E_DOC_CONCEPTS`) for every upload
regardless of the file's real content — the module's own docstring says so
("Fixed output... Echoing request content back would couple the constant to
route-side prompt assembly"). The pipeline was working exactly as designed;
the explorer just didn't know that. **The tell:** the "wrong" text is a
byte-for-byte match to one of `function_handlers_e2e.py`'s `E2E_DOC_*` (or
`E2E_TUTOR_REPLY`, `E2E_QUIZ_*`) constants — here, `E2E_DOC_ABSTRACT`'s
"gradient descent... loss surface... learning rate" wording and the fixed
`lecture_notes` category. If a finding's "wrong" content matches one of
those constants verbatim, it's the seam, not the app — drop it, and if it
recurs, improve `scripts/explore/explorer-prompt.md`'s ground rules to name
the pattern explicitly so a future explorer recognizes it before writing the
stub, rather than filing an issue against the harness's own known-fixed
output.

By contrast, #355 (the graph's duplicated CS subject-root hub) shows up in
the graph oracle on essentially every run against the rich seed data —
that's the known-not-new case category 3 above exists for.

## 8. Promotion pipeline

Triage produces GitHub issues; the pipeline that actually matters is turning
the *right* ones into permanent regression coverage. The epic bar (#403):
**at least 3 findings promoted into Chapter 1 regressions in the first
month.**

Promotion path: issue → a scripted journey in `frontend/e2e/*.spec.ts`,
written in the existing specs' style —

- Fixtures-based `test` (see `frontend/e2e/support/fixtures.ts`), not raw
Playwright boilerplate.
- Assertions against the database go through `support/db.ts` (raw SQL over a
direct Postgres connection) — journeys never hand-roll DB plumbing.
- Any new `data-testid` follows `docs/frontend-testids.md`'s naming
convention; see that doc's "Adding a surface" section for the doc-row +
`eslint.config.mjs` `files`-array update a new surface needs.
`npm run lint:baseline` is only for clearing legacy debt in
`eslint-suppressions.json`, not something a new surface should lean on.

If the promoted journey documents a bug that's still open and must stay red,
ride it in on `test.fixme` with the bug number — exactly how
`frontend/e2e/graph.spec.ts` carries its `#355` assertion today (marked
`fixme`, not relaxed, with companion tests asserting everything #355 does
*not* corrupt).

## 9. Cadence + cost

Suggested cadence: one bounded headless run a week, plus one after any merge
you'd call risky (auth, graph, upload/extraction, or anything touching the
Gemini/function-handler seam).

Cost is bounded but real: each run spends actual Claude tokens (bounded by
`EXPLORE_MAX_TURNS` — the default 40-turn budget is the knob to shrink if
you're cost-conscious). Wall-clock is warm-cache-dependent: observed full
runs (boot → explore → oracle → teardown) land in the 2-4 minute range with
a warm Next build and an already-cached `@playwright/mcp`; budget closer to
~10 minutes for the very first run on a machine, which additionally pays for
the `npx @playwright/mcp` download and a cold test-profile Next build (see
§3).

## 10. Lock protocol & hygiene

The local E2E stack is a **machine singleton** — one Postgres, one backend,
one frontend, bound to fixed local ports. `scripts/explore.sh` enforces this
by holding an `flock` on `/tmp/claude-<uid>/sapling-e2e-stack.lock` for the
entire session (both the headless full pipeline and the interactive
`up`/`down` pair).

Rules that follow from that:

- **Never run two explorations at once** on the same machine — the second
one fails fast with "stack lock busy" and touches nothing (no wipe, no
teardown of the first session's stack).
- **Never run an exploration alongside the Chapter 1 Playwright suite** —
both want the same stack and the same ports.
- **Always end a session via `down`.** For `make explore` this happens
automatically. For `/explore` or a manual `scripts/explore.sh up`, that
means `make explore-down` (or `scripts/explore.sh down`) — every time,
even after an error. Skipping it leaks both the running stack and the
lock, which then blocks every future exploration until someone runs
`explore-down` by hand.

## 11. Troubleshooting

- **"e2e stack lock busy"** — another session (yours or someone else's on
the same machine) is already exploring or mid-`up`. Find and finish it, or
run `make explore-down` if you're sure it's actually a stale/crashed run.
- **`POST /api/auth/test-login` 404s** — `backend/.env`'s `APP_ENV` is not
`local` or `test`. This route is hard-gated on that check; fix `.env` and
restart the backend.
- **The explorer never seemed to drive the browser** (`session.log` has no
`[tool_use] mcp__playwright__*` lines) — check `.explore/mcp.json`'s flags
against `npx -y @playwright/mcp@0.0.78 --help`; in particular confirm
`--isolated` is present (without it, the MCP server reuses one persistent
Chrome profile per repo path and silently ignores `--storage-state` on a
reused context — the historical #399 follow-up bug this harness had to
fix).
- **Oracle exits 2** — either the stack isn't actually up (`make e2e-up`
first, or check `.e2e/*.log`), or the DB URL the oracle resolved isn't
pointed at the local stack. Re-run with `--json` to see which check raised.
- **Stale `.explore/` from a previous run** — don't worry about manually
cleaning it; the next `up` wipes everything except `lock.ok` before
writing fresh artifacts. Triage anything you care about first (§7).
3 changes: 3 additions & 0 deletions docs/local-supabase.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -259,6 +259,9 @@ preflights each piece and fails fast with the exact command if one is missing.
Rootless Podman is the documented runtime; Docker is auto-detected as a
fallback (`$CONTAINER_CMD` overrides the choice).

Chapter 2 exploratory testing (`make explore`, the `/explore` skill, and the
e2e oracles) is documented in [e2e-exploration.md](e2e-exploration.md).

## Troubleshooting

- **`supabase start` hangs on health checks** — usually the analytics/vector
Expand Down
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotion (#401)#445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

AndresL230
merged 5 commits into

from

Jul 28, 2026
 Merged


docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotion (#401)
#445
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


5 commits


Select commit
Hold shift + click to select a range
e4a054f

docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotio…


AndresL230 Jul 28, 2026
befba6f

docs(e2e): stop citing a gitignored planning file from the runbook


AndresL230 Jul 28, 2026
3c65b97

docs(e2e): fix awkward line wrap in runbook


AndresL230 Jul 28, 2026
56511af

docs(e2e): fix runbook per code-review — seam artifact vs real bug, t…


AndresL230 Jul 28, 2026
df2672b

docs(e2e): rewrap code span so the module path doesn't split mid-toke…


AndresL230 Jul 28, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


313 changes: 313 additions & 0 deletions
docs/e2e-exploration.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,313 @@
# Chapter 2: exploratory testing

This is the runbook for Sapling's Chapter 2 — LLM-driven exploratory testing
of the running app. Read this before your first `make explore`.

## 1. What this is

Sapling has two E2E chapters, and they answer different questions.

| | Chapter 1 | Chapter 2 |
|---|---|---|
| What it is | Scripted, deterministic journeys | LLM-driven exploration |
| Where it lives | `frontend/e2e/*.spec.ts` (Playwright) | `scripts/explore.sh`, `.claude/skills/explore/`, `backend/e2e_oracles/` |
| Gates CI? | Yes — `.github/workflows/e2e.yml`, main-only | Never |
| Runs when | Every push to main | On demand, local-only (`make explore` / `/explore`) |
| Judge | Fixed assertions + `support/db.ts` raw SQL | The e2e oracles (`backend/e2e_oracles`) or a reproducible captured failure |

Chapter 2 exists because a scripted suite only ever re-checks what someone
already thought to assert. An LLM with a real browser will click things,
double-submit forms, hit Back mid-flow, and generally behave like a curious
student — surfacing bugs nobody wrote a journey for yet.

The core rule, verbatim from the epic (#403), governs everything below:

> The LLM explores; only oracles (or a reproducible captured failure) make
> something a finding.

That's why every exploration session ends with a deterministic oracle pass,
and why the triage protocol (§7) insists on reproduction before anything
becomes a GitHub issue. The explorer's own prose is a lead, not a verdict.

## 2. Prerequisites

Everything Chapter 1's local stack needs (see
[local-supabase.md](local-supabase.md)), plus two things specific to Chapter 2:

- Supabase CLI on `PATH`
- Rootless Podman (documented) or Docker (auto-detected fallback)
- `backend/venv` (`python -m venv backend/venv && backend/venv/bin/pip install -r backend/requirements.txt`)
- `frontend/node_modules` (`cd frontend && npm ci`)
- `backend/.env` with `APP_ENV=local` (or `test`) — without it, `POST
 /api/auth/test-login` 404s and the harness can't mint a session
- The `claude` CLI on `PATH` (drives the headless explorer via `claude -p`)
- Network access for the first run — `npx` downloads and pins
 `@playwright/mcp@0.0.78` the first time it's invoked; afterward it's cached

You do **not** need to set a real `GEMINI_API_KEY` or `SEED_RICH` yourself —
`scripts/explore.sh` forces both. A real key in your shell is not
used and never billed by this harness.

## 3. Kick-off: headless

```bash
make explore
```

This runs the full pipeline: boot the stack, mint a session, hand the
explorer prompt to a Playwright-MCP-armed `claude -p`, run the oracle final
pass, then tear everything down. It takes the machine-singleton stack lock
for the whole run (see §10). With warm caches (a repo you've already run
`make e2e-up` or `make explore` on before), the whole thing typically
finishes in a few minutes end-to-end; the first-ever run on a machine can
take closer to ~10 minutes (see §9's cost note — it's the `npx
@playwright/mcp` download plus a cold test-profile Next build, not the
exploration itself).

Env knobs (all optional, read by `scripts/explore.sh`):

| Knob | Default | Meaning |
|---|---|---|
| `EXPLORE_MAX_TURNS` | `40` | Turn budget passed to `claude -p --max-turns`. Bounds both wall-clock and token cost. |
| `EXPLORE_MODEL` | `sonnet` | Model alias passed to `claude -p --model`. |
| `EXPLORE_HEADED` | `0` | `1` = watch the browser drive itself (non-headless); `0` = headless. |
| `EXPLORE_USER` | `rich-user-active` | Which seeded `rich-*` user to sign in as. One of `rich-user-active`, `rich-user-second`, `rich-user-new`, `rich-user-pending`, `rich-user-admin`. |

Example: a slower, watchable run as a different seeded user:

```bash
EXPLORE_HEADED=1 EXPLORE_USER=rich-user-admin EXPLORE_MAX_TURNS=60 make explore
```

Two things worth internalizing before your first run:

- **`.explore/` is wiped at `up`, but only after the lock is acquired.** If
 you want to keep a previous run's findings, triage them (§7) before
 re-running — a second `make explore` deletes the prior artifacts (except
 the in-flight `lock.ok`, which is bookkeeping, not a finding).
- **A crashed run is recoverable.** If a prior `make explore` died mid-run
 (killed session, machine sleep, whatever), `make explore-down` (same as
 `scripts/explore.sh down`) runs the oracle final pass against whatever is
 still up, appends it to `findings.md`, tears the stack down, and releases
 the lock — safe to run even with nothing up.

## 4. Kick-off: interactive

```
/explore
```

Run this as a slash command in a Claude Code session opened on this repo.
Unlike `make explore`, you watch (and can steer) the exploration live:

1. The skill runs `scripts/explore.sh up` itself (this takes minutes —
 Supabase, migrations, seed, backend, a test-profile Next build — and takes
 the same stack lock `make explore` does; if another session holds it, the
 skill stops and tells you).
2. It opens a real browser tab (Playwright MCP if configured for the session,
 else the Claude-in-Chrome tools) at `http://localhost:3000` and signs in as
 `rich-user-active` by minting a session directly from the page.
3. It reads `scripts/explore/explorer-prompt.md` as its own mission briefing
 and starts exploring — narrating what it's trying so you can redirect it
 at any point. Same persona, same break-things mandate, same
 report-never-fix rule, same oracle cadence, same findings format as the
 headless run.
4. It finishes with `scripts/explore.sh down` — same oracle final pass,
 same teardown, same lock release.

The output is identical in shape to a headless run: `.explore/findings.md`
with explorer-written entries plus the harness-appended oracle pass. The only
difference is you're in the loop instead of reading a transcript after the
fact.

## 5. Outputs

Everything lands in `.explore/` at the repo root (gitignored in full):

| Path | What it is |
|---|---|
| `findings.md` | The running human-readable findings log — explorer-written `### F<N>:` entries plus a harness-appended `## Oracle final pass` section at teardown. Pre-created by the harness with a header so the explorer always has an existing file to append to. |
| `session.log` | The full headless-run transcript: `claude -p --output-format stream-json --verbose`, reformatted by `jq` into readable `[assistant]` / `[tool_use]` / `[tool_result]` / `[result]` lines (each truncated to 500 chars). |
| `session.stderr.log` | `claude -p`'s stderr, kept in its own file so it can never interleave with (and corrupt) the JSON stream `session.log` is parsed from. |
| `oracle-final.txt` / `oracle-final.json` | The final `python -m e2e_oracles` pass, run at teardown while the stack is still up, in both text and JSON form. |
| `traces/` | `@playwright/mcp` session-recording artifacts (page snapshots plus an MCP session directory) — these are MCP session recordings, not Playwright Trace Viewer `.zip` files. |
| `storageState.json`, `mcp.json` | Plumbing, not findings: the minted Playwright storage state and the `--mcp-config` fed to `claude -p`. Useful for debugging why the explorer couldn't reach a page, otherwise ignorable. |
| `lock.ok` | Internal lock-holder bookkeeping (holder-written, self-identifying — its content is the holder process's own PID). Not a finding artifact. |

## 6. The oracles, standalone

The explorer runs the oracles as it goes, and the harness runs one final pass
at teardown — but you can also run them yourself against a stack that's
already up (e.g. mid-`/explore`, or after driving the app by hand):

```bash
cd backend && venv/bin/python -m e2e_oracles [--json] [--check ciphertext|counts|graph|logscan|orphans] [--user ID]
```

- `--check` is repeatable; omit it to run all five (default, sorted order).
- `--json` emits structured `Finding` records instead of the text report —
 useful when quoting evidence into a GitHub issue.
- `--user` selects which seeded user's data to check (default
 `rich-user-active`).
- Exit codes: `0` clean, `1` findings, `2` infra error (a check crashed, or
 itself returned an `oracle-error` finding — treat the whole run's other
 results as untrustworthy when this happens, not just the failing check).
- Needs the stack up (`make e2e-up` or `scripts/explore.sh up`) — it hits the
 local Postgres directly and the backend's `/api/health`-adjacent surface.
- The `logscan` check allowlists known #439 RAG-indexing log noise — an
 allowlisted hit still counts toward "N suppressed" in the summary line, but
 doesn't produce a `Finding`.

## 7. Triage protocol

For each entry in `findings.md` (explorer-written or oracle-appended):

1. **Reproduce it.** By hand in a browser, or by re-running the relevant
 oracle `--check`. Not reproducible → drop it, with a one-line note in your
 triage record of why. Part of reproducing it is asking whether the
 "wrong" output is actually the function-mode seam working as designed
 (see the worked example below) — that's a drop too, just a different
 reason than "couldn't reproduce."
2. **Reproducible and real** → `gh issue create` with the finding block
 (steps/expected/actual), the oracle JSON if there is any, and the
 `.explore/traces/` path attached. Label it as sourced from exploration so
 it's traceable back to a session.
3. **Already-known** (currently: #355, #430, #435, #436, #439, #441) → don't
 file a new issue. Add the new evidence as a comment on the existing issue
 only if it's actually new evidence (a different surface, a clearer repro,
 fresh oracle output) — otherwise it's just noise on an already-tracked bug.
4. **Noise the oracle wrongly flagged** → that's a bug in the oracle itself
 (a missing allowlist entry, or a judge that's too strict/wrong). Fix it as
 a normal PR against `backend/e2e_oracles/` — don't just delete the finding
 from `findings.md` and move on.

Two worked examples from this harness's own acceptance testing show the
difference between a genuine finding and noise the *harness itself*
(not the oracle) produced:

**Genuine, promotable (category 2).** Resuming a tutor session in one
acceptance run produced a real `500` on `POST
/api/graph/<user>/concept-description`, traced through the browser console →
network tab → backend log to `LookupError: SAPLING_MODEL_MODE=function but
no handler is registered for task 'concept_describe'` — `concept_describe`
genuinely has no handler in `backend/agents/function_handlers_e2e.py`. The
`logscan` oracle independently caught the same `500` and traceback in the
backend log, so this has both a UI-observed repro and oracle corroboration.
It isn't on the known-bugs list — exactly the kind of thing this chapter
exists to surface, and a strong promotion candidate (§8).

**Seam artifact — drop it, then improve the harness, not the app (still
category 1).** A different run's explorer flagged what looked like a
"wrong-data" bug: uploading a real syllabus PDF came back with a summary
about gradient descent, category "Lecture notes", and concepts "Gradient
Descent"/"Learning Rate" — content belonging to an entirely different,
already-uploaded document. Alarming-looking, but not a product bug: Chapter
2 always runs the document pipeline in function mode
(`SAPLING_MODEL_MODE=function` /
`SAPLING_FUNCTION_HANDLERS=agents.function_handlers_e2e`, forced by
`scripts/explore.sh`), and
`agents/function_handlers_e2e.py`'s classifier/summary/concepts handlers are
*designed* to return the same fixed constants
(`E2E_DOC_CATEGORY`/`E2E_DOC_ABSTRACT`/`E2E_DOC_CONCEPTS`) for every upload
regardless of the file's real content — the module's own docstring says so
("Fixed output... Echoing request content back would couple the constant to
route-side prompt assembly"). The pipeline was working exactly as designed;
the explorer just didn't know that. **The tell:** the "wrong" text is a
byte-for-byte match to one of `function_handlers_e2e.py`'s `E2E_DOC_*` (or
`E2E_TUTOR_REPLY`, `E2E_QUIZ_*`) constants — here, `E2E_DOC_ABSTRACT`'s
"gradient descent... loss surface... learning rate" wording and the fixed
`lecture_notes` category. If a finding's "wrong" content matches one of
those constants verbatim, it's the seam, not the app — drop it, and if it
recurs, improve `scripts/explore/explorer-prompt.md`'s ground rules to name
the pattern explicitly so a future explorer recognizes it before writing the
stub, rather than filing an issue against the harness's own known-fixed
output.

By contrast, #355 (the graph's duplicated CS subject-root hub) shows up in
the graph oracle on essentially every run against the rich seed data —
that's the known-not-new case category 3 above exists for.

## 8. Promotion pipeline

Triage produces GitHub issues; the pipeline that actually matters is turning
the *right* ones into permanent regression coverage. The epic bar (#403):
**at least 3 findings promoted into Chapter 1 regressions in the first
month.**

Promotion path: issue → a scripted journey in `frontend/e2e/*.spec.ts`,
written in the existing specs' style —

- Fixtures-based `test` (see `frontend/e2e/support/fixtures.ts`), not raw
 Playwright boilerplate.
- Assertions against the database go through `support/db.ts` (raw SQL over a
 direct Postgres connection) — journeys never hand-roll DB plumbing.
- Any new `data-testid` follows `docs/frontend-testids.md`'s naming
 convention; see that doc's "Adding a surface" section for the doc-row +
 `eslint.config.mjs` `files`-array update a new surface needs.
 `npm run lint:baseline` is only for clearing legacy debt in
 `eslint-suppressions.json`, not something a new surface should lean on.

If the promoted journey documents a bug that's still open and must stay red,
ride it in on `test.fixme` with the bug number — exactly how
`frontend/e2e/graph.spec.ts` carries its `#355` assertion today (marked
`fixme`, not relaxed, with companion tests asserting everything #355 does
*not* corrupt).

## 9. Cadence + cost

Suggested cadence: one bounded headless run a week, plus one after any merge
you'd call risky (auth, graph, upload/extraction, or anything touching the
Gemini/function-handler seam).

Cost is bounded but real: each run spends actual Claude tokens (bounded by
`EXPLORE_MAX_TURNS` — the default 40-turn budget is the knob to shrink if
you're cost-conscious). Wall-clock is warm-cache-dependent: observed full
runs (boot → explore → oracle → teardown) land in the 2-4 minute range with
a warm Next build and an already-cached `@playwright/mcp`; budget closer to
~10 minutes for the very first run on a machine, which additionally pays for
the `npx @playwright/mcp` download and a cold test-profile Next build (see
§3).

## 10. Lock protocol & hygiene

The local E2E stack is a **machine singleton** — one Postgres, one backend,
one frontend, bound to fixed local ports. `scripts/explore.sh` enforces this
by holding an `flock` on `/tmp/claude-<uid>/sapling-e2e-stack.lock` for the
entire session (both the headless full pipeline and the interactive
`up`/`down` pair).

Rules that follow from that:

- **Never run two explorations at once** on the same machine — the second
 one fails fast with "stack lock busy" and touches nothing (no wipe, no
 teardown of the first session's stack).
- **Never run an exploration alongside the Chapter 1 Playwright suite** —
 both want the same stack and the same ports.
- **Always end a session via `down`.** For `make explore` this happens
 automatically. For `/explore` or a manual `scripts/explore.sh up`, that
 means `make explore-down` (or `scripts/explore.sh down`) — every time,
 even after an error. Skipping it leaks both the running stack and the
 lock, which then blocks every future exploration until someone runs
 `explore-down` by hand.

## 11. Troubleshooting

- **"e2e stack lock busy"** — another session (yours or someone else's on
 the same machine) is already exploring or mid-`up`. Find and finish it, or
 run `make explore-down` if you're sure it's actually a stale/crashed run.
- **`POST /api/auth/test-login` 404s** — `backend/.env`'s `APP_ENV` is not
 `local` or `test`. This route is hard-gated on that check; fix `.env` and
 restart the backend.
- **The explorer never seemed to drive the browser** (`session.log` has no
 `[tool_use] mcp__playwright__*` lines) — check `.explore/mcp.json`'s flags
 against `npx -y @playwright/mcp@0.0.78 --help`; in particular confirm
 `--isolated` is present (without it, the MCP server reuses one persistent
 Chrome profile per repo path and silently ignores `--storage-state` on a
 reused context — the historical #399 follow-up bug this harness had to
 fix).
- **Oracle exits 2** — either the stack isn't actually up (`make e2e-up`
 first, or check `.e2e/*.log`), or the DB URL the oracle resolved isn't
 pointed at the local stack. Re-run with `--json` to see which check raised.
- **Stale `.explore/` from a previous run** — don't worry about manually
 cleaning it; the next `up` wipes everything except `lock.ok` before
 writing fresh artifacts. Triage anything you care about first (§7).

3 changes: 3 additions & 0 deletions
docs/local-supabase.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -259,6 +259,9 @@ preflights each piece and fails fast with the exact command if one is missing.
Rootless Podman is the documented runtime; Docker is auto-detected as a
fallback (`$CONTAINER_CMD` overrides the choice).

Chapter 2 exploratory testing (`make explore`, the `/explore` skill, and the
e2e oracles) is documented in [e2e-exploration.md](e2e-exploration.md).

## Troubleshooting

- **`supabase start` hangs on health checks** — usually the analytics/vector
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotion (#401)#445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

AndresL230
merged 5 commits into

from

Jul 28, 2026
 Merged


docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotion (#401)
#445
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


5 commits


Select commit
Hold shift + click to select a range
e4a054f

docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotio…


AndresL230 Jul 28, 2026
befba6f

docs(e2e): stop citing a gitignored planning file from the runbook


AndresL230 Jul 28, 2026
3c65b97

docs(e2e): fix awkward line wrap in runbook


AndresL230 Jul 28, 2026
56511af

docs(e2e): fix runbook per code-review — seam artifact vs real bug, t…


AndresL230 Jul 28, 2026
df2672b

docs(e2e): rewrap code span so the module path doesn't split mid-toke…


AndresL230 Jul 28, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


313 changes: 313 additions & 0 deletions
docs/e2e-exploration.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,313 @@
# Chapter 2: exploratory testing

This is the runbook for Sapling's Chapter 2 — LLM-driven exploratory testing
of the running app. Read this before your first `make explore`.

## 1. What this is

Sapling has two E2E chapters, and they answer different questions.

| | Chapter 1 | Chapter 2 |
|---|---|---|
| What it is | Scripted, deterministic journeys | LLM-driven exploration |
| Where it lives | `frontend/e2e/*.spec.ts` (Playwright) | `scripts/explore.sh`, `.claude/skills/explore/`, `backend/e2e_oracles/` |
| Gates CI? | Yes — `.github/workflows/e2e.yml`, main-only | Never |
| Runs when | Every push to main | On demand, local-only (`make explore` / `/explore`) |
| Judge | Fixed assertions + `support/db.ts` raw SQL | The e2e oracles (`backend/e2e_oracles`) or a reproducible captured failure |

Chapter 2 exists because a scripted suite only ever re-checks what someone
already thought to assert. An LLM with a real browser will click things,
double-submit forms, hit Back mid-flow, and generally behave like a curious
student — surfacing bugs nobody wrote a journey for yet.

The core rule, verbatim from the epic (#403), governs everything below:

> The LLM explores; only oracles (or a reproducible captured failure) make
> something a finding.

That's why every exploration session ends with a deterministic oracle pass,
and why the triage protocol (§7) insists on reproduction before anything
becomes a GitHub issue. The explorer's own prose is a lead, not a verdict.

## 2. Prerequisites

Everything Chapter 1's local stack needs (see
[local-supabase.md](local-supabase.md)), plus two things specific to Chapter 2:

- Supabase CLI on `PATH`
- Rootless Podman (documented) or Docker (auto-detected fallback)
- `backend/venv` (`python -m venv backend/venv && backend/venv/bin/pip install -r backend/requirements.txt`)
- `frontend/node_modules` (`cd frontend && npm ci`)
- `backend/.env` with `APP_ENV=local` (or `test`) — without it, `POST
 /api/auth/test-login` 404s and the harness can't mint a session
- The `claude` CLI on `PATH` (drives the headless explorer via `claude -p`)
- Network access for the first run — `npx` downloads and pins
 `@playwright/mcp@0.0.78` the first time it's invoked; afterward it's cached

You do **not** need to set a real `GEMINI_API_KEY` or `SEED_RICH` yourself —
`scripts/explore.sh` forces both. A real key in your shell is not
used and never billed by this harness.

## 3. Kick-off: headless

```bash
make explore
```

This runs the full pipeline: boot the stack, mint a session, hand the
explorer prompt to a Playwright-MCP-armed `claude -p`, run the oracle final
pass, then tear everything down. It takes the machine-singleton stack lock
for the whole run (see §10). With warm caches (a repo you've already run
`make e2e-up` or `make explore` on before), the whole thing typically
finishes in a few minutes end-to-end; the first-ever run on a machine can
take closer to ~10 minutes (see §9's cost note — it's the `npx
@playwright/mcp` download plus a cold test-profile Next build, not the
exploration itself).

Env knobs (all optional, read by `scripts/explore.sh`):

| Knob | Default | Meaning |
|---|---|---|
| `EXPLORE_MAX_TURNS` | `40` | Turn budget passed to `claude -p --max-turns`. Bounds both wall-clock and token cost. |
| `EXPLORE_MODEL` | `sonnet` | Model alias passed to `claude -p --model`. |
| `EXPLORE_HEADED` | `0` | `1` = watch the browser drive itself (non-headless); `0` = headless. |
| `EXPLORE_USER` | `rich-user-active` | Which seeded `rich-*` user to sign in as. One of `rich-user-active`, `rich-user-second`, `rich-user-new`, `rich-user-pending`, `rich-user-admin`. |

Example: a slower, watchable run as a different seeded user:

```bash
EXPLORE_HEADED=1 EXPLORE_USER=rich-user-admin EXPLORE_MAX_TURNS=60 make explore
```

Two things worth internalizing before your first run:

- **`.explore/` is wiped at `up`, but only after the lock is acquired.** If
 you want to keep a previous run's findings, triage them (§7) before
 re-running — a second `make explore` deletes the prior artifacts (except
 the in-flight `lock.ok`, which is bookkeeping, not a finding).
- **A crashed run is recoverable.** If a prior `make explore` died mid-run
 (killed session, machine sleep, whatever), `make explore-down` (same as
 `scripts/explore.sh down`) runs the oracle final pass against whatever is
 still up, appends it to `findings.md`, tears the stack down, and releases
 the lock — safe to run even with nothing up.

## 4. Kick-off: interactive

```
/explore
```

Run this as a slash command in a Claude Code session opened on this repo.
Unlike `make explore`, you watch (and can steer) the exploration live:

1. The skill runs `scripts/explore.sh up` itself (this takes minutes —
 Supabase, migrations, seed, backend, a test-profile Next build — and takes
 the same stack lock `make explore` does; if another session holds it, the
 skill stops and tells you).
2. It opens a real browser tab (Playwright MCP if configured for the session,
 else the Claude-in-Chrome tools) at `http://localhost:3000` and signs in as
 `rich-user-active` by minting a session directly from the page.
3. It reads `scripts/explore/explorer-prompt.md` as its own mission briefing
 and starts exploring — narrating what it's trying so you can redirect it
 at any point. Same persona, same break-things mandate, same
 report-never-fix rule, same oracle cadence, same findings format as the
 headless run.
4. It finishes with `scripts/explore.sh down` — same oracle final pass,
 same teardown, same lock release.

The output is identical in shape to a headless run: `.explore/findings.md`
with explorer-written entries plus the harness-appended oracle pass. The only
difference is you're in the loop instead of reading a transcript after the
fact.

## 5. Outputs

Everything lands in `.explore/` at the repo root (gitignored in full):

| Path | What it is |
|---|---|
| `findings.md` | The running human-readable findings log — explorer-written `### F<N>:` entries plus a harness-appended `## Oracle final pass` section at teardown. Pre-created by the harness with a header so the explorer always has an existing file to append to. |
| `session.log` | The full headless-run transcript: `claude -p --output-format stream-json --verbose`, reformatted by `jq` into readable `[assistant]` / `[tool_use]` / `[tool_result]` / `[result]` lines (each truncated to 500 chars). |
| `session.stderr.log` | `claude -p`'s stderr, kept in its own file so it can never interleave with (and corrupt) the JSON stream `session.log` is parsed from. |
| `oracle-final.txt` / `oracle-final.json` | The final `python -m e2e_oracles` pass, run at teardown while the stack is still up, in both text and JSON form. |
| `traces/` | `@playwright/mcp` session-recording artifacts (page snapshots plus an MCP session directory) — these are MCP session recordings, not Playwright Trace Viewer `.zip` files. |
| `storageState.json`, `mcp.json` | Plumbing, not findings: the minted Playwright storage state and the `--mcp-config` fed to `claude -p`. Useful for debugging why the explorer couldn't reach a page, otherwise ignorable. |
| `lock.ok` | Internal lock-holder bookkeeping (holder-written, self-identifying — its content is the holder process's own PID). Not a finding artifact. |

## 6. The oracles, standalone

The explorer runs the oracles as it goes, and the harness runs one final pass
at teardown — but you can also run them yourself against a stack that's
already up (e.g. mid-`/explore`, or after driving the app by hand):

```bash
cd backend && venv/bin/python -m e2e_oracles [--json] [--check ciphertext|counts|graph|logscan|orphans] [--user ID]
```

- `--check` is repeatable; omit it to run all five (default, sorted order).
- `--json` emits structured `Finding` records instead of the text report —
 useful when quoting evidence into a GitHub issue.
- `--user` selects which seeded user's data to check (default
 `rich-user-active`).
- Exit codes: `0` clean, `1` findings, `2` infra error (a check crashed, or
 itself returned an `oracle-error` finding — treat the whole run's other
 results as untrustworthy when this happens, not just the failing check).
- Needs the stack up (`make e2e-up` or `scripts/explore.sh up`) — it hits the
 local Postgres directly and the backend's `/api/health`-adjacent surface.
- The `logscan` check allowlists known #439 RAG-indexing log noise — an
 allowlisted hit still counts toward "N suppressed" in the summary line, but
 doesn't produce a `Finding`.

## 7. Triage protocol

For each entry in `findings.md` (explorer-written or oracle-appended):

1. **Reproduce it.** By hand in a browser, or by re-running the relevant
 oracle `--check`. Not reproducible → drop it, with a one-line note in your
 triage record of why. Part of reproducing it is asking whether the
 "wrong" output is actually the function-mode seam working as designed
 (see the worked example below) — that's a drop too, just a different
 reason than "couldn't reproduce."
2. **Reproducible and real** → `gh issue create` with the finding block
 (steps/expected/actual), the oracle JSON if there is any, and the
 `.explore/traces/` path attached. Label it as sourced from exploration so
 it's traceable back to a session.
3. **Already-known** (currently: #355, #430, #435, #436, #439, #441) → don't
 file a new issue. Add the new evidence as a comment on the existing issue
 only if it's actually new evidence (a different surface, a clearer repro,
 fresh oracle output) — otherwise it's just noise on an already-tracked bug.
4. **Noise the oracle wrongly flagged** → that's a bug in the oracle itself
 (a missing allowlist entry, or a judge that's too strict/wrong). Fix it as
 a normal PR against `backend/e2e_oracles/` — don't just delete the finding
 from `findings.md` and move on.

Two worked examples from this harness's own acceptance testing show the
difference between a genuine finding and noise the *harness itself*
(not the oracle) produced:

**Genuine, promotable (category 2).** Resuming a tutor session in one
acceptance run produced a real `500` on `POST
/api/graph/<user>/concept-description`, traced through the browser console →
network tab → backend log to `LookupError: SAPLING_MODEL_MODE=function but
no handler is registered for task 'concept_describe'` — `concept_describe`
genuinely has no handler in `backend/agents/function_handlers_e2e.py`. The
`logscan` oracle independently caught the same `500` and traceback in the
backend log, so this has both a UI-observed repro and oracle corroboration.
It isn't on the known-bugs list — exactly the kind of thing this chapter
exists to surface, and a strong promotion candidate (§8).

**Seam artifact — drop it, then improve the harness, not the app (still
category 1).** A different run's explorer flagged what looked like a
"wrong-data" bug: uploading a real syllabus PDF came back with a summary
about gradient descent, category "Lecture notes", and concepts "Gradient
Descent"/"Learning Rate" — content belonging to an entirely different,
already-uploaded document. Alarming-looking, but not a product bug: Chapter
2 always runs the document pipeline in function mode
(`SAPLING_MODEL_MODE=function` /
`SAPLING_FUNCTION_HANDLERS=agents.function_handlers_e2e`, forced by
`scripts/explore.sh`), and
`agents/function_handlers_e2e.py`'s classifier/summary/concepts handlers are
*designed* to return the same fixed constants
(`E2E_DOC_CATEGORY`/`E2E_DOC_ABSTRACT`/`E2E_DOC_CONCEPTS`) for every upload
regardless of the file's real content — the module's own docstring says so
("Fixed output... Echoing request content back would couple the constant to
route-side prompt assembly"). The pipeline was working exactly as designed;
the explorer just didn't know that. **The tell:** the "wrong" text is a
byte-for-byte match to one of `function_handlers_e2e.py`'s `E2E_DOC_*` (or
`E2E_TUTOR_REPLY`, `E2E_QUIZ_*`) constants — here, `E2E_DOC_ABSTRACT`'s
"gradient descent... loss surface... learning rate" wording and the fixed
`lecture_notes` category. If a finding's "wrong" content matches one of
those constants verbatim, it's the seam, not the app — drop it, and if it
recurs, improve `scripts/explore/explorer-prompt.md`'s ground rules to name
the pattern explicitly so a future explorer recognizes it before writing the
stub, rather than filing an issue against the harness's own known-fixed
output.

By contrast, #355 (the graph's duplicated CS subject-root hub) shows up in
the graph oracle on essentially every run against the rich seed data —
that's the known-not-new case category 3 above exists for.

## 8. Promotion pipeline

Triage produces GitHub issues; the pipeline that actually matters is turning
the *right* ones into permanent regression coverage. The epic bar (#403):
**at least 3 findings promoted into Chapter 1 regressions in the first
month.**

Promotion path: issue → a scripted journey in `frontend/e2e/*.spec.ts`,
written in the existing specs' style —

- Fixtures-based `test` (see `frontend/e2e/support/fixtures.ts`), not raw
 Playwright boilerplate.
- Assertions against the database go through `support/db.ts` (raw SQL over a
 direct Postgres connection) — journeys never hand-roll DB plumbing.
- Any new `data-testid` follows `docs/frontend-testids.md`'s naming
 convention; see that doc's "Adding a surface" section for the doc-row +
 `eslint.config.mjs` `files`-array update a new surface needs.
 `npm run lint:baseline` is only for clearing legacy debt in
 `eslint-suppressions.json`, not something a new surface should lean on.

If the promoted journey documents a bug that's still open and must stay red,
ride it in on `test.fixme` with the bug number — exactly how
`frontend/e2e/graph.spec.ts` carries its `#355` assertion today (marked
`fixme`, not relaxed, with companion tests asserting everything #355 does
*not* corrupt).

## 9. Cadence + cost

Suggested cadence: one bounded headless run a week, plus one after any merge
you'd call risky (auth, graph, upload/extraction, or anything touching the
Gemini/function-handler seam).

Cost is bounded but real: each run spends actual Claude tokens (bounded by
`EXPLORE_MAX_TURNS` — the default 40-turn budget is the knob to shrink if
you're cost-conscious). Wall-clock is warm-cache-dependent: observed full
runs (boot → explore → oracle → teardown) land in the 2-4 minute range with
a warm Next build and an already-cached `@playwright/mcp`; budget closer to
~10 minutes for the very first run on a machine, which additionally pays for
the `npx @playwright/mcp` download and a cold test-profile Next build (see
§3).

## 10. Lock protocol & hygiene

The local E2E stack is a **machine singleton** — one Postgres, one backend,
one frontend, bound to fixed local ports. `scripts/explore.sh` enforces this
by holding an `flock` on `/tmp/claude-<uid>/sapling-e2e-stack.lock` for the
entire session (both the headless full pipeline and the interactive
`up`/`down` pair).

Rules that follow from that:

- **Never run two explorations at once** on the same machine — the second
 one fails fast with "stack lock busy" and touches nothing (no wipe, no
 teardown of the first session's stack).
- **Never run an exploration alongside the Chapter 1 Playwright suite** —
 both want the same stack and the same ports.
- **Always end a session via `down`.** For `make explore` this happens
 automatically. For `/explore` or a manual `scripts/explore.sh up`, that
 means `make explore-down` (or `scripts/explore.sh down`) — every time,
 even after an error. Skipping it leaks both the running stack and the
 lock, which then blocks every future exploration until someone runs
 `explore-down` by hand.

## 11. Troubleshooting

- **"e2e stack lock busy"** — another session (yours or someone else's on
 the same machine) is already exploring or mid-`up`. Find and finish it, or
 run `make explore-down` if you're sure it's actually a stale/crashed run.
- **`POST /api/auth/test-login` 404s** — `backend/.env`'s `APP_ENV` is not
 `local` or `test`. This route is hard-gated on that check; fix `.env` and
 restart the backend.
- **The explorer never seemed to drive the browser** (`session.log` has no
 `[tool_use] mcp__playwright__*` lines) — check `.explore/mcp.json`'s flags
 against `npx -y @playwright/mcp@0.0.78 --help`; in particular confirm
 `--isolated` is present (without it, the MCP server reuses one persistent
 Chrome profile per repo path and silently ignores `--storage-state` on a
 reused context — the historical #399 follow-up bug this harness had to
 fix).
- **Oracle exits 2** — either the stack isn't actually up (`make e2e-up`
 first, or check `.e2e/*.log`), or the DB URL the oracle resolved isn't
 pointed at the local stack. Re-run with `--json` to see which check raised.
- **Stale `.explore/` from a previous run** — don't worry about manually
 cleaning it; the next `up` wipes everything except `lock.ok` before
 writing fresh artifacts. Triage anything you care about first (§7).

3 changes: 3 additions & 0 deletions
docs/local-supabase.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -259,6 +259,9 @@ preflights each piece and fails fast with the exact command if one is missing.
Rootless Podman is the documented runtime; Docker is auto-detected as a
fallback (`$CONTAINER_CMD` overrides the choice).

Chapter 2 exploratory testing (`make explore`, the `/explore` skill, and the
e2e oracles) is documented in [e2e-exploration.md](e2e-exploration.md).

## Troubleshooting

- **`supabase start` hangs on health checks** — usually the analytics/vector
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotion (#401)#445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

AndresL230
merged 5 commits into

from

Jul 28, 2026
 Merged


docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotion (#401)
#445
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


5 commits


Select commit
Hold shift + click to select a range
e4a054f

docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotio…


AndresL230 Jul 28, 2026
befba6f

docs(e2e): stop citing a gitignored planning file from the runbook


AndresL230 Jul 28, 2026
3c65b97

docs(e2e): fix awkward line wrap in runbook


AndresL230 Jul 28, 2026
56511af

docs(e2e): fix runbook per code-review — seam artifact vs real bug, t…


AndresL230 Jul 28, 2026
df2672b

docs(e2e): rewrap code span so the module path doesn't split mid-toke…


AndresL230 Jul 28, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


313 changes: 313 additions & 0 deletions
docs/e2e-exploration.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,313 @@
# Chapter 2: exploratory testing

This is the runbook for Sapling's Chapter 2 — LLM-driven exploratory testing
of the running app. Read this before your first `make explore`.

## 1. What this is

Sapling has two E2E chapters, and they answer different questions.

| | Chapter 1 | Chapter 2 |
|---|---|---|
| What it is | Scripted, deterministic journeys | LLM-driven exploration |
| Where it lives | `frontend/e2e/*.spec.ts` (Playwright) | `scripts/explore.sh`, `.claude/skills/explore/`, `backend/e2e_oracles/` |
| Gates CI? | Yes — `.github/workflows/e2e.yml`, main-only | Never |
| Runs when | Every push to main | On demand, local-only (`make explore` / `/explore`) |
| Judge | Fixed assertions + `support/db.ts` raw SQL | The e2e oracles (`backend/e2e_oracles`) or a reproducible captured failure |

Chapter 2 exists because a scripted suite only ever re-checks what someone
already thought to assert. An LLM with a real browser will click things,
double-submit forms, hit Back mid-flow, and generally behave like a curious
student — surfacing bugs nobody wrote a journey for yet.

The core rule, verbatim from the epic (#403), governs everything below:

> The LLM explores; only oracles (or a reproducible captured failure) make
> something a finding.

That's why every exploration session ends with a deterministic oracle pass,
and why the triage protocol (§7) insists on reproduction before anything
becomes a GitHub issue. The explorer's own prose is a lead, not a verdict.

## 2. Prerequisites

Everything Chapter 1's local stack needs (see
[local-supabase.md](local-supabase.md)), plus two things specific to Chapter 2:

- Supabase CLI on `PATH`
- Rootless Podman (documented) or Docker (auto-detected fallback)
- `backend/venv` (`python -m venv backend/venv && backend/venv/bin/pip install -r backend/requirements.txt`)
- `frontend/node_modules` (`cd frontend && npm ci`)
- `backend/.env` with `APP_ENV=local` (or `test`) — without it, `POST
 /api/auth/test-login` 404s and the harness can't mint a session
- The `claude` CLI on `PATH` (drives the headless explorer via `claude -p`)
- Network access for the first run — `npx` downloads and pins
 `@playwright/mcp@0.0.78` the first time it's invoked; afterward it's cached

You do **not** need to set a real `GEMINI_API_KEY` or `SEED_RICH` yourself —
`scripts/explore.sh` forces both. A real key in your shell is not
used and never billed by this harness.

## 3. Kick-off: headless

```bash
make explore
```

This runs the full pipeline: boot the stack, mint a session, hand the
explorer prompt to a Playwright-MCP-armed `claude -p`, run the oracle final
pass, then tear everything down. It takes the machine-singleton stack lock
for the whole run (see §10). With warm caches (a repo you've already run
`make e2e-up` or `make explore` on before), the whole thing typically
finishes in a few minutes end-to-end; the first-ever run on a machine can
take closer to ~10 minutes (see §9's cost note — it's the `npx
@playwright/mcp` download plus a cold test-profile Next build, not the
exploration itself).

Env knobs (all optional, read by `scripts/explore.sh`):

| Knob | Default | Meaning |
|---|---|---|
| `EXPLORE_MAX_TURNS` | `40` | Turn budget passed to `claude -p --max-turns`. Bounds both wall-clock and token cost. |
| `EXPLORE_MODEL` | `sonnet` | Model alias passed to `claude -p --model`. |
| `EXPLORE_HEADED` | `0` | `1` = watch the browser drive itself (non-headless); `0` = headless. |
| `EXPLORE_USER` | `rich-user-active` | Which seeded `rich-*` user to sign in as. One of `rich-user-active`, `rich-user-second`, `rich-user-new`, `rich-user-pending`, `rich-user-admin`. |

Example: a slower, watchable run as a different seeded user:

```bash
EXPLORE_HEADED=1 EXPLORE_USER=rich-user-admin EXPLORE_MAX_TURNS=60 make explore
```

Two things worth internalizing before your first run:

- **`.explore/` is wiped at `up`, but only after the lock is acquired.** If
 you want to keep a previous run's findings, triage them (§7) before
 re-running — a second `make explore` deletes the prior artifacts (except
 the in-flight `lock.ok`, which is bookkeeping, not a finding).
- **A crashed run is recoverable.** If a prior `make explore` died mid-run
 (killed session, machine sleep, whatever), `make explore-down` (same as
 `scripts/explore.sh down`) runs the oracle final pass against whatever is
 still up, appends it to `findings.md`, tears the stack down, and releases
 the lock — safe to run even with nothing up.

## 4. Kick-off: interactive

```
/explore
```

Run this as a slash command in a Claude Code session opened on this repo.
Unlike `make explore`, you watch (and can steer) the exploration live:

1. The skill runs `scripts/explore.sh up` itself (this takes minutes —
 Supabase, migrations, seed, backend, a test-profile Next build — and takes
 the same stack lock `make explore` does; if another session holds it, the
 skill stops and tells you).
2. It opens a real browser tab (Playwright MCP if configured for the session,
 else the Claude-in-Chrome tools) at `http://localhost:3000` and signs in as
 `rich-user-active` by minting a session directly from the page.
3. It reads `scripts/explore/explorer-prompt.md` as its own mission briefing
 and starts exploring — narrating what it's trying so you can redirect it
 at any point. Same persona, same break-things mandate, same
 report-never-fix rule, same oracle cadence, same findings format as the
 headless run.
4. It finishes with `scripts/explore.sh down` — same oracle final pass,
 same teardown, same lock release.

The output is identical in shape to a headless run: `.explore/findings.md`
with explorer-written entries plus the harness-appended oracle pass. The only
difference is you're in the loop instead of reading a transcript after the
fact.

## 5. Outputs

Everything lands in `.explore/` at the repo root (gitignored in full):

| Path | What it is |
|---|---|
| `findings.md` | The running human-readable findings log — explorer-written `### F<N>:` entries plus a harness-appended `## Oracle final pass` section at teardown. Pre-created by the harness with a header so the explorer always has an existing file to append to. |
| `session.log` | The full headless-run transcript: `claude -p --output-format stream-json --verbose`, reformatted by `jq` into readable `[assistant]` / `[tool_use]` / `[tool_result]` / `[result]` lines (each truncated to 500 chars). |
| `session.stderr.log` | `claude -p`'s stderr, kept in its own file so it can never interleave with (and corrupt) the JSON stream `session.log` is parsed from. |
| `oracle-final.txt` / `oracle-final.json` | The final `python -m e2e_oracles` pass, run at teardown while the stack is still up, in both text and JSON form. |
| `traces/` | `@playwright/mcp` session-recording artifacts (page snapshots plus an MCP session directory) — these are MCP session recordings, not Playwright Trace Viewer `.zip` files. |
| `storageState.json`, `mcp.json` | Plumbing, not findings: the minted Playwright storage state and the `--mcp-config` fed to `claude -p`. Useful for debugging why the explorer couldn't reach a page, otherwise ignorable. |
| `lock.ok` | Internal lock-holder bookkeeping (holder-written, self-identifying — its content is the holder process's own PID). Not a finding artifact. |

## 6. The oracles, standalone

The explorer runs the oracles as it goes, and the harness runs one final pass
at teardown — but you can also run them yourself against a stack that's
already up (e.g. mid-`/explore`, or after driving the app by hand):

```bash
cd backend && venv/bin/python -m e2e_oracles [--json] [--check ciphertext|counts|graph|logscan|orphans] [--user ID]
```

- `--check` is repeatable; omit it to run all five (default, sorted order).
- `--json` emits structured `Finding` records instead of the text report —
 useful when quoting evidence into a GitHub issue.
- `--user` selects which seeded user's data to check (default
 `rich-user-active`).
- Exit codes: `0` clean, `1` findings, `2` infra error (a check crashed, or
 itself returned an `oracle-error` finding — treat the whole run's other
 results as untrustworthy when this happens, not just the failing check).
- Needs the stack up (`make e2e-up` or `scripts/explore.sh up`) — it hits the
 local Postgres directly and the backend's `/api/health`-adjacent surface.
- The `logscan` check allowlists known #439 RAG-indexing log noise — an
 allowlisted hit still counts toward "N suppressed" in the summary line, but
 doesn't produce a `Finding`.

## 7. Triage protocol

For each entry in `findings.md` (explorer-written or oracle-appended):

1. **Reproduce it.** By hand in a browser, or by re-running the relevant
 oracle `--check`. Not reproducible → drop it, with a one-line note in your
 triage record of why. Part of reproducing it is asking whether the
 "wrong" output is actually the function-mode seam working as designed
 (see the worked example below) — that's a drop too, just a different
 reason than "couldn't reproduce."
2. **Reproducible and real** → `gh issue create` with the finding block
 (steps/expected/actual), the oracle JSON if there is any, and the
 `.explore/traces/` path attached. Label it as sourced from exploration so
 it's traceable back to a session.
3. **Already-known** (currently: #355, #430, #435, #436, #439, #441) → don't
 file a new issue. Add the new evidence as a comment on the existing issue
 only if it's actually new evidence (a different surface, a clearer repro,
 fresh oracle output) — otherwise it's just noise on an already-tracked bug.
4. **Noise the oracle wrongly flagged** → that's a bug in the oracle itself
 (a missing allowlist entry, or a judge that's too strict/wrong). Fix it as
 a normal PR against `backend/e2e_oracles/` — don't just delete the finding
 from `findings.md` and move on.

Two worked examples from this harness's own acceptance testing show the
difference between a genuine finding and noise the *harness itself*
(not the oracle) produced:

**Genuine, promotable (category 2).** Resuming a tutor session in one
acceptance run produced a real `500` on `POST
/api/graph/<user>/concept-description`, traced through the browser console →
network tab → backend log to `LookupError: SAPLING_MODEL_MODE=function but
no handler is registered for task 'concept_describe'` — `concept_describe`
genuinely has no handler in `backend/agents/function_handlers_e2e.py`. The
`logscan` oracle independently caught the same `500` and traceback in the
backend log, so this has both a UI-observed repro and oracle corroboration.
It isn't on the known-bugs list — exactly the kind of thing this chapter
exists to surface, and a strong promotion candidate (§8).

**Seam artifact — drop it, then improve the harness, not the app (still
category 1).** A different run's explorer flagged what looked like a
"wrong-data" bug: uploading a real syllabus PDF came back with a summary
about gradient descent, category "Lecture notes", and concepts "Gradient
Descent"/"Learning Rate" — content belonging to an entirely different,
already-uploaded document. Alarming-looking, but not a product bug: Chapter
2 always runs the document pipeline in function mode
(`SAPLING_MODEL_MODE=function` /
`SAPLING_FUNCTION_HANDLERS=agents.function_handlers_e2e`, forced by
`scripts/explore.sh`), and
`agents/function_handlers_e2e.py`'s classifier/summary/concepts handlers are
*designed* to return the same fixed constants
(`E2E_DOC_CATEGORY`/`E2E_DOC_ABSTRACT`/`E2E_DOC_CONCEPTS`) for every upload
regardless of the file's real content — the module's own docstring says so
("Fixed output... Echoing request content back would couple the constant to
route-side prompt assembly"). The pipeline was working exactly as designed;
the explorer just didn't know that. **The tell:** the "wrong" text is a
byte-for-byte match to one of `function_handlers_e2e.py`'s `E2E_DOC_*` (or
`E2E_TUTOR_REPLY`, `E2E_QUIZ_*`) constants — here, `E2E_DOC_ABSTRACT`'s
"gradient descent... loss surface... learning rate" wording and the fixed
`lecture_notes` category. If a finding's "wrong" content matches one of
those constants verbatim, it's the seam, not the app — drop it, and if it
recurs, improve `scripts/explore/explorer-prompt.md`'s ground rules to name
the pattern explicitly so a future explorer recognizes it before writing the
stub, rather than filing an issue against the harness's own known-fixed
output.

By contrast, #355 (the graph's duplicated CS subject-root hub) shows up in
the graph oracle on essentially every run against the rich seed data —
that's the known-not-new case category 3 above exists for.

## 8. Promotion pipeline

Triage produces GitHub issues; the pipeline that actually matters is turning
the *right* ones into permanent regression coverage. The epic bar (#403):
**at least 3 findings promoted into Chapter 1 regressions in the first
month.**

Promotion path: issue → a scripted journey in `frontend/e2e/*.spec.ts`,
written in the existing specs' style —

- Fixtures-based `test` (see `frontend/e2e/support/fixtures.ts`), not raw
 Playwright boilerplate.
- Assertions against the database go through `support/db.ts` (raw SQL over a
 direct Postgres connection) — journeys never hand-roll DB plumbing.
- Any new `data-testid` follows `docs/frontend-testids.md`'s naming
 convention; see that doc's "Adding a surface" section for the doc-row +
 `eslint.config.mjs` `files`-array update a new surface needs.
 `npm run lint:baseline` is only for clearing legacy debt in
 `eslint-suppressions.json`, not something a new surface should lean on.

If the promoted journey documents a bug that's still open and must stay red,
ride it in on `test.fixme` with the bug number — exactly how
`frontend/e2e/graph.spec.ts` carries its `#355` assertion today (marked
`fixme`, not relaxed, with companion tests asserting everything #355 does
*not* corrupt).

## 9. Cadence + cost

Suggested cadence: one bounded headless run a week, plus one after any merge
you'd call risky (auth, graph, upload/extraction, or anything touching the
Gemini/function-handler seam).

Cost is bounded but real: each run spends actual Claude tokens (bounded by
`EXPLORE_MAX_TURNS` — the default 40-turn budget is the knob to shrink if
you're cost-conscious). Wall-clock is warm-cache-dependent: observed full
runs (boot → explore → oracle → teardown) land in the 2-4 minute range with
a warm Next build and an already-cached `@playwright/mcp`; budget closer to
~10 minutes for the very first run on a machine, which additionally pays for
the `npx @playwright/mcp` download and a cold test-profile Next build (see
§3).

## 10. Lock protocol & hygiene

The local E2E stack is a **machine singleton** — one Postgres, one backend,
one frontend, bound to fixed local ports. `scripts/explore.sh` enforces this
by holding an `flock` on `/tmp/claude-<uid>/sapling-e2e-stack.lock` for the
entire session (both the headless full pipeline and the interactive
`up`/`down` pair).

Rules that follow from that:

- **Never run two explorations at once** on the same machine — the second
 one fails fast with "stack lock busy" and touches nothing (no wipe, no
 teardown of the first session's stack).
- **Never run an exploration alongside the Chapter 1 Playwright suite** —
 both want the same stack and the same ports.
- **Always end a session via `down`.** For `make explore` this happens
 automatically. For `/explore` or a manual `scripts/explore.sh up`, that
 means `make explore-down` (or `scripts/explore.sh down`) — every time,
 even after an error. Skipping it leaks both the running stack and the
 lock, which then blocks every future exploration until someone runs
 `explore-down` by hand.

## 11. Troubleshooting

- **"e2e stack lock busy"** — another session (yours or someone else's on
 the same machine) is already exploring or mid-`up`. Find and finish it, or
 run `make explore-down` if you're sure it's actually a stale/crashed run.
- **`POST /api/auth/test-login` 404s** — `backend/.env`'s `APP_ENV` is not
 `local` or `test`. This route is hard-gated on that check; fix `.env` and
 restart the backend.
- **The explorer never seemed to drive the browser** (`session.log` has no
 `[tool_use] mcp__playwright__*` lines) — check `.explore/mcp.json`'s flags
 against `npx -y @playwright/mcp@0.0.78 --help`; in particular confirm
 `--isolated` is present (without it, the MCP server reuses one persistent
 Chrome profile per repo path and silently ignores `--storage-state` on a
 reused context — the historical #399 follow-up bug this harness had to
 fix).
- **Oracle exits 2** — either the stack isn't actually up (`make e2e-up`
 first, or check `.e2e/*.log`), or the DB URL the oracle resolved isn't
 pointed at the local stack. Re-run with `--json` to see which check raised.
- **Stale `.explore/` from a previous run** — don't worry about manually
 cleaning it; the next `up` wipes everything except `lock.ok` before
 writing fresh artifacts. Triage anything you care about first (§7).

3 changes: 3 additions & 0 deletions
docs/local-supabase.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -259,6 +259,9 @@ preflights each piece and fails fast with the exact command if one is missing.
Rootless Podman is the documented runtime; Docker is auto-detected as a
fallback (`$CONTAINER_CMD` overrides the choice).

Chapter 2 exploratory testing (`make explore`, the `/explore` skill, and the
e2e oracles) is documented in [e2e-exploration.md](e2e-exploration.md).

## Troubleshooting

- **`supabase start` hangs on health checks** — usually the analytics/vector
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotion (#401)#445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

AndresL230
merged 5 commits into

from

Jul 28, 2026
 Merged


docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotion (#401)
#445
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


5 commits


Select commit
Hold shift + click to select a range
e4a054f

docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotio…


AndresL230 Jul 28, 2026
befba6f

docs(e2e): stop citing a gitignored planning file from the runbook


AndresL230 Jul 28, 2026
3c65b97

docs(e2e): fix awkward line wrap in runbook


AndresL230 Jul 28, 2026
56511af

docs(e2e): fix runbook per code-review — seam artifact vs real bug, t…


AndresL230 Jul 28, 2026
df2672b

docs(e2e): rewrap code span so the module path doesn't split mid-toke…


AndresL230 Jul 28, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


313 changes: 313 additions & 0 deletions
docs/e2e-exploration.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,313 @@
# Chapter 2: exploratory testing

This is the runbook for Sapling's Chapter 2 — LLM-driven exploratory testing
of the running app. Read this before your first `make explore`.

## 1. What this is

Sapling has two E2E chapters, and they answer different questions.

| | Chapter 1 | Chapter 2 |
|---|---|---|
| What it is | Scripted, deterministic journeys | LLM-driven exploration |
| Where it lives | `frontend/e2e/*.spec.ts` (Playwright) | `scripts/explore.sh`, `.claude/skills/explore/`, `backend/e2e_oracles/` |
| Gates CI? | Yes — `.github/workflows/e2e.yml`, main-only | Never |
| Runs when | Every push to main | On demand, local-only (`make explore` / `/explore`) |
| Judge | Fixed assertions + `support/db.ts` raw SQL | The e2e oracles (`backend/e2e_oracles`) or a reproducible captured failure |

Chapter 2 exists because a scripted suite only ever re-checks what someone
already thought to assert. An LLM with a real browser will click things,
double-submit forms, hit Back mid-flow, and generally behave like a curious
student — surfacing bugs nobody wrote a journey for yet.

The core rule, verbatim from the epic (#403), governs everything below:

> The LLM explores; only oracles (or a reproducible captured failure) make
> something a finding.

That's why every exploration session ends with a deterministic oracle pass,
and why the triage protocol (§7) insists on reproduction before anything
becomes a GitHub issue. The explorer's own prose is a lead, not a verdict.

## 2. Prerequisites

Everything Chapter 1's local stack needs (see
[local-supabase.md](local-supabase.md)), plus two things specific to Chapter 2:

- Supabase CLI on `PATH`
- Rootless Podman (documented) or Docker (auto-detected fallback)
- `backend/venv` (`python -m venv backend/venv && backend/venv/bin/pip install -r backend/requirements.txt`)
- `frontend/node_modules` (`cd frontend && npm ci`)
- `backend/.env` with `APP_ENV=local` (or `test`) — without it, `POST
 /api/auth/test-login` 404s and the harness can't mint a session
- The `claude` CLI on `PATH` (drives the headless explorer via `claude -p`)
- Network access for the first run — `npx` downloads and pins
 `@playwright/mcp@0.0.78` the first time it's invoked; afterward it's cached

You do **not** need to set a real `GEMINI_API_KEY` or `SEED_RICH` yourself —
`scripts/explore.sh` forces both. A real key in your shell is not
used and never billed by this harness.

## 3. Kick-off: headless

```bash
make explore
```

This runs the full pipeline: boot the stack, mint a session, hand the
explorer prompt to a Playwright-MCP-armed `claude -p`, run the oracle final
pass, then tear everything down. It takes the machine-singleton stack lock
for the whole run (see §10). With warm caches (a repo you've already run
`make e2e-up` or `make explore` on before), the whole thing typically
finishes in a few minutes end-to-end; the first-ever run on a machine can
take closer to ~10 minutes (see §9's cost note — it's the `npx
@playwright/mcp` download plus a cold test-profile Next build, not the
exploration itself).

Env knobs (all optional, read by `scripts/explore.sh`):

| Knob | Default | Meaning |
|---|---|---|
| `EXPLORE_MAX_TURNS` | `40` | Turn budget passed to `claude -p --max-turns`. Bounds both wall-clock and token cost. |
| `EXPLORE_MODEL` | `sonnet` | Model alias passed to `claude -p --model`. |
| `EXPLORE_HEADED` | `0` | `1` = watch the browser drive itself (non-headless); `0` = headless. |
| `EXPLORE_USER` | `rich-user-active` | Which seeded `rich-*` user to sign in as. One of `rich-user-active`, `rich-user-second`, `rich-user-new`, `rich-user-pending`, `rich-user-admin`. |

Example: a slower, watchable run as a different seeded user:

```bash
EXPLORE_HEADED=1 EXPLORE_USER=rich-user-admin EXPLORE_MAX_TURNS=60 make explore
```

Two things worth internalizing before your first run:

- **`.explore/` is wiped at `up`, but only after the lock is acquired.** If
 you want to keep a previous run's findings, triage them (§7) before
 re-running — a second `make explore` deletes the prior artifacts (except
 the in-flight `lock.ok`, which is bookkeeping, not a finding).
- **A crashed run is recoverable.** If a prior `make explore` died mid-run
 (killed session, machine sleep, whatever), `make explore-down` (same as
 `scripts/explore.sh down`) runs the oracle final pass against whatever is
 still up, appends it to `findings.md`, tears the stack down, and releases
 the lock — safe to run even with nothing up.

## 4. Kick-off: interactive

```
/explore
```

Run this as a slash command in a Claude Code session opened on this repo.
Unlike `make explore`, you watch (and can steer) the exploration live:

1. The skill runs `scripts/explore.sh up` itself (this takes minutes —
 Supabase, migrations, seed, backend, a test-profile Next build — and takes
 the same stack lock `make explore` does; if another session holds it, the
 skill stops and tells you).
2. It opens a real browser tab (Playwright MCP if configured for the session,
 else the Claude-in-Chrome tools) at `http://localhost:3000` and signs in as
 `rich-user-active` by minting a session directly from the page.
3. It reads `scripts/explore/explorer-prompt.md` as its own mission briefing
 and starts exploring — narrating what it's trying so you can redirect it
 at any point. Same persona, same break-things mandate, same
 report-never-fix rule, same oracle cadence, same findings format as the
 headless run.
4. It finishes with `scripts/explore.sh down` — same oracle final pass,
 same teardown, same lock release.

The output is identical in shape to a headless run: `.explore/findings.md`
with explorer-written entries plus the harness-appended oracle pass. The only
difference is you're in the loop instead of reading a transcript after the
fact.

## 5. Outputs

Everything lands in `.explore/` at the repo root (gitignored in full):

| Path | What it is |
|---|---|
| `findings.md` | The running human-readable findings log — explorer-written `### F<N>:` entries plus a harness-appended `## Oracle final pass` section at teardown. Pre-created by the harness with a header so the explorer always has an existing file to append to. |
| `session.log` | The full headless-run transcript: `claude -p --output-format stream-json --verbose`, reformatted by `jq` into readable `[assistant]` / `[tool_use]` / `[tool_result]` / `[result]` lines (each truncated to 500 chars). |
| `session.stderr.log` | `claude -p`'s stderr, kept in its own file so it can never interleave with (and corrupt) the JSON stream `session.log` is parsed from. |
| `oracle-final.txt` / `oracle-final.json` | The final `python -m e2e_oracles` pass, run at teardown while the stack is still up, in both text and JSON form. |
| `traces/` | `@playwright/mcp` session-recording artifacts (page snapshots plus an MCP session directory) — these are MCP session recordings, not Playwright Trace Viewer `.zip` files. |
| `storageState.json`, `mcp.json` | Plumbing, not findings: the minted Playwright storage state and the `--mcp-config` fed to `claude -p`. Useful for debugging why the explorer couldn't reach a page, otherwise ignorable. |
| `lock.ok` | Internal lock-holder bookkeeping (holder-written, self-identifying — its content is the holder process's own PID). Not a finding artifact. |

## 6. The oracles, standalone

The explorer runs the oracles as it goes, and the harness runs one final pass
at teardown — but you can also run them yourself against a stack that's
already up (e.g. mid-`/explore`, or after driving the app by hand):

```bash
cd backend && venv/bin/python -m e2e_oracles [--json] [--check ciphertext|counts|graph|logscan|orphans] [--user ID]
```

- `--check` is repeatable; omit it to run all five (default, sorted order).
- `--json` emits structured `Finding` records instead of the text report —
 useful when quoting evidence into a GitHub issue.
- `--user` selects which seeded user's data to check (default
 `rich-user-active`).
- Exit codes: `0` clean, `1` findings, `2` infra error (a check crashed, or
 itself returned an `oracle-error` finding — treat the whole run's other
 results as untrustworthy when this happens, not just the failing check).
- Needs the stack up (`make e2e-up` or `scripts/explore.sh up`) — it hits the
 local Postgres directly and the backend's `/api/health`-adjacent surface.
- The `logscan` check allowlists known #439 RAG-indexing log noise — an
 allowlisted hit still counts toward "N suppressed" in the summary line, but
 doesn't produce a `Finding`.

## 7. Triage protocol

For each entry in `findings.md` (explorer-written or oracle-appended):

1. **Reproduce it.** By hand in a browser, or by re-running the relevant
 oracle `--check`. Not reproducible → drop it, with a one-line note in your
 triage record of why. Part of reproducing it is asking whether the
 "wrong" output is actually the function-mode seam working as designed
 (see the worked example below) — that's a drop too, just a different
 reason than "couldn't reproduce."
2. **Reproducible and real** → `gh issue create` with the finding block
 (steps/expected/actual), the oracle JSON if there is any, and the
 `.explore/traces/` path attached. Label it as sourced from exploration so
 it's traceable back to a session.
3. **Already-known** (currently: #355, #430, #435, #436, #439, #441) → don't
 file a new issue. Add the new evidence as a comment on the existing issue
 only if it's actually new evidence (a different surface, a clearer repro,
 fresh oracle output) — otherwise it's just noise on an already-tracked bug.
4. **Noise the oracle wrongly flagged** → that's a bug in the oracle itself
 (a missing allowlist entry, or a judge that's too strict/wrong). Fix it as
 a normal PR against `backend/e2e_oracles/` — don't just delete the finding
 from `findings.md` and move on.

Two worked examples from this harness's own acceptance testing show the
difference between a genuine finding and noise the *harness itself*
(not the oracle) produced:

**Genuine, promotable (category 2).** Resuming a tutor session in one
acceptance run produced a real `500` on `POST
/api/graph/<user>/concept-description`, traced through the browser console →
network tab → backend log to `LookupError: SAPLING_MODEL_MODE=function but
no handler is registered for task 'concept_describe'` — `concept_describe`
genuinely has no handler in `backend/agents/function_handlers_e2e.py`. The
`logscan` oracle independently caught the same `500` and traceback in the
backend log, so this has both a UI-observed repro and oracle corroboration.
It isn't on the known-bugs list — exactly the kind of thing this chapter
exists to surface, and a strong promotion candidate (§8).

**Seam artifact — drop it, then improve the harness, not the app (still
category 1).** A different run's explorer flagged what looked like a
"wrong-data" bug: uploading a real syllabus PDF came back with a summary
about gradient descent, category "Lecture notes", and concepts "Gradient
Descent"/"Learning Rate" — content belonging to an entirely different,
already-uploaded document. Alarming-looking, but not a product bug: Chapter
2 always runs the document pipeline in function mode
(`SAPLING_MODEL_MODE=function` /
`SAPLING_FUNCTION_HANDLERS=agents.function_handlers_e2e`, forced by
`scripts/explore.sh`), and
`agents/function_handlers_e2e.py`'s classifier/summary/concepts handlers are
*designed* to return the same fixed constants
(`E2E_DOC_CATEGORY`/`E2E_DOC_ABSTRACT`/`E2E_DOC_CONCEPTS`) for every upload
regardless of the file's real content — the module's own docstring says so
("Fixed output... Echoing request content back would couple the constant to
route-side prompt assembly"). The pipeline was working exactly as designed;
the explorer just didn't know that. **The tell:** the "wrong" text is a
byte-for-byte match to one of `function_handlers_e2e.py`'s `E2E_DOC_*` (or
`E2E_TUTOR_REPLY`, `E2E_QUIZ_*`) constants — here, `E2E_DOC_ABSTRACT`'s
"gradient descent... loss surface... learning rate" wording and the fixed
`lecture_notes` category. If a finding's "wrong" content matches one of
those constants verbatim, it's the seam, not the app — drop it, and if it
recurs, improve `scripts/explore/explorer-prompt.md`'s ground rules to name
the pattern explicitly so a future explorer recognizes it before writing the
stub, rather than filing an issue against the harness's own known-fixed
output.

By contrast, #355 (the graph's duplicated CS subject-root hub) shows up in
the graph oracle on essentially every run against the rich seed data —
that's the known-not-new case category 3 above exists for.

## 8. Promotion pipeline

Triage produces GitHub issues; the pipeline that actually matters is turning
the *right* ones into permanent regression coverage. The epic bar (#403):
**at least 3 findings promoted into Chapter 1 regressions in the first
month.**

Promotion path: issue → a scripted journey in `frontend/e2e/*.spec.ts`,
written in the existing specs' style —

- Fixtures-based `test` (see `frontend/e2e/support/fixtures.ts`), not raw
 Playwright boilerplate.
- Assertions against the database go through `support/db.ts` (raw SQL over a
 direct Postgres connection) — journeys never hand-roll DB plumbing.
- Any new `data-testid` follows `docs/frontend-testids.md`'s naming
 convention; see that doc's "Adding a surface" section for the doc-row +
 `eslint.config.mjs` `files`-array update a new surface needs.
 `npm run lint:baseline` is only for clearing legacy debt in
 `eslint-suppressions.json`, not something a new surface should lean on.

If the promoted journey documents a bug that's still open and must stay red,
ride it in on `test.fixme` with the bug number — exactly how
`frontend/e2e/graph.spec.ts` carries its `#355` assertion today (marked
`fixme`, not relaxed, with companion tests asserting everything #355 does
*not* corrupt).

## 9. Cadence + cost

Suggested cadence: one bounded headless run a week, plus one after any merge
you'd call risky (auth, graph, upload/extraction, or anything touching the
Gemini/function-handler seam).

Cost is bounded but real: each run spends actual Claude tokens (bounded by
`EXPLORE_MAX_TURNS` — the default 40-turn budget is the knob to shrink if
you're cost-conscious). Wall-clock is warm-cache-dependent: observed full
runs (boot → explore → oracle → teardown) land in the 2-4 minute range with
a warm Next build and an already-cached `@playwright/mcp`; budget closer to
~10 minutes for the very first run on a machine, which additionally pays for
the `npx @playwright/mcp` download and a cold test-profile Next build (see
§3).

## 10. Lock protocol & hygiene

The local E2E stack is a **machine singleton** — one Postgres, one backend,
one frontend, bound to fixed local ports. `scripts/explore.sh` enforces this
by holding an `flock` on `/tmp/claude-<uid>/sapling-e2e-stack.lock` for the
entire session (both the headless full pipeline and the interactive
`up`/`down` pair).

Rules that follow from that:

- **Never run two explorations at once** on the same machine — the second
 one fails fast with "stack lock busy" and touches nothing (no wipe, no
 teardown of the first session's stack).
- **Never run an exploration alongside the Chapter 1 Playwright suite** —
 both want the same stack and the same ports.
- **Always end a session via `down`.** For `make explore` this happens
 automatically. For `/explore` or a manual `scripts/explore.sh up`, that
 means `make explore-down` (or `scripts/explore.sh down`) — every time,
 even after an error. Skipping it leaks both the running stack and the
 lock, which then blocks every future exploration until someone runs
 `explore-down` by hand.

## 11. Troubleshooting

- **"e2e stack lock busy"** — another session (yours or someone else's on
 the same machine) is already exploring or mid-`up`. Find and finish it, or
 run `make explore-down` if you're sure it's actually a stale/crashed run.
- **`POST /api/auth/test-login` 404s** — `backend/.env`'s `APP_ENV` is not
 `local` or `test`. This route is hard-gated on that check; fix `.env` and
 restart the backend.
- **The explorer never seemed to drive the browser** (`session.log` has no
 `[tool_use] mcp__playwright__*` lines) — check `.explore/mcp.json`'s flags
 against `npx -y @playwright/mcp@0.0.78 --help`; in particular confirm
 `--isolated` is present (without it, the MCP server reuses one persistent
 Chrome profile per repo path and silently ignores `--storage-state` on a
 reused context — the historical #399 follow-up bug this harness had to
 fix).
- **Oracle exits 2** — either the stack isn't actually up (`make e2e-up`
 first, or check `.e2e/*.log`), or the DB URL the oracle resolved isn't
 pointed at the local stack. Re-run with `--json` to see which check raised.
- **Stale `.explore/` from a previous run** — don't worry about manually
 cleaning it; the next `up` wipes everything except `lock.ok` before
 writing fresh artifacts. Triage anything you care about first (§7).

3 changes: 3 additions & 0 deletions
docs/local-supabase.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -259,6 +259,9 @@ preflights each piece and fails fast with the exact command if one is missing.
Rootless Podman is the documented runtime; Docker is auto-detected as a
fallback (`$CONTAINER_CMD` overrides the choice).

Chapter 2 exploratory testing (`make explore`, the `/explore` skill, and the
e2e oracles) is documented in [e2e-exploration.md](e2e-exploration.md).

## Troubleshooting

- **`supabase start` hangs on health checks** — usually the analytics/vector
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotion (#401)#445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

AndresL230
merged 5 commits into

from

Jul 28, 2026
 Merged


docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotion (#401)
#445
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


5 commits


Select commit
Hold shift + click to select a range
e4a054f

docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotio…


AndresL230 Jul 28, 2026
befba6f

docs(e2e): stop citing a gitignored planning file from the runbook


AndresL230 Jul 28, 2026
3c65b97

docs(e2e): fix awkward line wrap in runbook


AndresL230 Jul 28, 2026
56511af

docs(e2e): fix runbook per code-review — seam artifact vs real bug, t…


AndresL230 Jul 28, 2026
df2672b

docs(e2e): rewrap code span so the module path doesn't split mid-toke…


AndresL230 Jul 28, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


313 changes: 313 additions & 0 deletions
docs/e2e-exploration.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,313 @@
# Chapter 2: exploratory testing

This is the runbook for Sapling's Chapter 2 — LLM-driven exploratory testing
of the running app. Read this before your first `make explore`.

## 1. What this is

Sapling has two E2E chapters, and they answer different questions.

| | Chapter 1 | Chapter 2 |
|---|---|---|
| What it is | Scripted, deterministic journeys | LLM-driven exploration |
| Where it lives | `frontend/e2e/*.spec.ts` (Playwright) | `scripts/explore.sh`, `.claude/skills/explore/`, `backend/e2e_oracles/` |
| Gates CI? | Yes — `.github/workflows/e2e.yml`, main-only | Never |
| Runs when | Every push to main | On demand, local-only (`make explore` / `/explore`) |
| Judge | Fixed assertions + `support/db.ts` raw SQL | The e2e oracles (`backend/e2e_oracles`) or a reproducible captured failure |

Chapter 2 exists because a scripted suite only ever re-checks what someone
already thought to assert. An LLM with a real browser will click things,
double-submit forms, hit Back mid-flow, and generally behave like a curious
student — surfacing bugs nobody wrote a journey for yet.

The core rule, verbatim from the epic (#403), governs everything below:

> The LLM explores; only oracles (or a reproducible captured failure) make
> something a finding.

That's why every exploration session ends with a deterministic oracle pass,
and why the triage protocol (§7) insists on reproduction before anything
becomes a GitHub issue. The explorer's own prose is a lead, not a verdict.

## 2. Prerequisites

Everything Chapter 1's local stack needs (see
[local-supabase.md](local-supabase.md)), plus two things specific to Chapter 2:

- Supabase CLI on `PATH`
- Rootless Podman (documented) or Docker (auto-detected fallback)
- `backend/venv` (`python -m venv backend/venv && backend/venv/bin/pip install -r backend/requirements.txt`)
- `frontend/node_modules` (`cd frontend && npm ci`)
- `backend/.env` with `APP_ENV=local` (or `test`) — without it, `POST
 /api/auth/test-login` 404s and the harness can't mint a session
- The `claude` CLI on `PATH` (drives the headless explorer via `claude -p`)
- Network access for the first run — `npx` downloads and pins
 `@playwright/mcp@0.0.78` the first time it's invoked; afterward it's cached

You do **not** need to set a real `GEMINI_API_KEY` or `SEED_RICH` yourself —
`scripts/explore.sh` forces both. A real key in your shell is not
used and never billed by this harness.

## 3. Kick-off: headless

```bash
make explore
```

This runs the full pipeline: boot the stack, mint a session, hand the
explorer prompt to a Playwright-MCP-armed `claude -p`, run the oracle final
pass, then tear everything down. It takes the machine-singleton stack lock
for the whole run (see §10). With warm caches (a repo you've already run
`make e2e-up` or `make explore` on before), the whole thing typically
finishes in a few minutes end-to-end; the first-ever run on a machine can
take closer to ~10 minutes (see §9's cost note — it's the `npx
@playwright/mcp` download plus a cold test-profile Next build, not the
exploration itself).

Env knobs (all optional, read by `scripts/explore.sh`):

| Knob | Default | Meaning |
|---|---|---|
| `EXPLORE_MAX_TURNS` | `40` | Turn budget passed to `claude -p --max-turns`. Bounds both wall-clock and token cost. |
| `EXPLORE_MODEL` | `sonnet` | Model alias passed to `claude -p --model`. |
| `EXPLORE_HEADED` | `0` | `1` = watch the browser drive itself (non-headless); `0` = headless. |
| `EXPLORE_USER` | `rich-user-active` | Which seeded `rich-*` user to sign in as. One of `rich-user-active`, `rich-user-second`, `rich-user-new`, `rich-user-pending`, `rich-user-admin`. |

Example: a slower, watchable run as a different seeded user:

```bash
EXPLORE_HEADED=1 EXPLORE_USER=rich-user-admin EXPLORE_MAX_TURNS=60 make explore
```

Two things worth internalizing before your first run:

- **`.explore/` is wiped at `up`, but only after the lock is acquired.** If
 you want to keep a previous run's findings, triage them (§7) before
 re-running — a second `make explore` deletes the prior artifacts (except
 the in-flight `lock.ok`, which is bookkeeping, not a finding).
- **A crashed run is recoverable.** If a prior `make explore` died mid-run
 (killed session, machine sleep, whatever), `make explore-down` (same as
 `scripts/explore.sh down`) runs the oracle final pass against whatever is
 still up, appends it to `findings.md`, tears the stack down, and releases
 the lock — safe to run even with nothing up.

## 4. Kick-off: interactive

```
/explore
```

Run this as a slash command in a Claude Code session opened on this repo.
Unlike `make explore`, you watch (and can steer) the exploration live:

1. The skill runs `scripts/explore.sh up` itself (this takes minutes —
 Supabase, migrations, seed, backend, a test-profile Next build — and takes
 the same stack lock `make explore` does; if another session holds it, the
 skill stops and tells you).
2. It opens a real browser tab (Playwright MCP if configured for the session,
 else the Claude-in-Chrome tools) at `http://localhost:3000` and signs in as
 `rich-user-active` by minting a session directly from the page.
3. It reads `scripts/explore/explorer-prompt.md` as its own mission briefing
 and starts exploring — narrating what it's trying so you can redirect it
 at any point. Same persona, same break-things mandate, same
 report-never-fix rule, same oracle cadence, same findings format as the
 headless run.
4. It finishes with `scripts/explore.sh down` — same oracle final pass,
 same teardown, same lock release.

The output is identical in shape to a headless run: `.explore/findings.md`
with explorer-written entries plus the harness-appended oracle pass. The only
difference is you're in the loop instead of reading a transcript after the
fact.

## 5. Outputs

Everything lands in `.explore/` at the repo root (gitignored in full):

| Path | What it is |
|---|---|
| `findings.md` | The running human-readable findings log — explorer-written `### F<N>:` entries plus a harness-appended `## Oracle final pass` section at teardown. Pre-created by the harness with a header so the explorer always has an existing file to append to. |
| `session.log` | The full headless-run transcript: `claude -p --output-format stream-json --verbose`, reformatted by `jq` into readable `[assistant]` / `[tool_use]` / `[tool_result]` / `[result]` lines (each truncated to 500 chars). |
| `session.stderr.log` | `claude -p`'s stderr, kept in its own file so it can never interleave with (and corrupt) the JSON stream `session.log` is parsed from. |
| `oracle-final.txt` / `oracle-final.json` | The final `python -m e2e_oracles` pass, run at teardown while the stack is still up, in both text and JSON form. |
| `traces/` | `@playwright/mcp` session-recording artifacts (page snapshots plus an MCP session directory) — these are MCP session recordings, not Playwright Trace Viewer `.zip` files. |
| `storageState.json`, `mcp.json` | Plumbing, not findings: the minted Playwright storage state and the `--mcp-config` fed to `claude -p`. Useful for debugging why the explorer couldn't reach a page, otherwise ignorable. |
| `lock.ok` | Internal lock-holder bookkeeping (holder-written, self-identifying — its content is the holder process's own PID). Not a finding artifact. |

## 6. The oracles, standalone

The explorer runs the oracles as it goes, and the harness runs one final pass
at teardown — but you can also run them yourself against a stack that's
already up (e.g. mid-`/explore`, or after driving the app by hand):

```bash
cd backend && venv/bin/python -m e2e_oracles [--json] [--check ciphertext|counts|graph|logscan|orphans] [--user ID]
```

- `--check` is repeatable; omit it to run all five (default, sorted order).
- `--json` emits structured `Finding` records instead of the text report —
 useful when quoting evidence into a GitHub issue.
- `--user` selects which seeded user's data to check (default
 `rich-user-active`).
- Exit codes: `0` clean, `1` findings, `2` infra error (a check crashed, or
 itself returned an `oracle-error` finding — treat the whole run's other
 results as untrustworthy when this happens, not just the failing check).
- Needs the stack up (`make e2e-up` or `scripts/explore.sh up`) — it hits the
 local Postgres directly and the backend's `/api/health`-adjacent surface.
- The `logscan` check allowlists known #439 RAG-indexing log noise — an
 allowlisted hit still counts toward "N suppressed" in the summary line, but
 doesn't produce a `Finding`.

## 7. Triage protocol

For each entry in `findings.md` (explorer-written or oracle-appended):

1. **Reproduce it.** By hand in a browser, or by re-running the relevant
 oracle `--check`. Not reproducible → drop it, with a one-line note in your
 triage record of why. Part of reproducing it is asking whether the
 "wrong" output is actually the function-mode seam working as designed
 (see the worked example below) — that's a drop too, just a different
 reason than "couldn't reproduce."
2. **Reproducible and real** → `gh issue create` with the finding block
 (steps/expected/actual), the oracle JSON if there is any, and the
 `.explore/traces/` path attached. Label it as sourced from exploration so
 it's traceable back to a session.
3. **Already-known** (currently: #355, #430, #435, #436, #439, #441) → don't
 file a new issue. Add the new evidence as a comment on the existing issue
 only if it's actually new evidence (a different surface, a clearer repro,
 fresh oracle output) — otherwise it's just noise on an already-tracked bug.
4. **Noise the oracle wrongly flagged** → that's a bug in the oracle itself
 (a missing allowlist entry, or a judge that's too strict/wrong). Fix it as
 a normal PR against `backend/e2e_oracles/` — don't just delete the finding
 from `findings.md` and move on.

Two worked examples from this harness's own acceptance testing show the
difference between a genuine finding and noise the *harness itself*
(not the oracle) produced:

**Genuine, promotable (category 2).** Resuming a tutor session in one
acceptance run produced a real `500` on `POST
/api/graph/<user>/concept-description`, traced through the browser console →
network tab → backend log to `LookupError: SAPLING_MODEL_MODE=function but
no handler is registered for task 'concept_describe'` — `concept_describe`
genuinely has no handler in `backend/agents/function_handlers_e2e.py`. The
`logscan` oracle independently caught the same `500` and traceback in the
backend log, so this has both a UI-observed repro and oracle corroboration.
It isn't on the known-bugs list — exactly the kind of thing this chapter
exists to surface, and a strong promotion candidate (§8).

**Seam artifact — drop it, then improve the harness, not the app (still
category 1).** A different run's explorer flagged what looked like a
"wrong-data" bug: uploading a real syllabus PDF came back with a summary
about gradient descent, category "Lecture notes", and concepts "Gradient
Descent"/"Learning Rate" — content belonging to an entirely different,
already-uploaded document. Alarming-looking, but not a product bug: Chapter
2 always runs the document pipeline in function mode
(`SAPLING_MODEL_MODE=function` /
`SAPLING_FUNCTION_HANDLERS=agents.function_handlers_e2e`, forced by
`scripts/explore.sh`), and
`agents/function_handlers_e2e.py`'s classifier/summary/concepts handlers are
*designed* to return the same fixed constants
(`E2E_DOC_CATEGORY`/`E2E_DOC_ABSTRACT`/`E2E_DOC_CONCEPTS`) for every upload
regardless of the file's real content — the module's own docstring says so
("Fixed output... Echoing request content back would couple the constant to
route-side prompt assembly"). The pipeline was working exactly as designed;
the explorer just didn't know that. **The tell:** the "wrong" text is a
byte-for-byte match to one of `function_handlers_e2e.py`'s `E2E_DOC_*` (or
`E2E_TUTOR_REPLY`, `E2E_QUIZ_*`) constants — here, `E2E_DOC_ABSTRACT`'s
"gradient descent... loss surface... learning rate" wording and the fixed
`lecture_notes` category. If a finding's "wrong" content matches one of
those constants verbatim, it's the seam, not the app — drop it, and if it
recurs, improve `scripts/explore/explorer-prompt.md`'s ground rules to name
the pattern explicitly so a future explorer recognizes it before writing the
stub, rather than filing an issue against the harness's own known-fixed
output.

By contrast, #355 (the graph's duplicated CS subject-root hub) shows up in
the graph oracle on essentially every run against the rich seed data —
that's the known-not-new case category 3 above exists for.

## 8. Promotion pipeline

Triage produces GitHub issues; the pipeline that actually matters is turning
the *right* ones into permanent regression coverage. The epic bar (#403):
**at least 3 findings promoted into Chapter 1 regressions in the first
month.**

Promotion path: issue → a scripted journey in `frontend/e2e/*.spec.ts`,
written in the existing specs' style —

- Fixtures-based `test` (see `frontend/e2e/support/fixtures.ts`), not raw
 Playwright boilerplate.
- Assertions against the database go through `support/db.ts` (raw SQL over a
 direct Postgres connection) — journeys never hand-roll DB plumbing.
- Any new `data-testid` follows `docs/frontend-testids.md`'s naming
 convention; see that doc's "Adding a surface" section for the doc-row +
 `eslint.config.mjs` `files`-array update a new surface needs.
 `npm run lint:baseline` is only for clearing legacy debt in
 `eslint-suppressions.json`, not something a new surface should lean on.

If the promoted journey documents a bug that's still open and must stay red,
ride it in on `test.fixme` with the bug number — exactly how
`frontend/e2e/graph.spec.ts` carries its `#355` assertion today (marked
`fixme`, not relaxed, with companion tests asserting everything #355 does
*not* corrupt).

## 9. Cadence + cost

Suggested cadence: one bounded headless run a week, plus one after any merge
you'd call risky (auth, graph, upload/extraction, or anything touching the
Gemini/function-handler seam).

Cost is bounded but real: each run spends actual Claude tokens (bounded by
`EXPLORE_MAX_TURNS` — the default 40-turn budget is the knob to shrink if
you're cost-conscious). Wall-clock is warm-cache-dependent: observed full
runs (boot → explore → oracle → teardown) land in the 2-4 minute range with
a warm Next build and an already-cached `@playwright/mcp`; budget closer to
~10 minutes for the very first run on a machine, which additionally pays for
the `npx @playwright/mcp` download and a cold test-profile Next build (see
§3).

## 10. Lock protocol & hygiene

The local E2E stack is a **machine singleton** — one Postgres, one backend,
one frontend, bound to fixed local ports. `scripts/explore.sh` enforces this
by holding an `flock` on `/tmp/claude-<uid>/sapling-e2e-stack.lock` for the
entire session (both the headless full pipeline and the interactive
`up`/`down` pair).

Rules that follow from that:

- **Never run two explorations at once** on the same machine — the second
 one fails fast with "stack lock busy" and touches nothing (no wipe, no
 teardown of the first session's stack).
- **Never run an exploration alongside the Chapter 1 Playwright suite** —
 both want the same stack and the same ports.
- **Always end a session via `down`.** For `make explore` this happens
 automatically. For `/explore` or a manual `scripts/explore.sh up`, that
 means `make explore-down` (or `scripts/explore.sh down`) — every time,
 even after an error. Skipping it leaks both the running stack and the
 lock, which then blocks every future exploration until someone runs
 `explore-down` by hand.

## 11. Troubleshooting

- **"e2e stack lock busy"** — another session (yours or someone else's on
 the same machine) is already exploring or mid-`up`. Find and finish it, or
 run `make explore-down` if you're sure it's actually a stale/crashed run.
- **`POST /api/auth/test-login` 404s** — `backend/.env`'s `APP_ENV` is not
 `local` or `test`. This route is hard-gated on that check; fix `.env` and
 restart the backend.
- **The explorer never seemed to drive the browser** (`session.log` has no
 `[tool_use] mcp__playwright__*` lines) — check `.explore/mcp.json`'s flags
 against `npx -y @playwright/mcp@0.0.78 --help`; in particular confirm
 `--isolated` is present (without it, the MCP server reuses one persistent
 Chrome profile per repo path and silently ignores `--storage-state` on a
 reused context — the historical #399 follow-up bug this harness had to
 fix).
- **Oracle exits 2** — either the stack isn't actually up (`make e2e-up`
 first, or check `.e2e/*.log`), or the DB URL the oracle resolved isn't
 pointed at the local stack. Re-run with `--json` to see which check raised.
- **Stale `.explore/` from a previous run** — don't worry about manually
 cleaning it; the next `up` wipes everything except `lock.ok` before
 writing fresh artifacts. Triage anything you care about first (§7).

3 changes: 3 additions & 0 deletions
docs/local-supabase.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -259,6 +259,9 @@ preflights each piece and fails fast with the exact command if one is missing.
Rootless Podman is the documented runtime; Docker is auto-detected as a
fallback (`$CONTAINER_CMD` overrides the choice).

Chapter 2 exploratory testing (`make explore`, the `/explore` skill, and the
e2e oracles) is documented in [e2e-exploration.md](e2e-exploration.md).

## Troubleshooting

- **`supabase start` hangs on health checks** — usually the analytics/vector
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotion (#401)#445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

AndresL230
merged 5 commits into

from

Jul 28, 2026
 Merged


docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotion (#401)
#445
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


5 commits


Select commit
Hold shift + click to select a range
e4a054f

docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotio…


AndresL230 Jul 28, 2026
befba6f

docs(e2e): stop citing a gitignored planning file from the runbook


AndresL230 Jul 28, 2026
3c65b97

docs(e2e): fix awkward line wrap in runbook


AndresL230 Jul 28, 2026
56511af

docs(e2e): fix runbook per code-review — seam artifact vs real bug, t…


AndresL230 Jul 28, 2026
df2672b

docs(e2e): rewrap code span so the module path doesn't split mid-toke…


AndresL230 Jul 28, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


313 changes: 313 additions & 0 deletions
docs/e2e-exploration.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,313 @@
# Chapter 2: exploratory testing

This is the runbook for Sapling's Chapter 2 — LLM-driven exploratory testing
of the running app. Read this before your first `make explore`.

## 1. What this is

Sapling has two E2E chapters, and they answer different questions.

| | Chapter 1 | Chapter 2 |
|---|---|---|
| What it is | Scripted, deterministic journeys | LLM-driven exploration |
| Where it lives | `frontend/e2e/*.spec.ts` (Playwright) | `scripts/explore.sh`, `.claude/skills/explore/`, `backend/e2e_oracles/` |
| Gates CI? | Yes — `.github/workflows/e2e.yml`, main-only | Never |
| Runs when | Every push to main | On demand, local-only (`make explore` / `/explore`) |
| Judge | Fixed assertions + `support/db.ts` raw SQL | The e2e oracles (`backend/e2e_oracles`) or a reproducible captured failure |

Chapter 2 exists because a scripted suite only ever re-checks what someone
already thought to assert. An LLM with a real browser will click things,
double-submit forms, hit Back mid-flow, and generally behave like a curious
student — surfacing bugs nobody wrote a journey for yet.

The core rule, verbatim from the epic (#403), governs everything below:

> The LLM explores; only oracles (or a reproducible captured failure) make
> something a finding.

That's why every exploration session ends with a deterministic oracle pass,
and why the triage protocol (§7) insists on reproduction before anything
becomes a GitHub issue. The explorer's own prose is a lead, not a verdict.

## 2. Prerequisites

Everything Chapter 1's local stack needs (see
[local-supabase.md](local-supabase.md)), plus two things specific to Chapter 2:

- Supabase CLI on `PATH`
- Rootless Podman (documented) or Docker (auto-detected fallback)
- `backend/venv` (`python -m venv backend/venv && backend/venv/bin/pip install -r backend/requirements.txt`)
- `frontend/node_modules` (`cd frontend && npm ci`)
- `backend/.env` with `APP_ENV=local` (or `test`) — without it, `POST
 /api/auth/test-login` 404s and the harness can't mint a session
- The `claude` CLI on `PATH` (drives the headless explorer via `claude -p`)
- Network access for the first run — `npx` downloads and pins
 `@playwright/mcp@0.0.78` the first time it's invoked; afterward it's cached

You do **not** need to set a real `GEMINI_API_KEY` or `SEED_RICH` yourself —
`scripts/explore.sh` forces both. A real key in your shell is not
used and never billed by this harness.

## 3. Kick-off: headless

```bash
make explore
```

This runs the full pipeline: boot the stack, mint a session, hand the
explorer prompt to a Playwright-MCP-armed `claude -p`, run the oracle final
pass, then tear everything down. It takes the machine-singleton stack lock
for the whole run (see §10). With warm caches (a repo you've already run
`make e2e-up` or `make explore` on before), the whole thing typically
finishes in a few minutes end-to-end; the first-ever run on a machine can
take closer to ~10 minutes (see §9's cost note — it's the `npx
@playwright/mcp` download plus a cold test-profile Next build, not the
exploration itself).

Env knobs (all optional, read by `scripts/explore.sh`):

| Knob | Default | Meaning |
|---|---|---|
| `EXPLORE_MAX_TURNS` | `40` | Turn budget passed to `claude -p --max-turns`. Bounds both wall-clock and token cost. |
| `EXPLORE_MODEL` | `sonnet` | Model alias passed to `claude -p --model`. |
| `EXPLORE_HEADED` | `0` | `1` = watch the browser drive itself (non-headless); `0` = headless. |
| `EXPLORE_USER` | `rich-user-active` | Which seeded `rich-*` user to sign in as. One of `rich-user-active`, `rich-user-second`, `rich-user-new`, `rich-user-pending`, `rich-user-admin`. |

Example: a slower, watchable run as a different seeded user:

```bash
EXPLORE_HEADED=1 EXPLORE_USER=rich-user-admin EXPLORE_MAX_TURNS=60 make explore
```

Two things worth internalizing before your first run:

- **`.explore/` is wiped at `up`, but only after the lock is acquired.** If
 you want to keep a previous run's findings, triage them (§7) before
 re-running — a second `make explore` deletes the prior artifacts (except
 the in-flight `lock.ok`, which is bookkeeping, not a finding).
- **A crashed run is recoverable.** If a prior `make explore` died mid-run
 (killed session, machine sleep, whatever), `make explore-down` (same as
 `scripts/explore.sh down`) runs the oracle final pass against whatever is
 still up, appends it to `findings.md`, tears the stack down, and releases
 the lock — safe to run even with nothing up.

## 4. Kick-off: interactive

```
/explore
```

Run this as a slash command in a Claude Code session opened on this repo.
Unlike `make explore`, you watch (and can steer) the exploration live:

1. The skill runs `scripts/explore.sh up` itself (this takes minutes —
 Supabase, migrations, seed, backend, a test-profile Next build — and takes
 the same stack lock `make explore` does; if another session holds it, the
 skill stops and tells you).
2. It opens a real browser tab (Playwright MCP if configured for the session,
 else the Claude-in-Chrome tools) at `http://localhost:3000` and signs in as
 `rich-user-active` by minting a session directly from the page.
3. It reads `scripts/explore/explorer-prompt.md` as its own mission briefing
 and starts exploring — narrating what it's trying so you can redirect it
 at any point. Same persona, same break-things mandate, same
 report-never-fix rule, same oracle cadence, same findings format as the
 headless run.
4. It finishes with `scripts/explore.sh down` — same oracle final pass,
 same teardown, same lock release.

The output is identical in shape to a headless run: `.explore/findings.md`
with explorer-written entries plus the harness-appended oracle pass. The only
difference is you're in the loop instead of reading a transcript after the
fact.

## 5. Outputs

Everything lands in `.explore/` at the repo root (gitignored in full):

| Path | What it is |
|---|---|
| `findings.md` | The running human-readable findings log — explorer-written `### F<N>:` entries plus a harness-appended `## Oracle final pass` section at teardown. Pre-created by the harness with a header so the explorer always has an existing file to append to. |
| `session.log` | The full headless-run transcript: `claude -p --output-format stream-json --verbose`, reformatted by `jq` into readable `[assistant]` / `[tool_use]` / `[tool_result]` / `[result]` lines (each truncated to 500 chars). |
| `session.stderr.log` | `claude -p`'s stderr, kept in its own file so it can never interleave with (and corrupt) the JSON stream `session.log` is parsed from. |
| `oracle-final.txt` / `oracle-final.json` | The final `python -m e2e_oracles` pass, run at teardown while the stack is still up, in both text and JSON form. |
| `traces/` | `@playwright/mcp` session-recording artifacts (page snapshots plus an MCP session directory) — these are MCP session recordings, not Playwright Trace Viewer `.zip` files. |
| `storageState.json`, `mcp.json` | Plumbing, not findings: the minted Playwright storage state and the `--mcp-config` fed to `claude -p`. Useful for debugging why the explorer couldn't reach a page, otherwise ignorable. |
| `lock.ok` | Internal lock-holder bookkeeping (holder-written, self-identifying — its content is the holder process's own PID). Not a finding artifact. |

## 6. The oracles, standalone

The explorer runs the oracles as it goes, and the harness runs one final pass
at teardown — but you can also run them yourself against a stack that's
already up (e.g. mid-`/explore`, or after driving the app by hand):

```bash
cd backend && venv/bin/python -m e2e_oracles [--json] [--check ciphertext|counts|graph|logscan|orphans] [--user ID]
```

- `--check` is repeatable; omit it to run all five (default, sorted order).
- `--json` emits structured `Finding` records instead of the text report —
 useful when quoting evidence into a GitHub issue.
- `--user` selects which seeded user's data to check (default
 `rich-user-active`).
- Exit codes: `0` clean, `1` findings, `2` infra error (a check crashed, or
 itself returned an `oracle-error` finding — treat the whole run's other
 results as untrustworthy when this happens, not just the failing check).
- Needs the stack up (`make e2e-up` or `scripts/explore.sh up`) — it hits the
 local Postgres directly and the backend's `/api/health`-adjacent surface.
- The `logscan` check allowlists known #439 RAG-indexing log noise — an
 allowlisted hit still counts toward "N suppressed" in the summary line, but
 doesn't produce a `Finding`.

## 7. Triage protocol

For each entry in `findings.md` (explorer-written or oracle-appended):

1. **Reproduce it.** By hand in a browser, or by re-running the relevant
 oracle `--check`. Not reproducible → drop it, with a one-line note in your
 triage record of why. Part of reproducing it is asking whether the
 "wrong" output is actually the function-mode seam working as designed
 (see the worked example below) — that's a drop too, just a different
 reason than "couldn't reproduce."
2. **Reproducible and real** → `gh issue create` with the finding block
 (steps/expected/actual), the oracle JSON if there is any, and the
 `.explore/traces/` path attached. Label it as sourced from exploration so
 it's traceable back to a session.
3. **Already-known** (currently: #355, #430, #435, #436, #439, #441) → don't
 file a new issue. Add the new evidence as a comment on the existing issue
 only if it's actually new evidence (a different surface, a clearer repro,
 fresh oracle output) — otherwise it's just noise on an already-tracked bug.
4. **Noise the oracle wrongly flagged** → that's a bug in the oracle itself
 (a missing allowlist entry, or a judge that's too strict/wrong). Fix it as
 a normal PR against `backend/e2e_oracles/` — don't just delete the finding
 from `findings.md` and move on.

Two worked examples from this harness's own acceptance testing show the
difference between a genuine finding and noise the *harness itself*
(not the oracle) produced:

**Genuine, promotable (category 2).** Resuming a tutor session in one
acceptance run produced a real `500` on `POST
/api/graph/<user>/concept-description`, traced through the browser console →
network tab → backend log to `LookupError: SAPLING_MODEL_MODE=function but
no handler is registered for task 'concept_describe'` — `concept_describe`
genuinely has no handler in `backend/agents/function_handlers_e2e.py`. The
`logscan` oracle independently caught the same `500` and traceback in the
backend log, so this has both a UI-observed repro and oracle corroboration.
It isn't on the known-bugs list — exactly the kind of thing this chapter
exists to surface, and a strong promotion candidate (§8).

**Seam artifact — drop it, then improve the harness, not the app (still
category 1).** A different run's explorer flagged what looked like a
"wrong-data" bug: uploading a real syllabus PDF came back with a summary
about gradient descent, category "Lecture notes", and concepts "Gradient
Descent"/"Learning Rate" — content belonging to an entirely different,
already-uploaded document. Alarming-looking, but not a product bug: Chapter
2 always runs the document pipeline in function mode
(`SAPLING_MODEL_MODE=function` /
`SAPLING_FUNCTION_HANDLERS=agents.function_handlers_e2e`, forced by
`scripts/explore.sh`), and
`agents/function_handlers_e2e.py`'s classifier/summary/concepts handlers are
*designed* to return the same fixed constants
(`E2E_DOC_CATEGORY`/`E2E_DOC_ABSTRACT`/`E2E_DOC_CONCEPTS`) for every upload
regardless of the file's real content — the module's own docstring says so
("Fixed output... Echoing request content back would couple the constant to
route-side prompt assembly"). The pipeline was working exactly as designed;
the explorer just didn't know that. **The tell:** the "wrong" text is a
byte-for-byte match to one of `function_handlers_e2e.py`'s `E2E_DOC_*` (or
`E2E_TUTOR_REPLY`, `E2E_QUIZ_*`) constants — here, `E2E_DOC_ABSTRACT`'s
"gradient descent... loss surface... learning rate" wording and the fixed
`lecture_notes` category. If a finding's "wrong" content matches one of
those constants verbatim, it's the seam, not the app — drop it, and if it
recurs, improve `scripts/explore/explorer-prompt.md`'s ground rules to name
the pattern explicitly so a future explorer recognizes it before writing the
stub, rather than filing an issue against the harness's own known-fixed
output.

By contrast, #355 (the graph's duplicated CS subject-root hub) shows up in
the graph oracle on essentially every run against the rich seed data —
that's the known-not-new case category 3 above exists for.

## 8. Promotion pipeline

Triage produces GitHub issues; the pipeline that actually matters is turning
the *right* ones into permanent regression coverage. The epic bar (#403):
**at least 3 findings promoted into Chapter 1 regressions in the first
month.**

Promotion path: issue → a scripted journey in `frontend/e2e/*.spec.ts`,
written in the existing specs' style —

- Fixtures-based `test` (see `frontend/e2e/support/fixtures.ts`), not raw
 Playwright boilerplate.
- Assertions against the database go through `support/db.ts` (raw SQL over a
 direct Postgres connection) — journeys never hand-roll DB plumbing.
- Any new `data-testid` follows `docs/frontend-testids.md`'s naming
 convention; see that doc's "Adding a surface" section for the doc-row +
 `eslint.config.mjs` `files`-array update a new surface needs.
 `npm run lint:baseline` is only for clearing legacy debt in
 `eslint-suppressions.json`, not something a new surface should lean on.

If the promoted journey documents a bug that's still open and must stay red,
ride it in on `test.fixme` with the bug number — exactly how
`frontend/e2e/graph.spec.ts` carries its `#355` assertion today (marked
`fixme`, not relaxed, with companion tests asserting everything #355 does
*not* corrupt).

## 9. Cadence + cost

Suggested cadence: one bounded headless run a week, plus one after any merge
you'd call risky (auth, graph, upload/extraction, or anything touching the
Gemini/function-handler seam).

Cost is bounded but real: each run spends actual Claude tokens (bounded by
`EXPLORE_MAX_TURNS` — the default 40-turn budget is the knob to shrink if
you're cost-conscious). Wall-clock is warm-cache-dependent: observed full
runs (boot → explore → oracle → teardown) land in the 2-4 minute range with
a warm Next build and an already-cached `@playwright/mcp`; budget closer to
~10 minutes for the very first run on a machine, which additionally pays for
the `npx @playwright/mcp` download and a cold test-profile Next build (see
§3).

## 10. Lock protocol & hygiene

The local E2E stack is a **machine singleton** — one Postgres, one backend,
one frontend, bound to fixed local ports. `scripts/explore.sh` enforces this
by holding an `flock` on `/tmp/claude-<uid>/sapling-e2e-stack.lock` for the
entire session (both the headless full pipeline and the interactive
`up`/`down` pair).

Rules that follow from that:

- **Never run two explorations at once** on the same machine — the second
 one fails fast with "stack lock busy" and touches nothing (no wipe, no
 teardown of the first session's stack).
- **Never run an exploration alongside the Chapter 1 Playwright suite** —
 both want the same stack and the same ports.
- **Always end a session via `down`.** For `make explore` this happens
 automatically. For `/explore` or a manual `scripts/explore.sh up`, that
 means `make explore-down` (or `scripts/explore.sh down`) — every time,
 even after an error. Skipping it leaks both the running stack and the
 lock, which then blocks every future exploration until someone runs
 `explore-down` by hand.

## 11. Troubleshooting

- **"e2e stack lock busy"** — another session (yours or someone else's on
 the same machine) is already exploring or mid-`up`. Find and finish it, or
 run `make explore-down` if you're sure it's actually a stale/crashed run.
- **`POST /api/auth/test-login` 404s** — `backend/.env`'s `APP_ENV` is not
 `local` or `test`. This route is hard-gated on that check; fix `.env` and
 restart the backend.
- **The explorer never seemed to drive the browser** (`session.log` has no
 `[tool_use] mcp__playwright__*` lines) — check `.explore/mcp.json`'s flags
 against `npx -y @playwright/mcp@0.0.78 --help`; in particular confirm
 `--isolated` is present (without it, the MCP server reuses one persistent
 Chrome profile per repo path and silently ignores `--storage-state` on a
 reused context — the historical #399 follow-up bug this harness had to
 fix).
- **Oracle exits 2** — either the stack isn't actually up (`make e2e-up`
 first, or check `.e2e/*.log`), or the DB URL the oracle resolved isn't
 pointed at the local stack. Re-run with `--json` to see which check raised.
- **Stale `.explore/` from a previous run** — don't worry about manually
 cleaning it; the next `up` wipes everything except `lock.ok` before
 writing fresh artifacts. Triage anything you care about first (§7).

3 changes: 3 additions & 0 deletions
docs/local-supabase.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -259,6 +259,9 @@ preflights each piece and fails fast with the exact command if one is missing.
Rootless Podman is the documented runtime; Docker is auto-detected as a
fallback (`$CONTAINER_CMD` overrides the choice).

Chapter 2 exploratory testing (`make explore`, the `/explore` skill, and the
e2e oracles) is documented in [e2e-exploration.md](e2e-exploration.md).

## Troubleshooting

- **`supabase start` hangs on health checks** — usually the analytics/vector
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotion (#401)#445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

AndresL230
merged 5 commits into

from

Jul 28, 2026
 Merged


docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotion (#401)
#445
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


5 commits


Select commit
Hold shift + click to select a range
e4a054f

docs(e2e): Chapter 2 exploration runbook — kick-off, triage, promotio…


AndresL230 Jul 28, 2026
befba6f

docs(e2e): stop citing a gitignored planning file from the runbook


AndresL230 Jul 28, 2026
3c65b97

docs(e2e): fix awkward line wrap in runbook


AndresL230 Jul 28, 2026
56511af

docs(e2e): fix runbook per code-review — seam artifact vs real bug, t…


AndresL230 Jul 28, 2026
df2672b

docs(e2e): rewrap code span so the module path doesn't split mid-toke…


AndresL230 Jul 28, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


313 changes: 313 additions & 0 deletions
docs/e2e-exploration.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,313 @@
# Chapter 2: exploratory testing

This is the runbook for Sapling's Chapter 2 — LLM-driven exploratory testing
of the running app. Read this before your first `make explore`.

## 1. What this is

Sapling has two E2E chapters, and they answer different questions.

| | Chapter 1 | Chapter 2 |
|---|---|---|
| What it is | Scripted, deterministic journeys | LLM-driven exploration |
| Where it lives | `frontend/e2e/*.spec.ts` (Playwright) | `scripts/explore.sh`, `.claude/skills/explore/`, `backend/e2e_oracles/` |
| Gates CI? | Yes — `.github/workflows/e2e.yml`, main-only | Never |
| Runs when | Every push to main | On demand, local-only (`make explore` / `/explore`) |
| Judge | Fixed assertions + `support/db.ts` raw SQL | The e2e oracles (`backend/e2e_oracles`) or a reproducible captured failure |

Chapter 2 exists because a scripted suite only ever re-checks what someone
already thought to assert. An LLM with a real browser will click things,
double-submit forms, hit Back mid-flow, and generally behave like a curious
student — surfacing bugs nobody wrote a journey for yet.

The core rule, verbatim from the epic (#403), governs everything below:

> The LLM explores; only oracles (or a reproducible captured failure) make
> something a finding.

That's why every exploration session ends with a deterministic oracle pass,
and why the triage protocol (§7) insists on reproduction before anything
becomes a GitHub issue. The explorer's own prose is a lead, not a verdict.

## 2. Prerequisites

Everything Chapter 1's local stack needs (see
[local-supabase.md](local-supabase.md)), plus two things specific to Chapter 2:

- Supabase CLI on `PATH`
- Rootless Podman (documented) or Docker (auto-detected fallback)
- `backend/venv` (`python -m venv backend/venv && backend/venv/bin/pip install -r backend/requirements.txt`)
- `frontend/node_modules` (`cd frontend && npm ci`)
- `backend/.env` with `APP_ENV=local` (or `test`) — without it, `POST
 /api/auth/test-login` 404s and the harness can't mint a session
- The `claude` CLI on `PATH` (drives the headless explorer via `claude -p`)
- Network access for the first run — `npx` downloads and pins
 `@playwright/mcp@0.0.78` the first time it's invoked; afterward it's cached

You do **not** need to set a real `GEMINI_API_KEY` or `SEED_RICH` yourself —
`scripts/explore.sh` forces both. A real key in your shell is not
used and never billed by this harness.

## 3. Kick-off: headless

```bash
make explore
```

This runs the full pipeline: boot the stack, mint a session, hand the
explorer prompt to a Playwright-MCP-armed `claude -p`, run the oracle final
pass, then tear everything down. It takes the machine-singleton stack lock
for the whole run (see §10). With warm caches (a repo you've already run
`make e2e-up` or `make explore` on before), the whole thing typically
finishes in a few minutes end-to-end; the first-ever run on a machine can
take closer to ~10 minutes (see §9's cost note — it's the `npx
@playwright/mcp` download plus a cold test-profile Next build, not the
exploration itself).

Env knobs (all optional, read by `scripts/explore.sh`):

| Knob | Default | Meaning |
|---|---|---|
| `EXPLORE_MAX_TURNS` | `40` | Turn budget passed to `claude -p --max-turns`. Bounds both wall-clock and token cost. |
| `EXPLORE_MODEL` | `sonnet` | Model alias passed to `claude -p --model`. |
| `EXPLORE_HEADED` | `0` | `1` = watch the browser drive itself (non-headless); `0` = headless. |
| `EXPLORE_USER` | `rich-user-active` | Which seeded `rich-*` user to sign in as. One of `rich-user-active`, `rich-user-second`, `rich-user-new`, `rich-user-pending`, `rich-user-admin`. |

Example: a slower, watchable run as a different seeded user:

```bash
EXPLORE_HEADED=1 EXPLORE_USER=rich-user-admin EXPLORE_MAX_TURNS=60 make explore
```

Two things worth internalizing before your first run:

- **`.explore/` is wiped at `up`, but only after the lock is acquired.** If
 you want to keep a previous run's findings, triage them (§7) before
 re-running — a second `make explore` deletes the prior artifacts (except
 the in-flight `lock.ok`, which is bookkeeping, not a finding).
- **A crashed run is recoverable.** If a prior `make explore` died mid-run
 (killed session, machine sleep, whatever), `make explore-down` (same as
 `scripts/explore.sh down`) runs the oracle final pass against whatever is
 still up, appends it to `findings.md`, tears the stack down, and releases
 the lock — safe to run even with nothing up.

## 4. Kick-off: interactive

```
/explore
```

Run this as a slash command in a Claude Code session opened on this repo.
Unlike `make explore`, you watch (and can steer) the exploration live:

1. The skill runs `scripts/explore.sh up` itself (this takes minutes —
 Supabase, migrations, seed, backend, a test-profile Next build — and takes
 the same stack lock `make explore` does; if another session holds it, the
 skill stops and tells you).
2. It opens a real browser tab (Playwright MCP if configured for the session,
 else the Claude-in-Chrome tools) at `http://localhost:3000` and signs in as
 `rich-user-active` by minting a session directly from the page.
3. It reads `scripts/explore/explorer-prompt.md` as its own mission briefing
 and starts exploring — narrating what it's trying so you can redirect it
 at any point. Same persona, same break-things mandate, same
 report-never-fix rule, same oracle cadence, same findings format as the
 headless run.
4. It finishes with `scripts/explore.sh down` — same oracle final pass,
 same teardown, same lock release.

The output is identical in shape to a headless run: `.explore/findings.md`
with explorer-written entries plus the harness-appended oracle pass. The only
difference is you're in the loop instead of reading a transcript after the
fact.

## 5. Outputs

Everything lands in `.explore/` at the repo root (gitignored in full):

| Path | What it is |
|---|---|
| `findings.md` | The running human-readable findings log — explorer-written `### F<N>:` entries plus a harness-appended `## Oracle final pass` section at teardown. Pre-created by the harness with a header so the explorer always has an existing file to append to. |
| `session.log` | The full headless-run transcript: `claude -p --output-format stream-json --verbose`, reformatted by `jq` into readable `[assistant]` / `[tool_use]` / `[tool_result]` / `[result]` lines (each truncated to 500 chars). |
| `session.stderr.log` | `claude -p`'s stderr, kept in its own file so it can never interleave with (and corrupt) the JSON stream `session.log` is parsed from. |
| `oracle-final.txt` / `oracle-final.json` | The final `python -m e2e_oracles` pass, run at teardown while the stack is still up, in both text and JSON form. |
| `traces/` | `@playwright/mcp` session-recording artifacts (page snapshots plus an MCP session directory) — these are MCP session recordings, not Playwright Trace Viewer `.zip` files. |
| `storageState.json`, `mcp.json` | Plumbing, not findings: the minted Playwright storage state and the `--mcp-config` fed to `claude -p`. Useful for debugging why the explorer couldn't reach a page, otherwise ignorable. |
| `lock.ok` | Internal lock-holder bookkeeping (holder-written, self-identifying — its content is the holder process's own PID). Not a finding artifact. |

## 6. The oracles, standalone

The explorer runs the oracles as it goes, and the harness runs one final pass
at teardown — but you can also run them yourself against a stack that's
already up (e.g. mid-`/explore`, or after driving the app by hand):

```bash
cd backend && venv/bin/python -m e2e_oracles [--json] [--check ciphertext|counts|graph|logscan|orphans] [--user ID]
```

- `--check` is repeatable; omit it to run all five (default, sorted order).
- `--json` emits structured `Finding` records instead of the text report —
 useful when quoting evidence into a GitHub issue.
- `--user` selects which seeded user's data to check (default
 `rich-user-active`).
- Exit codes: `0` clean, `1` findings, `2` infra error (a check crashed, or
 itself returned an `oracle-error` finding — treat the whole run's other
 results as untrustworthy when this happens, not just the failing check).
- Needs the stack up (`make e2e-up` or `scripts/explore.sh up`) — it hits the
 local Postgres directly and the backend's `/api/health`-adjacent surface.
- The `logscan` check allowlists known #439 RAG-indexing log noise — an
 allowlisted hit still counts toward "N suppressed" in the summary line, but
 doesn't produce a `Finding`.

## 7. Triage protocol

For each entry in `findings.md` (explorer-written or oracle-appended):

1. **Reproduce it.** By hand in a browser, or by re-running the relevant
 oracle `--check`. Not reproducible → drop it, with a one-line note in your
 triage record of why. Part of reproducing it is asking whether the
 "wrong" output is actually the function-mode seam working as designed
 (see the worked example below) — that's a drop too, just a different
 reason than "couldn't reproduce."
2. **Reproducible and real** → `gh issue create` with the finding block
 (steps/expected/actual), the oracle JSON if there is any, and the
 `.explore/traces/` path attached. Label it as sourced from exploration so
 it's traceable back to a session.
3. **Already-known** (currently: #355, #430, #435, #436, #439, #441) → don't
 file a new issue. Add the new evidence as a comment on the existing issue
 only if it's actually new evidence (a different surface, a clearer repro,
 fresh oracle output) — otherwise it's just noise on an already-tracked bug.
4. **Noise the oracle wrongly flagged** → that's a bug in the oracle itself
 (a missing allowlist entry, or a judge that's too strict/wrong). Fix it as
 a normal PR against `backend/e2e_oracles/` — don't just delete the finding
 from `findings.md` and move on.

Two worked examples from this harness's own acceptance testing show the
difference between a genuine finding and noise the *harness itself*
(not the oracle) produced:

**Genuine, promotable (category 2).** Resuming a tutor session in one
acceptance run produced a real `500` on `POST
/api/graph/<user>/concept-description`, traced through the browser console →
network tab → backend log to `LookupError: SAPLING_MODEL_MODE=function but
no handler is registered for task 'concept_describe'` — `concept_describe`
genuinely has no handler in `backend/agents/function_handlers_e2e.py`. The
`logscan` oracle independently caught the same `500` and traceback in the
backend log, so this has both a UI-observed repro and oracle corroboration.
It isn't on the known-bugs list — exactly the kind of thing this chapter
exists to surface, and a strong promotion candidate (§8).

**Seam artifact — drop it, then improve the harness, not the app (still
category 1).** A different run's explorer flagged what looked like a
"wrong-data" bug: uploading a real syllabus PDF came back with a summary
about gradient descent, category "Lecture notes", and concepts "Gradient
Descent"/"Learning Rate" — content belonging to an entirely different,
already-uploaded document. Alarming-looking, but not a product bug: Chapter
2 always runs the document pipeline in function mode
(`SAPLING_MODEL_MODE=function` /
`SAPLING_FUNCTION_HANDLERS=agents.function_handlers_e2e`, forced by
`scripts/explore.sh`), and
`agents/function_handlers_e2e.py`'s classifier/summary/concepts handlers are
*designed* to return the same fixed constants
(`E2E_DOC_CATEGORY`/`E2E_DOC_ABSTRACT`/`E2E_DOC_CONCEPTS`) for every upload
regardless of the file's real content — the module's own docstring says so
("Fixed output... Echoing request content back would couple the constant to
route-side prompt assembly"). The pipeline was working exactly as designed;
the explorer just didn't know that. **The tell:** the "wrong" text is a
byte-for-byte match to one of `function_handlers_e2e.py`'s `E2E_DOC_*` (or
`E2E_TUTOR_REPLY`, `E2E_QUIZ_*`) constants — here, `E2E_DOC_ABSTRACT`'s
"gradient descent... loss surface... learning rate" wording and the fixed
`lecture_notes` category. If a finding's "wrong" content matches one of
those constants verbatim, it's the seam, not the app — drop it, and if it
recurs, improve `scripts/explore/explorer-prompt.md`'s ground rules to name
the pattern explicitly so a future explorer recognizes it before writing the
stub, rather than filing an issue against the harness's own known-fixed
output.

By contrast, #355 (the graph's duplicated CS subject-root hub) shows up in
the graph oracle on essentially every run against the rich seed data —
that's the known-not-new case category 3 above exists for.

## 8. Promotion pipeline

Triage produces GitHub issues; the pipeline that actually matters is turning
the *right* ones into permanent regression coverage. The epic bar (#403):
**at least 3 findings promoted into Chapter 1 regressions in the first
month.**

Promotion path: issue → a scripted journey in `frontend/e2e/*.spec.ts`,
written in the existing specs' style —

- Fixtures-based `test` (see `frontend/e2e/support/fixtures.ts`), not raw
 Playwright boilerplate.
- Assertions against the database go through `support/db.ts` (raw SQL over a
 direct Postgres connection) — journeys never hand-roll DB plumbing.
- Any new `data-testid` follows `docs/frontend-testids.md`'s naming
 convention; see that doc's "Adding a surface" section for the doc-row +
 `eslint.config.mjs` `files`-array update a new surface needs.
 `npm run lint:baseline` is only for clearing legacy debt in
 `eslint-suppressions.json`, not something a new surface should lean on.

If the promoted journey documents a bug that's still open and must stay red,
ride it in on `test.fixme` with the bug number — exactly how
`frontend/e2e/graph.spec.ts` carries its `#355` assertion today (marked
`fixme`, not relaxed, with companion tests asserting everything #355 does
*not* corrupt).

## 9. Cadence + cost

Suggested cadence: one bounded headless run a week, plus one after any merge
you'd call risky (auth, graph, upload/extraction, or anything touching the
Gemini/function-handler seam).

Cost is bounded but real: each run spends actual Claude tokens (bounded by
`EXPLORE_MAX_TURNS` — the default 40-turn budget is the knob to shrink if
you're cost-conscious). Wall-clock is warm-cache-dependent: observed full
runs (boot → explore → oracle → teardown) land in the 2-4 minute range with
a warm Next build and an already-cached `@playwright/mcp`; budget closer to
~10 minutes for the very first run on a machine, which additionally pays for
the `npx @playwright/mcp` download and a cold test-profile Next build (see
§3).

## 10. Lock protocol & hygiene

The local E2E stack is a **machine singleton** — one Postgres, one backend,
one frontend, bound to fixed local ports. `scripts/explore.sh` enforces this
by holding an `flock` on `/tmp/claude-<uid>/sapling-e2e-stack.lock` for the
entire session (both the headless full pipeline and the interactive
`up`/`down` pair).

Rules that follow from that:

- **Never run two explorations at once** on the same machine — the second
 one fails fast with "stack lock busy" and touches nothing (no wipe, no
 teardown of the first session's stack).
- **Never run an exploration alongside the Chapter 1 Playwright suite** —
 both want the same stack and the same ports.
- **Always end a session via `down`.** For `make explore` this happens
 automatically. For `/explore` or a manual `scripts/explore.sh up`, that
 means `make explore-down` (or `scripts/explore.sh down`) — every time,
 even after an error. Skipping it leaks both the running stack and the
 lock, which then blocks every future exploration until someone runs
 `explore-down` by hand.

## 11. Troubleshooting

- **"e2e stack lock busy"** — another session (yours or someone else's on
 the same machine) is already exploring or mid-`up`. Find and finish it, or
 run `make explore-down` if you're sure it's actually a stale/crashed run.
- **`POST /api/auth/test-login` 404s** — `backend/.env`'s `APP_ENV` is not
 `local` or `test`. This route is hard-gated on that check; fix `.env` and
 restart the backend.
- **The explorer never seemed to drive the browser** (`session.log` has no
 `[tool_use] mcp__playwright__*` lines) — check `.explore/mcp.json`'s flags
 against `npx -y @playwright/mcp@0.0.78 --help`; in particular confirm
 `--isolated` is present (without it, the MCP server reuses one persistent
 Chrome profile per repo path and silently ignores `--storage-state` on a
 reused context — the historical #399 follow-up bug this harness had to
 fix).
- **Oracle exits 2** — either the stack isn't actually up (`make e2e-up`
 first, or check `.e2e/*.log`), or the DB URL the oracle resolved isn't
 pointed at the local stack. Re-run with `--json` to see which check raised.
- **Stale `.explore/` from a previous run** — don't worry about manually
 cleaning it; the next `up` wipes everything except `lock.ok` before
 writing fresh artifacts. Triage anything you care about first (§7).

3 changes: 3 additions & 0 deletions
docs/local-supabase.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -259,6 +259,9 @@ preflights each piece and fails fast with the exact command if one is missing.
Rootless Podman is the documented runtime; Docker is auto-detected as a
fallback (`$CONTAINER_CMD` overrides the choice).

Chapter 2 exploratory testing (`make explore`, the `/explore` skill, and the
e2e oracles) is documented in [e2e-exploration.md](e2e-exploration.md).

## Troubleshooting

- **`supabase start` hangs on health checks** — usually the analytics/vector
Expand Down
Loading