Uh oh!
There was an error while loading. Please reload this page.
feat(e2e): deterministic oracle module — python -m e2e_oracles (#400) - #443
Conversation
#400) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ts, orphans (#400) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Self-review cleanup on #400 judges — collapse the exception and plain==value paths in ciphertext_findings into one not_encrypted flag; same Finding either way, less repetition. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wires the Task 1-2 judges/findings/logscan to the live local stack: gather.py holds one shared lazily-opened psycopg connection plus authed httpx calls (mint_session cookie), guarded by require_local (exact-hostname loopback check) on both SUPABASE_DB_URL and --base-url so this can never reach staging/production. __main__.py exposes the CHECKS registry and the python -m e2e_oracles CLI contract (--json/--check/--user/--base-url/--log, exit 0/1/2) that the explore harness and explorer prompt depend on verbatim. Hermetic tests monkeypatch CHECKS wholesale -- no DB or HTTP in the suite. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… stale encryption docstring (#400) - __main__.main() now also sets infra_error when a check RETURNS a Finding(oracle="oracle-error", ...) (e.g. run_logscan's missing-log-file case), not just when a check raises. Previously --check logscan against a nonexistent log exited 1 instead of the contractually-required 2. - Corrected services/encryption.py's stale ROLLOUT NOTES claim that messages.content, room_messages.text, and sessions.summary_json were "not encrypted yet" — all three are encrypted in practice (seeded via encrypt_if_present/encrypt_json in db/seed_local_rich.py, listed in CLAUDE.md's Gotchas, and asserted ciphertext-at-rest by frontend/e2e/tutor.spec.ts and study-room.spec.ts). Comment-only change, no encryption code or manifest touched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Warning Review limit reached
Next review available in:38 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Uh oh!
There was an error while loading. Please reload this page.
| "2026-07-27 22:17:51,318 [ERROR] routes.documents: [RAG] " | ||
| "_index_document_chunks failed for doc 0b65", |
| "2026-07-27 22:17:08,060 [INFO] httpx: HTTP Request: POST " | ||
| 'http://127.0.0.1:54321/storage/v1/bucket "HTTP/1.1 400 Bad Request"', |
| @@ -0,0 +1,114 @@ | |||
| """Hermetic tests for the #400 backend-log scanner. No stack, no DB.""" | |||
| from e2e_oracles.findings import Finding, render_json, render_text # noqa: F401 — Finding is part of the public interface under test | |||
| _DEFAULT_LOG = _REPO_ROOT / ".e2e" / "backend.log" | ||
| CHECKS: dict[str, Callable[[argparse.Namespace], tuple[list[Finding], int]]] = { | ||
| "graph": lambda args: gather.run_graph(args), |
| CHECKS: dict[str, Callable[[argparse.Namespace], tuple[list[Finding], int]]] = { | ||
| "graph": lambda args: gather.run_graph(args), | ||
| "counts": lambda args: gather.run_counts(args), |
| CHECKS: dict[str, Callable[[argparse.Namespace], tuple[list[Finding], int]]] = { | ||
| "graph": lambda args: gather.run_graph(args), | ||
| "counts": lambda args: gather.run_counts(args), | ||
| "ciphertext": lambda args: gather.run_ciphertext(args), |
| "graph": lambda args: gather.run_graph(args), | ||
| "counts": lambda args: gather.run_counts(args), | ||
| "ciphertext": lambda args: gather.run_ciphertext(args), | ||
| "logscan": lambda args: gather.run_logscan(args), |
| "counts": lambda args: gather.run_counts(args), | ||
| "ciphertext": lambda args: gather.run_ciphertext(args), | ||
| "logscan": lambda args: gather.run_logscan(args), | ||
| "orphans": lambda args: gather.run_orphans(args), |
Deploying with |
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs | frontend-staging | e514fa5 | Commit Preview URL Branch Preview URL | Jul 28 2026, 07:53 AM |
Caution Failed to replace (edit) comment. This is likely due to insufficient permissions or the comment being deleted. Error details |
AndresL230
commented
Jul 28, 2026
Code reviewFound 2 issues:
Sapling/backend/e2e_oracles/logscan.py Lines 36 to 38 in b7d24dd
Sapling/backend/e2e_oracles/logscan.py Lines 32 to 34 in b7d24dd 🤖 Generated with Claude Code - If this code review was useful, please react with 👍. Otherwise, react with 👎. |
…e-request block close (PR #443) Two PR-review findings on backend/e2e_oracles/logscan.py: - KEY_RE required a class-name prefix (e.g. `ValueError`) before Error/Exception/Interrupt/Exit, so bare `Exception: boom` terminal lines never matched; _block_key then fell back to block_lines[0], aggregating unrelated tracebacks under the generic "Traceback (most recent call last):" key. Made the prefix optional. - NEW_LOG_LINE_RE didn't recognize the post-ANSI-strip HH:MM:SS.mmm pre-request log line shape, so it got swallowed into an open traceback block instead of closing it, merging a following traceback into the same block. Added that alternative. Added three regression tests (RED before fix, GREEN after); the existing ExceptionGroup test still resolves to the KeyError line since it's still the last KEY_RE match in that block. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| # The third alternative is the post-ANSI-strip shape of the pre-request log line | ||
| # (e.g. "22:17:08.488 GET /api/health") — scanning runs after ANSI stripping. | ||
| NEW_LOG_LINE_RE = re.compile( | ||
| r"^(INFO|ERROR|WARNING|DEBUG|CRITICAL):|^\d{4}-\d{2}-\d{2} |^\d{2}:\d{2}:\d{2}\.\d+ " |
AndresL230
commented
Jul 28, 2026
Both review findings fixed in e514fa5 (bare-Exception KEY_RE match + ANSI pre-request block close), TDD'd with 3 new regression tests; scoped re-review confirms no regex regressions (ExceptionGroup keying, ASGI header, chained-exception blocks all probed). |
Chapter 2 of the E2E program (epic #403), PR 1 of 3. Closes#400.
Adds
backend/e2e_oracles/— deterministic checks the exploratory-testing harness (#399, next PR) invokes mid-session and post-session. The LLM explores; these oracles decide what counts as a finding.What's here
findings.py— sharedFindingrecord + text/JSON rendering.logscan.py— streaming scan of.e2e/backend.log: 5xx from both the uvicorn access log and thesapling.requestcorrelation lines (aggregated per route), traceback blocks in all three observed shapes (bare, ASGI, ExceptionGroup+/|gutters), ANSI-stripped, with the RAG embedding path sits below the SAPLING_MODEL_MODE seam — live embed calls fire even in function mode #439 RAG-indexing noise allowlisted (suppressed + counted, never findings).judges.py— pure functions: graph payload integrity vs DB (the Duplicate subject_root node in /api/graph → React duplicate-key warnings in KnowledgeGraph2D #355 oracle — duplicate node/edge ids, expected counts = db nodes + distinct enrolled courses / drawable edges + spokes, per-course subject-root multiplicity), ciphertext-at-rest (decrypt-must-succeed-and-differ; injected decrypt_fn), API-vs-DB count integrity, orphaned rows.gather.py+__main__.py— the CLI:venv/bin/python -m e2e_oracles [--json] [--check graph|counts|ciphertext|logscan|orphans] [--user] [--base-url] [--log]. Exit 0 clean / 1 findings / 2 infra error. Raw SQL (psycopg, SELECT-only) is sanctioned here as test tooling, mirroringtests/integration/conftest.py: same exact-hostname loopback guards (raise, never skip) on bothSUPABASE_DB_URLand--base-url; session minted viaservices.session_tokens.CHECKSregistry is monkeypatched).services/encryption.py—messages.content/room_messages.text/sessions.summary_jsonare encrypted (seeder + Chapter 1 journeys prove it at rest).Live verification (evidence in the SDD report)
Against the seeded stack (function mode, dummy key): full run exits 1 catching the #355 signature — duplicate
subject_root__rich-course-cs101, node/edge count mismatches with formula parity tograph.spec.ts'sgraphExpectations()(absolute numbers reconciled against 2 legitimate non-seed rows in the long-lived local volume);--check ciphertext,orphans,logscanall exit 0 with the logscan result spot-checked against the raw log;--jsonparses.Hermetic suite: 1151 passed / 26 skipped (+ the pre-existing #354
test_ocr_pipelineloop-affinity error, unrelated).ruff checkclean.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Tests
Documentation