Uh oh!
There was an error while loading. Please reload this page.
feat(evals): complete extraction-accuracy harness + baselines (#148) - #455
Merged
Conversation
Finish the agent eval harness so migrated agents are validated on accuracy, not just smoke-tested — the evidence base the gemini_service cutover (#151) needs. - Record 80 cassettes across the five offline datasets (classification, summary, concepts, syllabus, quiz); replay is now deterministic + keyless. - Gate on regression below a committed baseline (baselines.json) instead of "< 1.0" — the harness measures accuracy, it doesn't assume perfection. - Retry transient 503/429 while recording; force UTF-8 output so non-ASCII cases don't crash rich on a Windows cp1252 console. - Add run_all.py (one combined scored run) and enable evals.yml to run it in replay mode on PRs touching backend/agents/** or the harness. - Document the record/refresh workflow (tests/evals/README.md, README) and the design + baselines (ADR 0020). - Exclude chat_tutor: its retrieval tool reads a live Supabase and can't run offline; folded into the graph-grounded tutor work (#149). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Warning Review limit reached
Next review available in:44 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (45)
📝 WalkthroughWalkthroughThe pull request adds a unified offline evaluation harness with committed cassettes, per-evaluator baselines, transient-provider retries, consolidated CI execution, and expanded fixtures for concept extraction, classification, summaries, quizzes, and syllabus extraction. ChangesOffline evaluation harness
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
| last_exc = exc | ||
| await asyncio.sleep(_TRANSIENT_BACKOFF_SECONDS * (attempt + 1)) | ||
| # Unreachable: the loop either returns or raises. Kept for type-checkers. | ||
| raise last_exc # type: ignore[misc] |
| for stream in (sys.stdout, sys.stderr): | ||
| try: | ||
| stream.reconfigure(encoding="utf-8", errors="replace") | ||
| except (AttributeError, ValueError): |
Deploying with |
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs | frontend-staging | c32a671 | Commit Preview URL Branch Preview URL | Jul 29 2026, 09:01 AM |
There was a problem hiding this comment.
Actionable comments posted: 11
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/tests/evals/_replay.py`:
- Around line 146-158: Make baseline refreshes atomic: in
backend/tests/evals/_replay.py lines 146-158, update the replay baseline-refresh
flow to expose calculated scores without writing immediately when invoked by the
combined runner, while preserving standalone persistence behavior; in
backend/tests/evals/run_all.py lines 46-53, have the combined runner collect
every dataset’s scores and write the complete baseline map only after all
datasets succeed.
- Around line 169-174: Update the replay score validation around the baseline
comparison loop to detect evaluator names present in scores but absent from
baseline, append a regression requiring an explicit baseline refresh, and ensure
these unexpected keys cause the replay to fail alongside existing missing and
below-tolerance checks.
In
`@backend/tests/evals/cassettes/document_summary/bullet_only_presentation.json`:
- Around line 2-10: Update the cassette’s abstract and key_points so ATP and
NADPH are described as inputs consumed by the Calvin cycle and products of the
light-dependent reactions, not as overall net or final products. Preserve
glucose as the relevant carbon product and retain the existing coverage of
photosynthesis stages and limiting factors.
In `@backend/tests/evals/cassettes/document_summary/lab_procedure.json`:
- Line 2: Update the abstract value in the lab procedure cassette to describe
the persistent pink phenolphthalein color as the titration endpoint instead of
the equivalence point; leave the remaining procedure description unchanged.
In `@backend/tests/evals/cassettes/document_summary/short_syllabus_excerpt.json`:
- Around line 4-8: Rewrite every string in the cassette’s key_points array as a
complete sentence while preserving the same course details. Ensure the four
entries remain semantically equivalent and conform to the sentence-based output
contract described by the summary implementation.
In
`@backend/tests/evals/cassettes/quiz_generation/adaptive_downshift_struggling_student.json`:
- Around line 8-14: Update the options for the Fibonacci base-case question in
the cassette so only one option expresses the correct behavior; replace the
equivalent options 1–3 duplicates with genuinely incorrect distractors, while
retaining the incorrect option that reverses the base-case results.
In
`@backend/tests/evals/cassettes/quiz_generation/easy_mcq_physics_definitions.json`:
- Around line 5-9: Update both occurrences of the Newton’s First Law answer text
in the quiz cassette—under “correct_answer” and the matching “options” entry—to
include objects remaining at rest as well as continuing in uniform motion, using
the complete definition consistent with the explanation.
In `@backend/tests/evals/cassettes/quiz_generation/hard_mcq_real_analysis.json`:
- Around line 18-28: Update the question field in the uniform-convergence
cassette to remove the clause asserting that f is continuous on E. Keep the
remaining premises about uniform convergence and continuity of each f_n, so the
correct answer establishes continuity of f as the theorem’s conclusion.
- Around line 4-14: The real-analysis fixture contains two defensible answers:
update the distractor in hard_mcq_real_analysis.json at lines 4-14 to be clearly
false while preserving the intended correct answer. In
hard_mcq_theory_of_computation.json at lines 18-28, remove or rewrite the true
context-free-language option and complete the membership explanation so every
fixture has exactly one defensible answer.
In `@backend/tests/evals/cassettes/quiz_generation/medium_mcq_econ.json`:
- Around line 18-28: Update the quiz item’s question, correct_answer, options,
and explanation to use one consistent price-elasticity sign convention: either
explicitly ask for the absolute value and retain 2.0, or consistently use -2.0
throughout. Keep all related fields aligned.
In `@backend/tests/evals/README.md`:
- Around line 3-4: Adjust the introductory paragraph in the README so the `#148`
issue reference is moved to the preceding line while remaining part of the
paragraph, avoiding the markdownlint MD018 violation. Preserve the existing
wording and both issue references.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 5b4d320f-83b4-40ff-824e-1f316274f708
📒 Files selected for processing (87)
.github/workflows/evals.ymlREADME.mdbackend/tests/evals/README.mdbackend/tests/evals/_replay.pybackend/tests/evals/baselines.jsonbackend/tests/evals/cassettes/concept_extraction/adversarial_problem_n_headers.jsonbackend/tests/evals/cassettes/concept_extraction/adversarial_week_n_slides.jsonbackend/tests/evals/cassettes/concept_extraction/bio_cellular_respiration.jsonbackend/tests/evals/cassettes/concept_extraction/chemistry_thermodynamics.jsonbackend/tests/evals/cassettes/concept_extraction/code_heavy_algorithms.jsonbackend/tests/evals/cassettes/concept_extraction/cs_networking_tcp.jsonbackend/tests/evals/cassettes/concept_extraction/econ_supply_demand.jsonbackend/tests/evals/cassettes/concept_extraction/exploratory_empty_input.jsonbackend/tests/evals/cassettes/concept_extraction/history_reading_industrial_rev.jsonbackend/tests/evals/cassettes/concept_extraction/long_lecture_neural_networks.jsonbackend/tests/evals/cassettes/concept_extraction/math_text_regression_complexity.jsonbackend/tests/evals/cassettes/concept_extraction/narrow_assignment_proofs.jsonbackend/tests/evals/cassettes/concept_extraction/physics_classical_mechanics.jsonbackend/tests/evals/cassettes/concept_extraction/psychology_study_guide.jsonbackend/tests/evals/cassettes/concept_extraction/whole_course_syllabus_calc1.jsonbackend/tests/evals/cassettes/document_classification/assignment.jsonbackend/tests/evals/cassettes/document_classification/assignment_list_with_weights.jsonbackend/tests/evals/cassettes/document_classification/course_handbook_with_schedule.jsonbackend/tests/evals/cassettes/document_classification/course_welcome_letter_no_schedule.jsonbackend/tests/evals/cassettes/document_classification/cs_programming_project.jsonbackend/tests/evals/cassettes/document_classification/exam.jsonbackend/tests/evals/cassettes/document_classification/french_assignment.jsonbackend/tests/evals/cassettes/document_classification/grading_rubric.jsonbackend/tests/evals/cassettes/document_classification/lab_handout_chemistry.jsonbackend/tests/evals/cassettes/document_classification/latex_source_problem_set.jsonbackend/tests/evals/cassettes/document_classification/lecture_notes.jsonbackend/tests/evals/cassettes/document_classification/math_heavy_lecture_notes.jsonbackend/tests/evals/cassettes/document_classification/math_problem_set.jsonbackend/tests/evals/cassettes/document_classification/minimal_syllabus.jsonbackend/tests/evals/cassettes/document_classification/near_empty_doc.jsonbackend/tests/evals/cassettes/document_classification/past_history_exam.jsonbackend/tests/evals/cassettes/document_classification/reading_list_pdf.jsonbackend/tests/evals/cassettes/document_classification/required_readings_with_weeks.jsonbackend/tests/evals/cassettes/document_classification/research_paper.jsonbackend/tests/evals/cassettes/document_classification/slides_sparse_bullets.jsonbackend/tests/evals/cassettes/document_classification/spanish_syllabus.jsonbackend/tests/evals/cassettes/document_classification/study_guide_overlapping_syllabus.jsonbackend/tests/evals/cassettes/document_classification/syllabus_embedded_in_handbook.jsonbackend/tests/evals/cassettes/document_classification/textbook_chapter.jsonbackend/tests/evals/cassettes/document_classification/typical_university_syllabus.jsonbackend/tests/evals/cassettes/document_summary/bullet_only_presentation.jsonbackend/tests/evals/cassettes/document_summary/course_handbook_policies.jsonbackend/tests/evals/cassettes/document_summary/empty_document.jsonbackend/tests/evals/cassettes/document_summary/glossary_style_reading.jsonbackend/tests/evals/cassettes/document_summary/lab_procedure.jsonbackend/tests/evals/cassettes/document_summary/lecture_backpropagation.jsonbackend/tests/evals/cassettes/document_summary/linear_algebra_problem_set.jsonbackend/tests/evals/cassettes/document_summary/literature_review.jsonbackend/tests/evals/cassettes/document_summary/math_heavy_notes_with_dollar_signs.jsonbackend/tests/evals/cassettes/document_summary/multi_topic_study_guide.jsonbackend/tests/evals/cassettes/document_summary/short_syllabus_excerpt.jsonbackend/tests/evals/cassettes/document_summary/spanish_syllabus.jsonbackend/tests/evals/cassettes/document_summary/statement_of_work.jsonbackend/tests/evals/cassettes/document_summary/textbook_chapter_thesis.jsonbackend/tests/evals/cassettes/document_summary/verbose_redundant_topic.jsonbackend/tests/evals/cassettes/quiz_generation/adaptive_downshift_struggling_student.jsonbackend/tests/evals/cassettes/quiz_generation/easy_mcq_basic_biology.jsonbackend/tests/evals/cassettes/quiz_generation/easy_mcq_intro_calculus.jsonbackend/tests/evals/cassettes/quiz_generation/easy_mcq_physics_definitions.jsonbackend/tests/evals/cassettes/quiz_generation/hard_mcq_real_analysis.jsonbackend/tests/evals/cassettes/quiz_generation/hard_mcq_theory_of_computation.jsonbackend/tests/evals/cassettes/quiz_generation/medium_mcq_data_structures.jsonbackend/tests/evals/cassettes/quiz_generation/medium_mcq_econ.jsonbackend/tests/evals/cassettes/quiz_generation/medium_mcq_organic_chem.jsonbackend/tests/evals/cassettes/quiz_generation/spaced_repetition_revives_stale_concept.jsonbackend/tests/evals/cassettes/syllabus_extraction/course_handbook_disguised.jsonbackend/tests/evals/cassettes/syllabus_extraction/exam_only_schedule.jsonbackend/tests/evals/cassettes/syllabus_extraction/grading_tbd.jsonbackend/tests/evals/cassettes/syllabus_extraction/mandarin_syllabus_excerpt.jsonbackend/tests/evals/cassettes/syllabus_extraction/minimal_topics_only.jsonbackend/tests/evals/cassettes/syllabus_extraction/mixed_concrete_and_relative_dates.jsonbackend/tests/evals/cassettes/syllabus_extraction/no_grading_breakdown.jsonbackend/tests/evals/cassettes/syllabus_extraction/overlapping_deadlines.jsonbackend/tests/evals/cassettes/syllabus_extraction/points_based_rubric.jsonbackend/tests/evals/cassettes/syllabus_extraction/policies_only_no_graded_items.jsonbackend/tests/evals/cassettes/syllabus_extraction/relative_dates_only.jsonbackend/tests/evals/cassettes/syllabus_extraction/spanish_syllabus_concrete_dates.jsonbackend/tests/evals/cassettes/syllabus_extraction/typical_syllabus_with_dates.jsonbackend/tests/evals/cassettes/syllabus_extraction/weekly_topics_no_deliverables.jsonbackend/tests/evals/cassettes/syllabus_extraction/weights_stated_twice.jsonbackend/tests/evals/run_all.pydocs/decisions/0020-agent-eval-harness-baselines.md
Comment on lines
+146
to
+158
| if update: | ||
| # Never persist baselines from a partially-broken run — that would bake | ||
| # a crash's degraded scores in as the new "expected". | ||
| if not ok: | ||
| print( | ||
| f"\nRefusing to update {name} baselines: run had case failures.", | ||
| file=sys.stderr, | ||
| ) | ||
| return False | ||
| all_baselines[name] = {ev: round(scores[ev], 6) for ev in scores} | ||
| _write_baselines(all_baselines) | ||
| print(f"\nUpdated baselines for {name} in {BASELINES_PATH.name}.") | ||
| return True |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Make combined baseline refreshes atomic. Per-dataset writes allow a failed run_all.py refresh to leave a partially updated committed baseline file.
backend/tests/evals/_replay.py#L146-L158: expose calculated baseline scores without immediately writing them when called by the combined runner.backend/tests/evals/run_all.py#L46-L53: collect all dataset scores and persist the full baseline map only after every dataset succeeds.
📍 Affects 2 files
backend/tests/evals/_replay.py#L146-L158(this comment)backend/tests/evals/run_all.py#L46-L53
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/tests/evals/_replay.py` around lines 146 - 158, Make baseline
refreshes atomic: in backend/tests/evals/_replay.py lines 146-158, update the
replay baseline-refresh flow to expose calculated scores without writing
immediately when invoked by the combined runner, while preserving standalone
persistence behavior; in backend/tests/evals/run_all.py lines 46-53, have the
combined runner collect every dataset’s scores and write the complete baseline
map only after all datasets succeed.
Uh oh!
There was an error while loading. Please reload this page.
Comment on lines
+2
to
+10
| "abstract": "This document provides a concise overview of photosynthesis, detailing the key stages of the light-dependent reactions, including Photosystems II and I, water splitting, and electron transport. It further explains the Calvin cycle, covering carbon fixation, reduction, and the regeneration of RuBP. The net products of photosynthesis, such as ATP, NADPH, and glucose, are identified, along with the primary limiting factors: light intensity, carbon dioxide concentration, and temperature.", | ||
| "headline": "Photosynthesis: An Overview of Light Reactions, Calvin Cycle, and Limiting Factors", | ||
| "key_points": [ | ||
| "Photosynthesis involves light-dependent reactions and the Calvin cycle.", | ||
| "The light reactions utilize water and light energy to produce ATP and NADPH.", | ||
| "The Calvin cycle uses ATP, NADPH, and CO2 to synthesize glucose.", | ||
| "Water splitting in the light reactions releases oxygen.", | ||
| "Key factors limiting photosynthesis include light, CO2, and temperature.", | ||
| "The process culminates in the production of glucose, ATP, and NADPH." |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Do not encode ATP/NADPH as overall final products.
The cassette says the Calvin cycle consumes ATP and NADPH on Line 7, but describes them as net/final products on Lines 2 and 10. This scientifically inaccurate gold output can reward incorrect agent responses.
Suggested correction
- "The process culminates in the production of glucose, ATP, and NADPH."+ "The light reactions produce ATP and NADPH, which the Calvin cycle uses to synthesize glucose."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/tests/evals/cassettes/document_summary/bullet_only_presentation.json`
around lines 2 - 10, Update the cassette’s abstract and key_points so ATP and
NADPH are described as inputs consumed by the Calvin cycle and products of the
light-dependent reactions, not as overall net or final products. Preserve
glucose as the relevant carbon product and retain the existing coverage of
photosynthesis stages and limiting factors.
| @@ -0,0 +1,12 @@ | |||
| { | |||
| "abstract": "This document outlines the procedure for an acid-base titration to determine the concentration of an unknown hydrochloric acid (HCl) solution. The process involves calibrating a burette, pipetting a precise volume of HCl into an Erlenmeyer flask, and adding phenolphthalein indicator. The HCl is then titrated with a standardized sodium hydroxide (NaOH) solution until a persistent pink color indicates the equivalence point. The volume of NaOH used is recorded, and the experiment is repeated for accuracy. ", | |||
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bashset -euo pipefail
echo"Repository files matching lab_procedure.json:"
fd -a 'lab_procedure\.json$'.||trueechoecho"Cassette excerpt:"
file="$(fd 'lab_procedure\.json$'.| head -n1 || true)"if [ -n"${file:-}" ];then
cat -n "$file"| sed -n '1,20p'fiechoecho"Search for abstract and phenolphthalein/equivalence references:"
rg -n '"abstract"|"phenolphthalein"|"equivalence point"|"endpoint"'. --glob '!**/.git/**'||trueRepository: SaplingLearn/Sapling
Length of output: 9627
🌐 Web query:
phenolphthalein persistent pink color endpoint equivalence point titration end point
💡 Result:
In a titration, the equivalence point and the endpoint are distinct concepts, though they are closely related [1][2]. The equivalence point is a theoretical, stoichiometric point at which the amount of titrant added is chemically equivalent to the amount of analyte present in the sample [1][2]. The endpoint is an experimentally observed point where a physical change, typically a color change of an indicator, signals that the titration should be stopped [1][3]. For phenolphthalein, the endpoint is defined as the appearance of a pale, persistent pink color [4][5][6]. Key considerations regarding the persistent pink endpoint include: 1. Persistence: The pink color should be faint and persist for at least 30 seconds while the solution is swirled [5][6]. This persistence ensures that the reaction between the titrant and analyte has reached completion and is not merely a transient, localized concentration effect [4][6]. 2. Definition and Precision: Because phenolphthalein changes color over a pH range (typically 8.2–10.0), the endpoint is an approximation of the equivalence point [7][8][9]. If the titration is carried out correctly, this approximation is sufficiently accurate for most analytical purposes [8]. 3. Overshooting: If the solution turns a deep or bright pink, or if the color persists for an excessively long time, it often indicates that the endpoint has been overshot, meaning an excess of titrant has been added beyond the equivalence point [4][5]. 4. Color Fading: In highly alkaline conditions (pH > 10), phenolphthalein can actually become colorless again as the pink dianion converts to a different, colorless form [10][11]. This process is slow, which is why a properly conducted titration aims for a stable, pale pink color rather than a dark or intense hue [10][11]. In practice, to achieve the most accurate results, chemists use a white background to detect the faintest possible pink tint and add the final drops of titrant slowly, ensuring the color change is observed exactly at the transition point [5][6].
Citations:
- 1: https://www.vedantu.com/jee-main/chemistry-difference-between-endpoint-and-equivalence-point
- 2: https://en.wikipedia.org/wiki/Equivalence_point
- 3: https://chemistry.stackexchange.com/questions/171091/what-does-indicator-colour-change-mean-at-equivalence-and-end-point-during-tit
- 4: https://www2.chem.wisc.edu/deptfiles/genchem/lab/labdocs/modules/titrate/titrqpink.htm
- 5: https://www.moorparkcollege.edu/sites/moorparkcollege/files/media/pdf_document/2023/chem_12_expt12.pdf
- 6: https://chem.libretexts.org/Courses/Los_Medanos_College/General_College_Chemistry_Lab_Manual_(Semester_2)/01%3A_Experiments/1.09%3A_New_Page
- 7: http://ibchemmilam.blogspot.com/2015/10/demonstration-comparing-endpoint-and.html
- 8: https://www.titrations.info/acid-base-titration-end-point-detection
- 9: https://www.sciencing.com/phenolphthalein-change-color-5271431/
- 10: https://www.flinnsci.com/globalassets/flinn-scientific/all-free-pdfs/dc91055.pdf
- 11: https://www.uv.es/qflab/2022_23/descargas/cuadernillos/qf1/english/p3_teaching_guide.pdf
Use “endpoint” rather than “equivalence point.”
A persistent phenolphthalein color is the titration endpoint; the equivalence point is the underlying stoichiometric condition.
Suggested correction
- "The HCl is then titrated with a standardized sodium hydroxide (NaOH) solution until a persistent pink color indicates the equivalence point."+ "The HCl is then titrated with a standardized sodium hydroxide (NaOH) solution until a persistent pink color marks the endpoint."📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| "abstract": "This document outlines the procedure for an acid-base titration to determine the concentration of an unknown hydrochloric acid (HCl) solution. The process involves calibrating a burette, pipetting a precise volume of HCl into an Erlenmeyer flask, and adding phenolphthalein indicator. The HCl is then titrated with a standardized sodium hydroxide (NaOH) solution until a persistent pink color indicates the equivalence point. The volume of NaOH used is recorded, and the experiment is repeated for accuracy. ", | |
| "abstract": "This document outlines the procedure for an acid-base titration to determine the concentration of an unknown hydrochloric acid (HCl) solution. The process involves calibrating a burette, pipetting a precise volume of HCl into an Erlenmeyer flask, and adding phenolphthalein indicator. The HCl is then titrated with a standardized sodium hydroxide (NaOH) solution until a persistent pink color marks the endpoint. The volume of NaOH used is recorded, and the experiment is repeated for accuracy. ", |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/tests/evals/cassettes/document_summary/lab_procedure.json` at line 2,
Update the abstract value in the lab procedure cassette to describe the
persistent pink phenolphthalein color as the titration endpoint instead of the
equivalence point; leave the remaining procedure description unchanged.
Comment on lines
4
to
+8
| "key_points": [ | ||
| "Course: PSY 240 Cognitive Psychology, Fall 2026", | ||
| "Grading: 20% quizzes, 30% midterm, 50% final paper", | ||
| "Meeting times: MWF 9-9:50am", | ||
| "Location: Room 214" | ||
| "Course Name: PSY 240 Cognitive Psychology", | ||
| "Meeting Time: MWF 9-9:50am", | ||
| "Location: Room 214", | ||
| "Grading Breakdown: 20% quizzes, 30% midterm, 50% final paper" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Keep key_points as complete sentences.
The summary contract in backend/agents/summary.py:19-34 describes each key point as one sentence, but all four values here are label fragments. Rewrite them as sentences so this cassette does not encode a nonconforming expected output.
Proposed fix
"key_points": [
- "Course Name: PSY 240 Cognitive Psychology",- "Meeting Time: MWF 9-9:50am",- "Location: Room 214",- "Grading Breakdown: 20% quizzes, 30% midterm, 50% final paper"+ "PSY 240 Cognitive Psychology is offered in Fall 2026.",+ "The class meets Monday, Wednesday, and Friday from 9:00 to 9:50 AM in Room 214.",+ "Weekly quizzes count for 20% of the final grade.",+ "The midterm counts for 30% and the final paper counts for 50%."
]📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| "key_points": [ | |
| "Course: PSY 240 Cognitive Psychology, Fall 2026", | |
| "Grading: 20% quizzes, 30% midterm, 50% final paper", | |
| "Meeting times: MWF 9-9:50am", | |
| "Location: Room 214" | |
| "Course Name: PSY 240 Cognitive Psychology", | |
| "Meeting Time: MWF 9-9:50am", | |
| "Location: Room 214", | |
| "Grading Breakdown: 20% quizzes, 30% midterm, 50% final paper" | |
| "key_points": [ | |
| "PSY 240 Cognitive Psychology is offered in Fall 2026.", | |
| "The class meets Monday, Wednesday, and Friday from 9:00 to 9:50 AM in Room 214.", | |
| "Weekly quizzes count for 20% of the final grade.", | |
| "The midterm counts for 30% and the final paper counts for 50%." |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/tests/evals/cassettes/document_summary/short_syllabus_excerpt.json`
around lines 4 - 8, Rewrite every string in the cassette’s key_points array as a
complete sentence while preserving the same course details. Ensure the four
entries remain semantically equivalent and conform to the sentence-based output
contract described by the summary implementation.
Comment on lines
+5
to
+9
| "correct_answer": "An object in motion stays in motion with the same speed and in the same direction unless acted upon by an unbalanced force.", | ||
| "difficulty": "easy", | ||
| "explanation": "Newton's First Law states that an object will remain at rest or in uniform motion in a straight line unless acted upon by an external force. This is also known as the law of inertia.", | ||
| "options": [ | ||
| "An object in motion stays in motion with the same speed and in the same direction unless acted upon by an unbalanced force.", |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Use the complete definition of Newton’s First Law.
The answer only describes objects already in motion, while the explanation correctly includes objects at rest. Replace both occurrences with the full “remains at rest or continues in uniform motion” statement.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@backend/tests/evals/cassettes/quiz_generation/easy_mcq_physics_definitions.json`
around lines 5 - 9, Update both occurrences of the Newton’s First Law answer
text in the quiz cassette—under “correct_answer” and the matching “options”
entry—to include objects remaining at rest as well as continuing in uniform
motion, using the complete definition consistent with the explanation.
Comment on lines
+4
to
+14
| "concept": "Cauchy Sequences", | ||
| "correct_answer": "A sequence is Cauchy if and only if it converges in a complete metric space.", | ||
| "difficulty": "hard", | ||
| "explanation": "In a complete metric space, the definition of Cauchy sequences and convergent sequences are equivalent. This means every Cauchy sequence converges to a limit within the space, and every convergent sequence is a Cauchy sequence. The completeness of the space is crucial for this equivalence.", | ||
| "options": [ | ||
| "A Cauchy sequence is convergent if and only if it is bounded.", | ||
| "A sequence is Cauchy if and only if it converges in a complete metric space.", | ||
| "A convergent sequence is always Cauchy, but the converse is only true in complete metric spaces.", | ||
| "A sequence converges if and only if it is a Cauchy sequence in any metric space." | ||
| ], | ||
| "question": "Which of the following statements best characterizes the relationship between Cauchy sequences and convergent sequences in a complete metric space?", |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Ensure every multiple-choice fixture has exactly one defensible answer.
backend/tests/evals/cassettes/quiz_generation/hard_mcq_real_analysis.json#L4-L14: replace the second valid Cauchy-sequence characterization with a clearly false distractor.backend/tests/evals/cassettes/quiz_generation/hard_mcq_theory_of_computation.json#L18-L28: remove or rewrite the true context-free-language option and update the incomplete membership explanation.
📍 Affects 2 files
backend/tests/evals/cassettes/quiz_generation/hard_mcq_real_analysis.json#L4-L14(this comment)backend/tests/evals/cassettes/quiz_generation/hard_mcq_theory_of_computation.json#L18-L28
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/tests/evals/cassettes/quiz_generation/hard_mcq_real_analysis.json`
around lines 4 - 14, The real-analysis fixture contains two defensible answers:
update the distractor in hard_mcq_real_analysis.json at lines 4-14 to be clearly
false while preserving the intended correct answer. In
hard_mcq_theory_of_computation.json at lines 18-28, remove or rewrite the true
context-free-language option and complete the membership explanation so every
fixture has exactly one defensible answer.
Comment on lines
+18
to
+28
| "concept": "Uniform Convergence", | ||
| "correct_answer": "f(x) must be continuous on E.", | ||
| "difficulty": "hard", | ||
| "explanation": "A fundamental theorem in real analysis states that the uniform limit of a sequence of continuous functions is continuous. Therefore, if {f_n(x)} converges uniformly to f(x) and each f_n is continuous, then f(x) must also be continuous on E.", | ||
| "options": [ | ||
| "f(x) must be continuous on E.", | ||
| "f(x) is not necessarily continuous on E.", | ||
| "f(x) is continuous on E if E is a compact set.", | ||
| "f(x) is continuous on E if the convergence is pointwise." | ||
| ], | ||
| "question": "Consider a sequence of functions {f_n(x)} converging uniformly to f(x) on a set E. If each f_n is continuous on E, and f is also continuous on E, what can be said about the limit function f(x)?", |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Remove the answer from the uniform-convergence stem.
The stem already states that f is continuous, so Line 19 merely repeats a premise instead of testing the uniform-limit theorem. Remove that clause from Line 28 so continuity is a consequence rather than an input.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/tests/evals/cassettes/quiz_generation/hard_mcq_real_analysis.json`
around lines 18 - 28, Update the question field in the uniform-convergence
cassette to remove the clause asserting that f is continuous on E. Keep the
remaining premises about uniform convergence and continuity of each f_n, so the
correct answer establishes continuity of f as the theorem’s conclusion.
Comment on lines
+18
to
+28
| "concept": "Price Elasticity of Demand", | ||
| "correct_answer": "2.0", | ||
| "difficulty": "medium", | ||
| "explanation": "Price elasticity of demand is calculated as the percentage change in quantity demanded divided by the percentage change in price. In this case, 20% / -10% = -2. The absolute value is 2.0, indicating elastic demand.", | ||
| "options": [ | ||
| "0.5", | ||
| "2.0", | ||
| "10%", | ||
| "20%" | ||
| ], | ||
| "question": "If a 10% decrease in price leads to a 20% increase in the quantity demanded, what is the price elasticity of demand?", |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bashset -euo pipefail
echo"== Locate cassette and related quiz generation files =="
git ls-files | rg '(^backend/tests/evals/cassettes/quiz_generation/medium_mcq_econ\.json$|quiz_generation|elasticity|Price Elasticity|Elasticity)'||trueechoecho"== Inspect the cassette =="if [ -f backend/tests/evals/cassettes/quiz_generation/medium_mcq_econ.json ];then
cat -n backend/tests/evals/cassettes/quiz_generation/medium_mcq_econ.json
fiechoecho"== Search for elasticity conventions/answers across repo =="
rg -n "Elasticity|elasticity|correct_answer|options".*["0-9]|[0-9]\.0|percentage change in quantity demanded|percentage change in price" backend/tests backend -g'!*cassettes*'2>/dev/null ||trueRepository: SaplingLearn/Sapling
Length of output: 50376
Specify the elasticity sign convention.
The explanation computes -2, but the correct option is 2.0; change the question to ask for the absolute value, or use -2.0 consistently in the answer/options.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/tests/evals/cassettes/quiz_generation/medium_mcq_econ.json` around
lines 18 - 28, Update the quiz item’s question, correct_answer, options, and
explanation to use one consistent price-elasticity sign convention: either
explicitly ask for the absolute value and retain 2.0, or consistently use -2.0
throughout. Keep all related fields aligned.
Comment on lines
+3
to
+4
| Offline accuracy harness for the migrated Pydantic AI agents (epic #152, issue | ||
| #148). It answers "did this prompt/model change make extraction worse?" without |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Keep the issue reference in the paragraph.
The wrapped #148 triggers markdownlint MD018. Move it onto the preceding line.
Proposed fix
-Offline accuracy harness for the migrated Pydantic AI agents (epic `#152`, issue-#148). It answers "did this prompt/model change make extraction worse?" without+Offline accuracy harness for the migrated Pydantic AI agents (epic `#152`, issue `#148`).+It answers "did this prompt/model change make extraction worse?" without📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| Offline accuracy harness for the migrated Pydantic AI agents (epic #152, issue | |
| #148). It answers "did this prompt/model change make extraction worse?" without | |
| Offline accuracy harness for the migrated Pydantic AI agents (epic `#152`, issue`#148`). | |
| It answers "did this prompt/model change make extraction worse?" without |
🧰 Tools
🪛 markdownlint-cli2 (0.23.1)
[warning] 4-4: No space after hash on atx style heading
(MD018, no-missing-space-atx)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/tests/evals/README.md` around lines 3 - 4, Adjust the introductory
paragraph in the README so the `#148` issue reference is moved to the preceding
line while remaining part of the paragraph, avoiding the markdownlint MD018
violation. Preserve the existing wording and both issue references.
Source: Linters/SAST tools
#454) services/rag_service.py's _embed_query/_embed_document/_embed_documents_batch and routes/documents.py::_index_document_chunks's catalog-relevance gate construct raw google.genai.Client objects directly, predating the #391 SAPLING_MODEL_MODE seam — so function mode (the hermetic E2E default) still fired live gemini-embedding-001 calls on every document upload, quiz generate, and tutor turn with a course_code, silently billing whenever a real key was present. Add agents/_providers.py::model_mode() as the one sanctioned public read of the seam for call sites outside agents/, and gate every embed call site on it. rag_service.py's client is now built lazily (_get_client()) and only ever reached after a model_mode() == "real" check; in non-real mode the _embed_* helpers raise before touching the client, which the existing broad try/except in retrieve_chunks/index_document_chunks already catches — reusing that path makes the deterministic empty/no-op result the designed behavior instead of an accident of a swallowed exception. Same pattern for the documents.py relevance-gate client, scoped tightly to that block. Interim mitigations (scripts/explore.sh, e2e.yml dummy-key forcing, the e2e_oracles logscan allowlist) are untouched — now defense in depth.
…) (#451) * fix(documents): resolve abstract course_id in /api/documents/user/{id} Library.tsx filters and labels documents on d.course_id, but the route only ever returned offering_id — every upload silently fell into "Uncategorized" and never matched a course filter. Resolve each row's course_id via services.academics.offering_course_id, batching per unique offering_id (mirrors routes/learn.py::list_sessions) rather than once per row. Adds backend coverage for the enriched response shape (single/batched/ missing offering_id) and extends the #387 upload journey with a library-filter assertion: after upload, filtering by the seeded course (resolved from the persisted row's offering_id) must show the document, and the "Uncategorized" filter must not. Fixes#435. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(documents): PR review round 1 — nullable type, honest tests, safe decrypt Address review findings on the #435 course_id fix before merge: - frontend/src/lib/types.ts: Document.course_id is string | null (the fix's own tests pin course_id: null as a real response shape). Library.tsx already treated it as possibly-falsy; tsc surfaced one spot assuming non-null (courseLookup[d.course_id]) — guarded with `?? ""`. - backend/tests/test_documents_routes.py: relabeled the offering_id=None test as defensive-code coverage (schema-unreachable — 0025 makes documents.offering_id NOT NULL) and added the actually-reachable null branch: a present offering_id that offering_course_id fails to resolve. - backend/routes/documents.py: wrap the concept_notes decrypt_json call in list_documents in try/except, matching the established pattern at _existing_doc_by_request_id and scan_document_concepts — decrypt_json re-raises when both decrypt and plaintext-parse fail, so one corrupted row no longer 500s the whole list. Added a regression test (red-first) proving the corrupted row degrades to concept_notes: [] while sibling rows still return. - frontend/e2e/upload.spec.ts: header now notes the #435 library-course-filter regression coverage the journey carries. Refs #435. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(library): dedupe course filter pills by course_id (#435) Stack verification of the #435 library-filter journey caught a real duplicate-render bug: GET /api/graph/{userId}/courses returns one row per enrollment, so a course with two offerings (e.g. CS101 fall + spring) surfaced as two rows sharing the same course_id. Library.tsx built its filter pills directly off that per-enrollment list, so the same course rendered two identical `library-course-filter-{courseId}` pills — a strict-mode Playwright locator violation, and a real UX bug (duplicate rows in the sidebar). Root cause is #449 (get_courses one-row-per-enrollment), which stays out of scope here — the gradebook depends on the per-enrollment shape. This is the frontend render-side fix: dedupe by course_id via a Map at the two derivation points that render per-course UI (the filter pills and the course label lookup), keeping the first enrollment's row as the stable representative. The raw per-enrollment `courses` list is otherwise untouched (course-scan lookup, upload-button disabled check, and the upload modal's course list still see every enrollment). The e2e library-filter assertion (fronten/e2e/upload.spec.ts, #435) was left as strict-mode (no .first() masking) per review — it should now pass because the pills are unique, not because the locator was weakened. Refs #435, #449. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ed' flake (#436, #354) (#453) * fix(agents): make the shared Gemini provider loop-safe, not a sweep (#436, #354) test_ocr_pipeline.py::test_save_to_db errored with "RuntimeError: Event loop is closed" on main — the #354 root cause: agents/_providers.py's module-level GoogleProvider eagerly builds an httpx.AsyncClient whose connection pool binds internal asyncio primitives to whichever event loop is running the first time a request goes out over it. Every agent is built once at import time sharing that one provider, so run_agent_sync's per-call asyncio.run() (and, just as much, any test calling asyncio.run() more than once against the same shared agent in one process, as test_agent_parse then the parsed_assignments fixture do here) trips the stale-loop reuse on every second real call. PR #358 (never merged; reviewed clean, just fell through the cracks) fixed this by sweeping eight run_agent_sync call sites to pass a fresh-provider model= override per call. Adapting it verbatim wouldn't have fixed#436: calendar_service's syllabus_extraction_agent, the actual caller behind this test, isn't on that sweep list — and agents/ocr_vision.py already needed its own ad hoc copy of the same idea for a path #358 didn't cover, proof the sweep needs rediscovering at every new call site. The issue itself named this "the tactical sweep" with "make the shared provider loop-safe" as the strategic fix. Since every agent gets its model from model_for()/google_model() exactly once, fixing those two functions fixes every caller — present and future — with no sweep required. _LoopSafeGoogleModel (a GoogleModel subclass) keeps one GoogleProvider per currently-running event loop (a WeakKeyDictionary keyed by the loop object, self-cleaning once a throwaway loop is GC'd), rebuilding only when a NEW loop calls it and reusing the cached one for as long as that loop lives — identical connection-pooling behavior to the old singleton for FastAPI's one persistent per-process loop, and no stale-loop reuse for run_agent_sync's or a test's disposable ones. This also makes ocr_vision.py's ad hoc fresh_ocr_vision_model() workaround redundant; removed it and its call-site override in gemini_vision_backend.py. Proof: tests/test_loop_safe_google_model.py (new, hermetic) pins the per-loop cache directly. tests/test_ocr_pipeline.py run 3x in one process (pytest.main() loop, since repeated identical CLI paths dedup to a single collection) put 6 asyncio.run() cycles through the same shared agent — 33/33 green, zero "Event loop is closed". Full suite green twice back-to-back (1161 passed, 26 skipped, 0 errors both times). ruff check clean. Fixes#436Fixes#354 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(agents): eliminate the loop-safe provider's shared-pointer race (#436, #354) PR review on the prior fix found a Critical concurrency bug, backed by an empirical repro: 6 threads x 20 sequential asyncio.run calls against one shared _LoopSafeGoogleModel, with an asyncio.sleep standing in for the real await gap, produced 95/120 mismatches between the provider bound for a call and the one actually read back. Root cause: _bind_to_current_loop() resolved the right provider under a lock, but handed it off via `self._provider = provider` — a single shared mutable attribute on a Model instance that is itself a process-wide singleton (every agent is built once at import time). GoogleModel._generate_ content reads self.client (-> self._provider.client) only AFTER an await (self._build_content_and_config(...)); in that window, a different thread running a different event loop — exactly what happens under concurrent requests, since every sync-def route drives run_agent_sync on a fresh thread + throwaway loop, and gemini_vision_backend._run_from_anywhere does the same for OCR — could rebind that same shared attribute. The first call would then resume and read a provider bound to someone else's, possibly already-closed, loop: a deterministic every-second-call flake became a probabilistic, load-dependent one. Fix: remove the hand-off entirely. self._provider (the base GoogleModel/ Model attribute) is now a fixed template, set once and never reassigned, backing only the reads that are genuinely loop-independent (system/base_url, and the bare .name/.base_url pydantic-ai's own count_tokens/usage-metadata code reads directly) — safe because every provider this module constructs uses identical arguments. .client — the one read that IS loop-affine — is now a property that resolves asyncio.get_running_loop() -> the loop-keyed WeakKeyDictionary fresh, at the exact moment of every access, with no instance-attribute write in between "resolve" and "use". Every inherited GoogleModel method reads self.client through ordinary attribute lookup, so this one override covers all of them — request/count_tokens/request_stream no longer need (and no longer have) their own overrides. __aenter__/ __aexit__ still act on the current loop's provider explicitly. Verified standalone: the buggy hand-off shape reproduces 119/120 mismatches; the fixed property-based design shows 0/120, both under the same 6x20 stress. Added TestConcurrentAccessIsRaceFree to tests/test_loop_safe_google_model.py: a minimal stand-in of the original hand-off design proves the test shape itself would have caught the round-0 bug (asserts it DOES mismatch), then the same shape run against the real _LoopSafeGoogleModel asserts zero mismatches. Deterministic and hermetic — no network. agents/ocr_vision.py and gemini_vision_backend.py needed no further changes: they already call through model_for("ocr_vision")'s default model, so the OCR per-page-loop concurrency concern the review raised falls out of this fix directly. Re-verified: threaded test suite 5x back-to-back (9/9 every time), the OCR pipeline module 3x in one process (pytest.main() loop, 33/33 green), full hermetic suite once (1164 passed, 26 skipped, 0 errors). ruff check clean. Fixes#436Fixes#354 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* ci(e2e): pin local/CI Postgres to 15, matching staging/prod (#441) supabase/config.toml's major_version pins the local/CI Postgres (via supabase start) independently of hosted staging/prod; PR #440 set it to 17 for local-dev/CI consistency, which drifted the deterministic lane away from what production actually runs. Pin it down to 15 instead — hosted staging/prod stay untouched (bumping them is a separate, outward-facing ops action), and local/CI consistency is preserved since both still follow the same config.toml pin, just at 15. - supabase/config.toml: major_version 17 -> 15 (also the Supabase CLI's own documented default for this key). - .github/workflows/e2e.yml: update the header comment that previously explained "deliberately going against" the PG15 leaning. - backend/tests/integration/test_postgres_version.py: assert the real server's major version via SHOW server_version_num, so drift is loud. Lives in the opt-in integration suite because .github/workflows/integration.yml boots the local stack from empty and runs `RUN_INTEGRATION=1 pytest -m integration` on every push to main — this actually executes in CI, against the same config.toml-pinned Postgres the browser lane (e2e.yml) also boots. - docs/local-supabase.md: update the "Postgres version" troubleshooting entry and add a reset note for stacks provisioned before this pin (the CLI does not swap a running container's image just because config.toml changed). Migration chain audited for PG16+-only constructs (MERGE, JSON_TABLE, REGEXP_*, EXCLUDE, etc.) — none found; UNIQUE NULLS NOT DISTINCT (0023_graph_integrity.sql) is itself a PG15 feature, so it's fine on the target version. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(e2e): fix#441 PR-review findings — history + reset guidance Two documentation-accuracy fixes from PR review, no behavior change: 1. backend/tests/integration/test_postgres_version.py docstring misattributed the PG17 pin's origin to PR #440. Git history shows the pin was born with the local Supabase stack itself (9f54739, config.toml created with major_version = 17) — #440 only added the e2e.yml comment rationalizing the already-existing PG17 against epic #402 decision 2's PG15 leaning, and noted the skew was tracked in #441. Rewrote the causal narrative to match; updated the assert failure message's reset guidance to match fix 2 below. 2. docs/local-supabase.md's "Postgres version" troubleshooting entry was self-contradictory: it said the CLI "does not swap a running container's image just because config.toml changed," then claimed scripts/local-db-reset.sh (supabase db reset) "recreates the Postgres container against the pinned version." Reading local-db-reset.sh: it never calls supabase stop/start, so it can't replace a running container's major version — confirmed against supabase/cli issues #5555 and #4522 (db reset does not reliably pick up a changed major_version and can leave a half-upgraded, broken container). Restructured so the reliable path is primary for a version change: `supabase stop --no-backup` + `supabase start` (fresh container, correct major, no app schema yet), then local-db-reset.sh / make e2e-up for their actual job — migrate + seed. Static-only per the controller's instructions (no stack boot); hermetic backend suite re-run clean (1155 passed, same pre-existing unrelated test_ocr_pipeline.py event-loop flake, #354). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…#456) * docs(e2e): refresh known-bugs catalogs after the #402 follow-up batch Fixed and closed: #355, #430, #435, #436 (+root cause #354), #439, #446 (merged via #447/#448/#450/#451/#453/#454). #441's fix (PR #452, PG15 pin) is in final verification, merging imminently. Known-open remaining: #449 (get_courses per-enrollment fan-out produces duplicate course_id rows; Library's instance fixed render-side in #451, Tree/Dashboard/etc. and the DocumentUploadModal picker still exposed). Updates docs/e2e-exploration.md (§6 logscan allowlist note, §7 triage category 3 + worked examples, §8 fixme lifecycle example) and scripts/explore/explorer-prompt.md's known-bugs section to match. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(e2e): #441 landed — move it into the recently-fixed list Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…conventions (#457) Every agent session on this repo now learns the E2E system up front: the deterministic stack commands (e2e-up/down, Playwright lane, oracles, explore), the pre-merge three-way verification expectation, the fix→promoted-journey pairing, the machine-singleton flock protocol, and the function-mode seam rules (fixed constants, handler registration, no raw genai clients below the seam). Follows the #402/#403 epics and the 2026-07-28 bug-queue batch. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ollow-up) (#358) * fix(agents): guard run_agent_sync against running event loops (#354 follow-up) Reworked from the original fresh-client sweep: the cross-loop client problem is now solved by _LoopSafeGoogleModel (#453), so the per-call fresh-client plumbing and the subject_root dedup (landed separately, #355) are dropped. What remains is the piece main still lacks: - run_agent_sync detects a running event loop, closes the handed coroutine (no 'never awaited' warning) and raises a clear error the try/except-guarded sync-from-async callers can degrade on, instead of letting asyncio.run raise opaquely. - HEALTH_PROBE_MODEL constant so probe sites can't drift. - Regression tests for both loop paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(agents): name the real async->sync call chain in the loop-guard rationale Review found the cited example chain (build_system_prompt -> get_course_context) never reaches run_agent_sync; the reachable chain is _legacy_chat -> apply_graph_update -> update_course_context -> _generate_summary_with_gemini -> run_agent_sync. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
#74) (#349) * feat(learn): stream tutor replies over SSE with live graph deltas (#70, #74) — rebased onto main Net rework of feat/streaming-tutor against current main: - Ported: services/chat_stream.py (stream_agent_turn rung ladder), agent_events.py chat-event vocabulary, /chat/stream + /start-session/stream SSE routes, frontend sse.ts + api.ts stream consumers, Learn.tsx/ChatPanel wiring + tests, ADR as 0020. - Dropped the fresh-client plumbing (_fresh_stream_model, fresh_google_model, per-call model= overrides): superseded by _LoopSafeGoogleModel (#453). The streamed routes now inherit the agent's own model so the SAPLING_MODEL_MODE seam applies. - NEW: function-mode seam serves streamed runs — _function_model_for gains a stream_function that replays the registered handler's ModelResponse as deltas (text + DeltaToolCall), keeping E2E_* constants byte-identical across JSON and SSE lanes; covered by two new seam tests. - Learn.tsx edgeKey NUL byte rewritten as \u0000 escape (text-clean). - tutor-stop testid added (e2e-surface lint #382) + docs entry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(learn): harden stream persistence + concurrent-stream guard (review findings) - stream_agent_turn: on_complete failures after a fully-streamed reply now yield the structured ADR-0020 error event instead of aborting the SSE response uncaught (headers are already flushed at that point). Regression test added. - Learn.tsx: abort any in-flight stream controller before starting a new one — a graph-node click could begin a session while a reply streamed, interleaving two streams into shared state. - Docstrings: the on_complete/legacy_fallback invariant is 'at most one, never both' (error rungs run neither), not 'exactly one'; PENDING_SESSIONS wording updated to match. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…o feat/148-agent-eval-harness-rework
…evaluator must not PASS CI Review finding: the regression gate iterated only committed baseline keys, so a dataset absent from baselines.json (or an evaluator added without refreshing it) reported PASS with zero protection — right as evals.yml becomes a PR gate. Both paths now FAIL with a pointed message; verified empirically (removed dataset + evaluator entries -> exit 1, restored -> exit 0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
11 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Completes the extraction-accuracy eval harness for the migrated Pydantic AI agents. Closes #148 (part of the agent-migration epic #152). This is the evidence base the final
gemini_servicecutover (#151) needs — it turns "did this prompt/model change make extraction worse?" from a smoke test into a measured, gated check.The harness previously existed only as a non-functional scaffold: 4 of ~95 cases had cassettes,
quiz_generation/chat_tutorhad none, the runner failed on any evaluator score< 1.0(wrong for an accuracy harness), and the CI workflow wasworkflow_dispatch-only.Changes
document_classification,document_summary,concept_extraction,syllabus_extraction,quiz_generation). Replay is now deterministic and needs no key.baselines.jsonor a cassette is missing — not on< 1.0.SAPLING_EVAL_UPDATE_BASELINES=1refreshes baselines.richon a Windows cp1252 console.tests/evals/run_all.pyruns every offline dataset and prints a per-task accuracy summary;evals.ymlruns it in replay mode on every PR touchingbackend/agents/**or the harness.backend/tests/evals/README.md(record/refresh workflow), README update, and ADR 0020 (design + baseline numbers).chat_tutorexcluded from the offline harness — its retrieval tool reads a live Supabase and can't run against cassettes; folded into the graph-grounded tutor work ([P1] Agent migration: graph-grounded tutor retrieval + tool-use loop #149).Baselines (2026-07-28, current per-task models)
These are a floor, not a target. Re-record and re-baseline when a prompt/model changes.
Testing
python tests/evals/run_all.py→ 5/5 PASS (replay, gate on baselines).ruff checkclean on new files.Follow-ups (not in scope)
chat_tutorinto the harness once it has an offline retrieval seam.recordpass is still how live-model regressions surface (replay validates against a snapshot). Called out in ADR 0020.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation
Tests