Uh oh!
There was an error while loading. Please reload this page.
feat(quiz): quiz-context + course-summary → agents, retire quiz-gen fallback (#145) - #299
Conversation
…uiz-gen fallback (#145) Removes the last raw call_gemini* seams from the quiz surface: - New agents/quiz_context.py (QuizContext: weak_areas, common_mistakes, questions_seen_summary, recommended_difficulty, notes — mirrors quiz_context_update.txt). routes/quiz.py's post-submit background task runs it via run_agent_sync and saves .model_dump() (unchanged context_json shape). - New agents/course_summary.py (CourseSummary.summary). course_context_service's _generate_summary_with_gemini runs it via run_agent_sync and keeps the deterministic template fallback on agent failure (no second LLM call). - Removed _legacy_generate_quiz + the MODEL_LITE/MODEL_SMART/call_gemini_json import. On agent guardrail-trip or failure, generate_quiz now returns 502 instead of a raw-Gemini fallback; the pre-agent 404 (unknown node) is unchanged. _providers registers course_summary (flash) + quiz_context (lite). Coordination: does NOT touch scoring / apply_graph_update / quiz_attempts writes (#128/#129 territory) — only the LLM seams. Tests: rewrote the legacy-fallback quiz tests to degrade-to-502; repointed the submit context-update patches onto quiz_context_agent; removed the obsolete legacy prompt-augmentation + legacy-model tests; added quiz-context save, course-summary success/fallback tests. Full suite 826 passed (2 pre-existing storage-env failures). ruff clean. Spec: specs/145-quiz-course-context-agents.md Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Warning Review limit reached
Next review available in:31 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughThis PR migrates quiz generation and course-context summary regeneration from raw Gemini calls to typed Pydantic AI agents. It adds ChangesAgent migration
Estimated code review effort: 3 (Moderate) | ~30 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant QuizRoute as routes/quiz.py
participant QuizContextAgent as quiz_context_agent
participant CourseContextService as course_context_service.py
participant CourseSummaryAgent as course_summary_agent
Client->>QuizRoute: POST /api/quiz/generate
QuizRoute->>QuizRoute: run quiz agent
alt agent fails (UsageLimitExceeded / unexpected error)
QuizRoute-->>Client: HTTP 502
else success
QuizRoute-->>Client: quiz payload
end
Client->>QuizRoute: submit quiz answers
QuizRoute->>QuizContextAgent: run_agent_sync(quiz_context_agent.run(prompt))
QuizContextAgent-->>QuizRoute: QuizContext output
QuizRoute->>QuizRoute: save_quiz_context(result.output.model_dump())
CourseContextService->>CourseSummaryAgent: run_agent_sync(course_summary_agent.run(user_message))
alt agent succeeds
CourseSummaryAgent-->>CourseContextService: CourseSummary
else agent fails
CourseContextService->>CourseContextService: deterministic fallback summary
end
Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Deploying with |
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs | frontend-staging | e8127ae | Commit Preview URL Branch Preview URL | Jul 02 2026, 06:11 AM |
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (3)
backend/routes/quiz.py (1)
202-207: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUpdate the stale legacy-fallback comment.
Lines 203-205 still say this routes to the legacy fallback, but the new catch block returns HTTP 502. Please align the comment with the removed fallback path.
Suggested comment update
- # All questions dropped — degrade to legacy rather than serve- # an empty quiz. Raise a sentinel that generate_quiz catches- # and routes to the legacy fallback.+ # All questions dropped — do not serve an empty quiz. Raise a+ # sentinel that generate_quiz catches and degrades to HTTP 502.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@backend/routes/quiz.py` around lines 202 - 207, The comment in the quiz generation path is stale because the `RuntimeError` from `quiz_agent` no longer routes to a legacy fallback and is now handled by the `generate_quiz` catch block as an HTTP 502. Update the inline comment near the `wire_questions` validation in `generate_quiz`/the surrounding fallback logic to describe the current behavior only, removing any mention of legacy fallback and keeping the wording aligned with the new error handling.backend/services/course_context_service.py (2)
53-54: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winNo guard against a falsy/empty agent summary.
If the agent returns an empty or whitespace-only
summary, no exception is raised, so the deterministic fallback never kicks in and a blank summary gets persisted tooffering_summary.summary_text.🛡️ Suggested fix
try: result = run_agent_sync(course_summary_agent.run(user_message)) - return result.output.summary+ summary = result.output.summary+ if not summary or not summary.strip():+ raise ValueError("course_summary_agent returned empty summary")+ return summary except Exception:🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@backend/services/course_context_service.py` around lines 53 - 54, The course summary flow in course_context_service’s summary generation path currently returns result.output.summary directly, which allows empty or whitespace-only values to pass through. Update the logic around run_agent_sync(course_summary_agent.run(user_message)) to validate result.output.summary and raise/trigger the existing fallback when it is falsy or blank, so the deterministic summary is used instead of persisting an empty offering summary.
52-61: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winLog the agent failure before falling back
Add a warning log withexc_info=Truebefore returning the template so agent/runtime failures are visible instead of disappearing into the fallback path.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@backend/services/course_context_service.py` around lines 52 - 61, The fallback in course_context_service’s summary generation swallows agent/runtime failures silently. Update the try/except around run_agent_sync(course_summary_agent.run(user_message)) so the exception is logged as a warning with exc_info=True before returning the template summary, keeping the fallback behavior but making failures visible; use the surrounding course_context_service method and course_summary_agent.run as the key locations.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/tests/test_quiz_routes.py`:
- Around line 23-29: The submit-route tests are only neutralizing
`quiz_context_agent.run`, but `_update_context` in `routes.quiz` still persists
context via `save_quiz_context(...)`, which can trigger hidden writes or masked
failures. Update the affected tests/helpers around `_noop_ctx_agent()` to also
patch `routes.quiz.save_quiz_context` to a no-op wherever the submit flow is
exercised, so the background context update is fully disabled.
---
Nitpick comments:
In `@backend/routes/quiz.py`:
- Around line 202-207: The comment in the quiz generation path is stale because
the `RuntimeError` from `quiz_agent` no longer routes to a legacy fallback and
is now handled by the `generate_quiz` catch block as an HTTP 502. Update the
inline comment near the `wire_questions` validation in `generate_quiz`/the
surrounding fallback logic to describe the current behavior only, removing any
mention of legacy fallback and keeping the wording aligned with the new error
handling.
In `@backend/services/course_context_service.py`:
- Around line 53-54: The course summary flow in course_context_service’s summary
generation path currently returns result.output.summary directly, which allows
empty or whitespace-only values to pass through. Update the logic around
run_agent_sync(course_summary_agent.run(user_message)) to validate
result.output.summary and raise/trigger the existing fallback when it is falsy
or blank, so the deterministic summary is used instead of persisting an empty
offering summary.
- Around line 52-61: The fallback in course_context_service’s summary generation
swallows agent/runtime failures silently. Update the try/except around
run_agent_sync(course_summary_agent.run(user_message)) so the exception is
logged as a warning with exc_info=True before returning the template summary,
keeping the fallback behavior but making failures visible; use the surrounding
course_context_service method and course_summary_agent.run as the key locations.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: c7510e3b-9aed-40df-a904-a8939c75af36
📒 Files selected for processing (8)
backend/agents/_providers.pybackend/agents/course_summary.pybackend/agents/quiz_context.pybackend/routes/quiz.pybackend/services/course_context_service.pybackend/tests/test_quiz_routes.pybackend/tests/test_shared_course_context.pyspecs/145-quiz-course-context-agents.md
Uh oh!
There was an error while loading. Please reload this page.
…_context in submit mocks Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The all-questions-drift branch's comment still described routing to the legacy fallback, which #145 deleted. The RuntimeError is now caught by generate_quiz's bare-except and returned as HTTP 502 (per test_degrades_when_all_questions_drift). Comment-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes#145 (Agent-migration epic #152, milestone #2).
What
Removes the last three raw
call_gemini*seams from the quiz surface, folding them into the agent layer:quiz.pypost-submit background taskcall_gemini_json(quiz-context regen)agents/quiz_context.py(quiz_context_agent)course_context_service.pycall_gemini(class summary)agents/course_summary.py(course_summary_agent)quiz.py_legacy_generate_quizcall_gemini_jsonfallbackNotes
QuizContextmirrorsquiz_context_update.txtexactly (weak_areas,common_mistakes,questions_seen_summary,recommended_difficulty,notes);_update_contextsaves.model_dump(), so the storedquiz_context.context_jsonshape is unchanged.generate_quizreturns 502 rather than serving a quiz from a second LLM path. The pre-agent 404 (unknown concept node) is unchanged, and the fast/smartmodel_prefoverride still works on the agent path.run_agent_syncbridge (from [P2] Agent migration: remaining one-shot LLM calls (study guide, social, health) → agents/seam #147).This PR is scoped to the LLM seams only. It does not touch quiz scoring,
apply_graph_update, or thequiz_attemptswrites insubmit_quiz— the #128/#129 territory. The only change insidesubmit_quizis swapping the background context-update's LLM call for the agent.Testing
quiz_context_agent; removed the obsolete legacy prompt-augmentation class + legacy-model tests; added quiz-context-save, course-summary success/fallback, and degrade tests.test_storage_servicefailures pre-exist onmain— missing SUPABASE env).ruffclean.Out of scope: flashcards (#146), deleting
services/gemini_service.py(#151).🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes