Uh oh!
There was an error while loading. Please reload this page.
feat(quiz): adaptive iteration — spaced repetition + history + difficulty - #77
Merged
Merged
Conversation
…ifficulty Closes the three "doesn't do (yet)" gaps from ADR 0013 with one new tool and a prompt update. No wire-format change, no fallback-contract change. - agents/tools/quiz_history.py — read_recent_quiz_attempts surfaces the per-(user, concept) digest from quiz_context plus the last 5 completed attempts (newest first, accuracy precomputed). DB failures degrade silently to empty history. - agents/quiz.py — registers the new tool, expands prompt with spaced-repetition rules (weight last_reviewed_at on graph_nodes) and adaptive-difficulty rules (modulate by recent_attempts.accuracy with a one-step bound). Prompt hash bumps to 358613666dbc. - routes/quiz.py — _quiz_via_agent user message nudges the agent to call the new tool with the target concept_node_id. - docs/decisions/0014-adaptive-quiz-iteration.md — captures the decision, what we deliberately didn't do, and the rollback path. - Tests: new test_quiz_history_tool.py pins shape coercion, the completed_at IS NOT NULL filter, and silent-degrade contract; test_quiz_agent_imports.py asserts the new tool is registered. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Warning Rate limit exceeded
To keep reviews running without waiting, you can enable usage-based add-on for your organization. This allows additional reviews beyond the hourly cap. Account admins can enable it under billing. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
📝 WalkthroughWalkthroughA new tool ChangesAdaptive Quiz Difficulty via Recent Attempts
Sequence DiagramsequenceDiagram
participant Router as Quiz Route
participant Agent as Quiz Agent<br/>(Gemini)
participant HistoryTool as History Tool
participant DB as Supabase<br/>(quiz_context,<br/>quiz_attempts)
Router->>Agent: user_message: find weakest,<br/>stalest concepts & call<br/>read_recent_quiz_attempts
Agent->>Agent: Call read_concepts_for_user<br/>(concept selection)
loop For each selected concept
Agent->>HistoryTool: read_recent_quiz_attempts<br/>(concept_node_id)
HistoryTool->>DB: Fetch quiz_context<br/>summary (async)
HistoryTool->>DB: Fetch quiz_attempts<br/>(completed_at ≠ null,<br/>limit 5, order recent)
DB-->>HistoryTool: context + attempts
HistoryTool->>HistoryTool: Coerce summary<br/>(handle legacy shapes)
HistoryTool->>HistoryTool: Parse & validate<br/>score/total, compute<br/>accuracy ∈ [0,1]
HistoryTool-->>Agent: QuizHistory{summary,<br/>recent_attempts}
end
Agent->>Agent: Generate questions<br/>with difficulty adapted<br/>by recent_attempts.accuracy
Agent-->>Router: Quiz with adaptive<br/>difficulty
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Deploying with |
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs | frontend | e2112a8 | Commit Preview URL Branch Preview URL | May 05 2026, 02:40 AM |
…nits Closes the four issues flagged on PR #77's self-review. - tests/evals/quiz_generation.py — adds 2 cases (count: 8 -> 10) and 2 evaluators that pin the new prompt rules structurally: - AdaptiveDifficultyEvaluator: requested vs. produced difficulty rank stays within ±1 step. Permits the prompt's allowed adaptive shift, flags overshoots. - SpacedRepetitionConceptEvaluator: when metadata names a `stale_concept`, at least one question must target it. Smoke-tested against synthetic Quiz outputs: bounds in → 1.0, overshoots → 0.0. - agents/tools/quiz_history.py — drop rows with score outside [0, total] entirely (with a logger.warning) instead of clamping accuracy and passing impossible numbers (score=7, total=5) to the LLM. Matching test_corrupt_rows_are_dropped replaces the old clamp-asserting test. - agents/tools/quiz_history.py — read_recent_quiz_attempts_tool docstring is now LLM-facing ("returns this student's history…") instead of engineering-facing. Pydantic AI surfaces this as the tool's description to the model. - routes/quiz.py — _quiz_via_agent user message trimmed to routing-only; the workflow + adaptive rules already live in the system prompt and don't need to be restated per request. - docs/decisions/0014-adaptive-quiz-iteration.md — Date corrected (2026-05-03 → 2026-05-04). Eval-coverage section updated to describe the two new structural sentinels. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the two issues from PR #77's second review. - AdaptiveDifficultyEvaluator now scores per-question (fraction compliant) instead of average rank. The prompt rule is per-question ("Never override the user-requested difficulty by more than one step"); the previous avg-based check let a 2-step outlier slip through if the rest of the mix balanced it out — e.g. requested hard with [easy, hard, hard, hard] used to score 1.0 (avg=1.5, within 1 of target=2) and now correctly scores 0.75. Switched to subscript _DIFF_RANK[q.difficulty] (Literal-constrained, fallback was unreachable). - The two new ADR-0014 cases now carry NOTE comments explaining that recency/staleness state is baked into the user message for replay determinism, while production sources it via read_recent_quiz_attempts. The cases pin the prompt's rule application; live-mode evals are the right place to catch tool-wiring regressions. Smoke-tested: hard + [easy, hard, hard, hard] -> 0.75; hard + all medium -> 1.0 (allowed shift); hard + all easy -> 0.0 (overshoot). 46 quiz unit tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The section comment above adaptive_downshift_struggling_student described the old average-based scoring; commit 6493988 rewrote the evaluator to per-question fraction but missed this comment. Bring the prose in line with the code so future readers don't trust an outdated description. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
submit_quiz writes score+total atomically (routes/quiz.py:426), so any
row with completed_at IS NOT NULL but a null score or null total is
corruption. The previous coercion (`r.get('score') or 0`) silently
turned that into a 0/5 = 0% accuracy reading, which the LLM could
trust and use to trigger a spurious adaptive downshift on a perfectly
healthy concept.
Drop those rows alongside the existing score-out-of-bounds drop
branch (with the same logger.warning treatment) and tighten the test
to cover null score, null total, and total=0 in one sweep.
Surfaced by an independent code-reviewer pass on PR #77.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>Uh oh!
There was an error while loading. Please reload this page.
This was referenced May 5, 2026
This was referenced Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the three "doesn't do (yet)" gaps from ADR 0013 with one new tool and a prompt update on
quiz_agent. No wire-format change, no fallback-contract change, no new agent.graph_nodes.last_studied_at. Stale (~7d+) and unreviewed (null) concepts surface even when their mastery is mid-tier.read_recent_quiz_attempts(concept_node_id)tool exposes the per-(user, concept) digest fromquiz_contextplus the last 5 completedquiz_attemptsrows (newest first, accuracy precomputed). The legacy fallback already read this; the agent path now does too.Prompt hash bumps from
17ab80b30316→358613666dbc.Files
backend/agents/tools/quiz_history.py— new pure-async function +_toolwrapper.completed_at IS NOT NULLfilter excludes the in-flight row thatgenerate_quizwrites pre-submission. DB failures degrade silently to empty history.backend/agents/quiz.py— registers the new tool; system prompt adds explicit spaced-repetition + adaptive-difficulty rule blocks.backend/routes/quiz.py—_quiz_via_agentuser message nudges the agent to call the new tool with the target concept_node_id.docs/decisions/0014-adaptive-quiz-iteration.md— captures the decision, deliberate non-goals (no decay-formula spaced rep, no cross-concept history yet, no tool consolidation), and a single-revert rollback path.backend/tests/test_quiz_history_tool.py(+9 tests) — pins shape coercion (_coerce_summaryhandles legacy string,{summary: ...}, and{misconceptions, weak_areas}shapes), accuracy math + clamping, the filter wiring (completed_at=not.is.null,order=completed_at.desc,limit=5), and the silent-degrade-on-DB-error contract.backend/tests/test_quiz_agent_imports.py— extended to assert the new tool is registered.Why this isn't refactor #4
ADR 0005 carved out "adaptive quiz history" as a future iteration on the same agent, not a separate refactor. This ships under that carve-out: no new agent, no new route, no wire-format change, no fallback-contract change.
Test plan
pytest tests/test_quiz_history_tool.py tests/test_quiz_agent_imports.py tests/test_quiz_routes.py tests/test_graph_read_tools.py -q→ 51 passed_PROMPT_HASHround-trips and is logged on every quiz run via the existing Logfire instrumentationSAPLING_EVAL_MODE=live pytest tests/evals/quiz_generation.py -q) — the prompt-driven adaptive behaviors are LLM-decided; unit tests can only pin the tool's I/O contract. Recommended before merge.summaryinquiz_context.Rollback
Single-revert clean: revert this commit and the tool import + registration disappear, the prompt reverts to
17ab80b30316, and the route's user message reverts to refactor-#2 wording. The newquiz_history.pyfile is pure-leaf (no other module imports it), so it's harmless after rollback.🤖 Generated with Claude Code
Summary by CodeRabbit
Release Notes
New Features
Documentation