Uh oh!
There was an error while loading. Please reload this page.
feat(agents): prompt-injection hardening on student content (#150) - #471
Conversation
Third link of the seam endgame; gates the public beta. - One shared containment helper (services/prompt_safety.py): wrap_untrusted builds a delimited BEGIN/END UNTRUSTED CONTENT envelope with data-not-instructions framing; neutralize_delimiters defangs embedded marker forgeries (case/whitespace-insensitive, idempotent) so content can't fake an early END and escape. Applied at ASSEMBLY boundaries only — storage keeps raw text: RAG chunks (format_rag_context — covers agent chat, legacy chat, quiz context in one place), the graph seed block's student-derived concept names, the legacy prompt's COURSE MATERIALS + shared-context JSON, and the tool→ LLM boundary (search_course_materials, read_active_note, read_misconceptions, quiz-history; note-worker user prompts). - INJECTION_GUARD_PROMPT (single source) in the tutor's three preambles, note_chat, quiz, and the legacy preamble; the ACADEMIC INTEGRITY block deferred out of #149 lands in the agent preamble for parity. - Tool-use constraint: ConceptMasteryUpdate.mastery_delta schema clamped ±1.0 → the instructed [-0.1, +0.3] band — an injected 'set my mastery to 1.0' now fails validation into the #153 retry loop; plus a contract test freezing that no tutor/note tool signature exposes user_id/course_id/session_id/note_id to the model. - Documented scope calls: the student's own message channel and session history stay unwrapped (their instruction channel; wrapping the tool but not message_history would be theater); catalog chunks stay trusted (script-ingested official data); misconceptions tool stays unregistered on the tutor (consent enforcement still deferred — the data that DOES reach prompts is now contained); document-pipeline workers untouched (no tools to coerce, ~70 cassettes at stake) — noted as follow-up. - Evals: chat_tutor (16) + quiz_generation (10) honestly re-recorded (their prompts changed; replay keys on case names and would have stayed silently green). The re-record also landed ADR-0023's tracked prompt-shape fix (never end the turn on a tool call) after the quirk reproduced live. Scores: all evaluators 1.000 on both datasets (GroundedConcept ratcheted 0.875 → 1.0); other four datasets untouched. 27-case red-first injection suite (14 red at base) incl. an end-to-end FunctionModel test proving an injected tool return reaches the model enveloped. Gates: backend 1518 passed + ruff clean; lockvenv 278 passed; evals replay green ×6. Closes#150. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Deploying with |
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs | frontend-staging | c09d245 | Commit Preview URL Branch Preview URL | Jul 30 2026, 12:16 PM |
Warning Review limit reached
Next review available in:3 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (46)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This pull request has been ignored for the connected project Preview Branches by Supabase. |
…ero-width forgeries stripped, docs squared - read_session_history_tool neutralizes replayed content (a student could plant a literal envelope delimiter in one turn and have it handed back as a REAL byte-match later); read_concepts_for_user_tool and read_graph_neighborhood_tool neutralize student-derived concept names (the same data the seed block already defangs). Three red-style tests. - neutralize_delimiters strips zero-width/invisible Unicode before matching — a visually identical forged delimiter threaded with U+200B no longer dodges the regex (empirically demonstrated in review). - Docs squared: ADR 0023's follow-up note marked shipped (via #470+#471), prompt_safety's 'every agent' overclaim corrected (note workers carry their own one-line guards, deliberately), graph_context's budget docstring notes the envelope overhead. - FYI-class same-PR fix: {last_session_summary} (LLM-generated text of student content) now wrapped like the quiz digest. Backend 1521 + ruff green; lockvenv 78 green; evals replay green ×6. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AndresL230
commented
Jul 30, 2026
Review pass complete: findings all fixed — the convergent one (sibling tools on the same agent returning raw content: session history could replay a student-planted REAL forged delimiter; both graph readers returned raw concept names) now neutralized with tests; zero-width delimiter forgeries stripped before matching; three doc staleness items squared; and the reviewers' FYI (last_session_summary unwrapped) folded in since it's the same class. send_to_tutor's preface deliberately stays unwrapped (the student's own instruction channel, consistent with the PR's documented scope call). Backend 1521 + lockvenv + evals ×6 green. e2e cycle next. |
| def test_session_history_tool_neutralizes_replayed_delimiters(): | ||
| """A student can plant a literal envelope delimiter in one turn; the | ||
| history tool must not replay it as a REAL byte-match later.""" | ||
| import asyncio |
| def test_graph_read_tools_neutralize_concept_names(): | ||
| """The sibling read surfaces return the SAME student-derived concept | ||
| names the seed block neutralizes — they must be defanged here too.""" | ||
| import asyncio |
AndresL230
commented
Jul 30, 2026
Pre-merge gate: full lane 28/28 passed + oracles clean. Merging. |
Uh oh!
There was an error while loading. Please reload this page.
Every CI run since the #471 merge fails at the lint step with exit 2: the agent-only rung ladder cutover removed three suppressed no-restricted-syntax occurrences from screens/Learn.tsx, and eslint exits 2 when suppressions in eslint-suppressions.json no longer match anything. Prune the stale entries (20 -> 17) via --prune-suppressions. Verified against origin/main: eslint exit 0, tsc exit 0, vitest 372/372. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…#480) Every CI run since the #471 merge fails at the lint step with exit 2: the agent-only rung ladder cutover removed three suppressed no-restricted-syntax occurrences from screens/Learn.tsx, and eslint exits 2 when suppressions in eslint-suppressions.json no longer match anything. Prune the stale entries (20 -> 17) via --prune-suppressions. Verified against origin/main: eslint exit 0, tsc exit 0, vitest 372/372. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
What
Third link of the B7 seam-endgame chain; gates the public beta. Full detail in the commit message. The shape:
services/prompt_safety.py— delimited untrusted-content envelopes with data-not-instructions framing and delimiter-forgery neutralization (embedded BEGIN/END copies are defanged, case/whitespace-insensitively, idempotently). Applied at every assembly boundary where student or peer-derived text enters a prompt: RAG chunks, the graph seed block, the legacy COURSE MATERIALS + shared-context JSON, and the tool→LLM boundary. Storage stays raw.Verification
27-case red-first injection suite (14 red at base) including an end-to-end FunctionModel test proving injected tool returns reach the model enveloped. Backend 1518 passed + ruff clean; 278 passed under the lock-pinned 1.107 venv; evals replay green across all six datasets. Full local e2e cycle pre-merge; results below.
Closes#150.
🤖 Generated with Claude Code