Add privacy-safe Sentry AI agent monitoring for OpenAI answer calls

[BigSimmo]

Summary

• Wrap the OpenAI client with Sentry's instrumentOpenAiClient in a new src/lib/observability/agent-monitoring.ts module so Sentry's AI agents views show per-call operation, model, latency, and token usage. recordInputs / recordOutputs are false: clinical queries, source evidence, prompts, generated answers, and embedding inputs are never recorded.
• The wrap is gated on SENTRY_DSN being set, a positive traces sample rate, and an initialized Sentry client, so it is inert in the ingestion worker, in Vitest, and in demo/dev mode without Sentry configuration.
• Extend privacySafeTransactionEvent with an explicit allowlist of gen_ai metadata attributes (system, operation name, request/response model, response id, finish reasons, token usage, conversation id) and rebuild gen_ai span descriptions as <operation> <model> from allowlisted attributes; message, prompt, tool-payload, and embedding-input attributes are stripped on export even if a future SDK version records them.
• Set dataCollection.genAI: { inputs: false, outputs: false } in both src/sentry.server.config.ts and src/sentry.edge.config.ts as defense in depth.
• Group each answer request's LLM calls (embedding, generation, verification, summaries) into one Sentry agent-monitoring conversation by calling Sentry.setConversationId(interactionId) in /api/answer and /api/answer/stream — the id is the request's synthetic UUID, never query text.
• Raise the @sentry/nextjs manifest floor from ^10.17.0 to ^10.67.0 (the minimum carrying the agent-monitoring APIs; 10.69.0 is installed and pinned in the lockfile) and sync package-lock.json (npm also normalized pre-existing @emnapi/* optional-dep drift).
• Sentry.setUser is deliberately not wired: the committed privacy boundary (tests/error-tracking.test.ts) asserts user is stripped from every outgoing event, and identity-linking clinical-query telemetry would need its own governance review first.
• New unit tests cover the gen_ai scrubbing boundary and the monitoring gate; docs/error-tracking.md and docs/codebase-index.md are updated.
• Enabling the integration in production remains an operator action: set SENTRY_DSN (and optionally raise SENTRY_TRACES_SAMPLE_RATE above the 0.1 default) in the Railway service environment per the operator-approval section of docs/error-tracking.md. No DSN is committed to the repository.

RAG impact: no retrieval behaviour change — instrumentation-only telemetry wrapper around the OpenAI client and request-scoped conversation ids; no ranking, selection, ordering, or eval surface is touched, and no ragRanking-protected path is in the diff.

Verification

• npm run verify:pr-local — exit 0; plan ran check:runtime, check:installed-lock-parity, format:changed, lint, typecheck, test, build, eval:rag:offline; decisive lines: Offline RAG fixture and manifest validation passed (36 golden cases, 23 suites). and Offline RAG fixture and production-contract checks passed.
• npm run verify:cheap — exit 0; Gate-manifest OK: all 32 verify:cheap gates are enforced in CI; unit suite Test Files 458 passed (458) / Tests 4781 passed | 4 skipped (4785)
• Focused: npx vitest run tests/error-tracking.test.ts tests/agent-monitoring.test.ts — Test Files 2 passed (2) / Tests 9 passed (9)

UI verification not run: no UI, routing, styling, or browser-facing change — the diff is server-side observability, API-route scope calls, tests, and docs.

npm run eval:retrieval:quality / eval:rag / check:production-readiness not run: provider-backed live gates requiring explicit approval; no retrieval, ranking, or answer-content behaviour changed (offline RAG fixture validation passed in verify:pr-local).

Risk and rollout

• Risk: low — the instrumentation is metadata-only, wrapped in try/catch, and inert unless Sentry is initialized with a DSN and tracing enabled; a wrap failure returns the unwrapped client so answer generation is unaffected. The scrubber change only widens an allowlist with non-payload gen_ai keys; all payload-bearing attributes remain stripped and the updated tests pin that.
• Rollback: revert this PR's squash commit; at runtime, set SENTRY_TRACES_SAMPLE_RATE=0 (stops all agent spans, keeps error capture) or unset SENTRY_DSN (stops all Sentry export) and restart the service.
• Provider or production effects: None from this diff by itself — no provider was contacted during development or verification. Telemetry to Sentry begins only when the operator sets SENTRY_DSN in the deployment environment.

Clinical Governance Preflight

• Source-backed claims still require linked source verification before clinical use
• No patient-identifiable document workflow was introduced or expanded without explicit governance approval
• Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy)
• Service-role keys and private document access remain server-only
• Demo/synthetic content remains clearly separated from real clinical sources
• Source metadata, review status, and outdated/unknown-source behavior remain conservative
• Deployment classification/TGA SaMD impact was checked when clinical decision-support behavior changed

Notes

• responses.parse (schema-parsed generation) is not in the Sentry SDK's instrumentation registry and emits no gen_ai span; responses.create and embeddings.create are covered.
• The Sentry ConversationId default integration stamps gen_ai.conversation.id onto gen_ai spans from the isolation scope, which is why the attribute is in the scrubber allowlist.
• The worker imports src/lib/openai.ts; the client gate (Sentry.getClient()) keeps the wrap a no-op there because the worker never runs Sentry.init().

🤖 Generated with Claude Code

https://claude.ai/code/session_01SBBj97cbs3rNGk8BX3g1CV

---

Generated by Claude Code

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in:50 minutes

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 16c49936-7467-45c9-a7c7-ce1d7de1d792

📥 Commits

Reviewing files that changed from the base of the PR and between 578ca5d and 4f00cdc.

⛔ Files ignored due to path filters (1)

• package-lock.json is excluded by !**/package-lock.json

📒 Files selected for processing (13)

• docs/branch-review-ledger.md
• docs/codebase-index.md
• docs/error-tracking.md
• package.json
• src/app/api/answer/route.ts
• src/app/api/answer/stream/route.ts
• src/lib/observability/agent-monitoring.ts
• src/lib/observability/error-tracking.ts
• src/lib/openai.ts
• src/sentry.edge.config.ts
• src/sentry.server.config.ts
• tests/agent-monitoring.test.ts
• tests/error-tracking.test.ts

---

_{Comment @coderabbitai help to get the list of available commands.}

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[ecc-tools]

🔒 Upgrade Required

Private repository analysis requires Pro or Enterprise.

Upgrade: https://ecc.tools/pricing?plan=pro

---

_{ECC Tools keeps the core app open, and puts private repos, team features, and enterprise controls behind paid tiers.}

[github-actions]

CI triage

CI failed on this PR. Automated classification of the 8 failed job(s):

• Lighthouse budget (advisory) — needs investigation: inspect the failing step and uploaded diagnostics; rerun only after classifying the cause.
• Safety and config checks — needs investigation: inspect the failing step and uploaded diagnostics; rerun only after classifying the cause.
• Build — needs investigation: inspect the failing step and uploaded diagnostics; rerun only after classifying the cause.
• Unit coverage — needs investigation: inspect the failing step and uploaded diagnostics; rerun only after classifying the cause.
• Static PR checks — needs investigation: inspect the failing step and uploaded diagnostics; rerun only after classifying the cause.
• Container images / app-image — needs investigation: inspect the failing step and uploaded diagnostics; rerun only after classifying the cause.
• Container images / worker-image — needs investigation: inspect the failing step and uploaded diagnostics; rerun only after classifying the cause.
• PR required — needs investigation: inspect the failing step and uploaded diagnostics; rerun only after classifying the cause.

Compared with main CI run #7634 (cancelled).

Classification is evidence routing, not permission to ignore a failure. Exact quarantined Playwright identities remain governed by the flake ledger.

[ecc-tools]

🔒 Upgrade Required

Private repository analysis requires Pro or Enterprise.

Upgrade: https://ecc.tools/pricing?plan=pro

---

_{ECC Tools keeps the core app open, and puts private repos, team features, and enterprise controls behind paid tiers.}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[ecc-tools]

🔒 Upgrade Required

Private repository analysis requires Pro or Enterprise.

Upgrade: https://ecc.tools/pricing?plan=pro

---

_{ECC Tools keeps the core app open, and puts private repos, team features, and enterprise controls behind paid tiers.}

[ecc-tools]

🔒 Upgrade Required

Private repository analysis requires Pro or Enterprise.

Upgrade: https://ecc.tools/pricing?plan=pro

---

_{ECC Tools keeps the core app open, and puts private repos, team features, and enterprise controls behind paid tiers.}