feat(observability): admin analytics + cost API /api/admin/analytics (#120) by Darkest-Teddy · Pull Request #376 · SaplingLearn/Sapling

Darkest-Teddy · 2026-07-22T00:46:14Z

What & why

Closes#120 — the admin-only, read-only analytics + cost-rollup API over the events / llm_usage tables. This is the backend the future admin dashboard (#121/#122) consumes.

New routes/admin_analytics.py, mounted at /api/admin/analytics, every endpoint gated by require_admin.

Endpoints

Method	Path	Returns
GET	`/usage/summary`	per-`event_type` counts, total events, distinct active users
GET	`/usage/by-user`	per-user event counts (by category) + LLM `cost_usd` + `total_tokens`, paginated
GET	`/llm/cost?group_by=user\|feature\|model`	token + cost rollups by dimension, with `totals`
GET	`/errors`	`error.*` events with path/method/status/duration from payload, paginated

All accept from/to ISO bounds (default: last 30 days) and return typed Pydantic models. No raw content — fingerprints only.

Notes

PostgREST has no GROUP BY, so the grouped endpoints scan the date-bounded rows and aggregate in Python (as the issue sanctions). Scans page via select_with_count and use its exact count both to stop and to detect the _SCAN_CAP truncation ceiling — logged, never a silent row-cap loss. /errors needs no aggregation, so it paginates server-side.
Date-range filtering uses a list-valued created_at param (gte.… + lte.…), which PostgREST ANDs.

Success criteria (#120)

All four endpoints return correct aggregates against seeded rows (tested).
Every endpoint rejected for non-admins / allowed for admins (require_admin, tested).
group_by switches user/feature/model correctly.
Date-range filtering + pagination work; default range last 30 days.
Router mounted in main.py under /api/admin/analytics.
Tests in backend/tests/, pass with a faithful mocked table().

Testing

11 cases over a fake table() that honors eq/gte/lte/like filters, ordering, and limit/offset — so aggregation, group_by switching, date filtering, default range, pagination, and admin gating are all genuinely exercised.

Stacking

Stacked on #375 (#118) — needs the 0032events/llm_usage tables. Base branch is feat/118-llm-usage-observability; retarget to main once #375 merges.

🤖 Generated with Claude Code

…120) Closes#120. Read-only, admin-only query API over the events + llm_usage tables (from #118) — the backend the future admin dashboard consumes. New routes/admin_analytics.py, mounted at /api/admin/analytics, every endpoint gated by require_admin: - GET /usage/summary — per-event_type counts, total events, distinct active users in range. - GET /usage/by-user — per-user event counts (by category) + LLM cost_usd + total_tokens; paginated. - GET /llm/cost?group_by= — token + cost rollups grouped by user|feature| model, with totals. - GET /errors — error.* events with path/method/status/duration from payload; paginated server-side. All endpoints take from/to ISO bounds (default last 30 days) and return typed Pydantic models (fingerprints only, no raw content). PostgREST has no GROUP BY, so grouped endpoints scan the date-bounded rows and aggregate in Python; scans page via select_with_count and use its exact count to stop and to detect (and log) the truncation ceiling — no silent row-cap loss. Tests: 11 cases over a faithful table() fake covering aggregation, group_by switching, date-range filtering, default range, pagination, and admin gating. Stacked on #118 (needs the 0032 events/llm_usage tables). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

coderabbitai · 2026-07-22T00:46:23Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (2)

^production$
^staging$

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Free

Run ID: c720c9b7-9665-4141-8916-1b657f906f0d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

_{Comment @coderabbitai help to get the list of available commands.}

cloudflare-workers-and-pages · 2026-07-22T00:50:09Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	frontend-staging	`7f1e776`	Commit Preview URL Branch Preview URL	Jul 22 2026, 12:52 AM

… admin analytics (#120) Rebase of PR #375 (which included stacked #376) onto current main, rebuilt as a fresh application of the branch diff plus the adaptations main now requires. Core (unchanged from #375/#376): - agents/usage.py: record_agent_usage(result, feature=, task=, user_id=) — one-line, never-raising usage capture for every Pydantic AI run. - services/events_service.py: bounded-queue fire-and-forget writer draining events/llm_usage rows through db/connection.table() off the request thread. - services/llm_pricing.py: token-field normalization + per-1K price map; unknown real models record cost_usd=NULL with a one-time warning. - gemini_service: call_gemini / call_gemini_multiturn log via _log_gemini_usage (feature= threaded from callers; json delegates). - routes/admin_analytics.py (+ mount): admin usage/cost rollup endpoints. - tests/test_usage_instrumentation_coverage.py: AST guard — a module that runs an agent without referencing record_agent_usage fails CI. Rebase adaptations: - Migration renumbered 0032_observability.sql -> 0035_observability.sql (main grew 0032-0034 in the meantime); content unchanged, header comment updated. - main.py lifespan: events_service.start_worker()/shutdown() coexists with #406's Logfire wiring (configure/instrument_pydantic_ai/instrument_fastapi). - Conflict resolutions keep main's guardrail semantics and add capture on top: notes.py wraps _run_note_worker results (WORKER_LIMITS + 413/500 mapping intact) and the note_chat try/except (ORCHESTRATOR_LIMITS + degraded reply intact); learn.py wraps the _prepare_chat_run-based _chat_via_agent; calendar_service keeps usage_limits=WORKER_LIMITS. - NEW run-sites landed on main since the branch: * SSE streaming tutor (#349): stream_agent_turn grows an optional on_usage(run_result) hook fed by the final AgentRunResultEvent, called once on the success path before on_complete (tokens are spent even if persistence fails); /chat/stream and /start-session/stream pass record_agent_usage(feature="chat_tutor", task="chat_tutor"). Error rungs and the Rung-1 legacy fallback don't fire it — legacy usage is captured inside call_gemini_multiturn(feature=). Covered in test_chat_stream.py. * gemini_vision_backend: per-page ocr_vision_agent run wrapped (feature="document", task="ocr_vision"; no user_id — extraction is content-addressed and user-agnostic). - SAPLING_MODEL_MODE=function: 'function:<task>' model names record with cost_usd=NULL and NO unpriced-model warning (they are the e2e/CI seam, not real spend); real unknown models keep the one-time warning. Tested. Verification: full backend suite 1275 passed / 27 skipped; ruff clean; AST guard green over all current run-sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(observability): capture LLM token usage + cost per call (#118) + admin analytics (#120) Rebase of PR #375 (which included stacked #376) onto current main, rebuilt as a fresh application of the branch diff plus the adaptations main now requires. Core (unchanged from #375/#376): - agents/usage.py: record_agent_usage(result, feature=, task=, user_id=) — one-line, never-raising usage capture for every Pydantic AI run. - services/events_service.py: bounded-queue fire-and-forget writer draining events/llm_usage rows through db/connection.table() off the request thread. - services/llm_pricing.py: token-field normalization + per-1K price map; unknown real models record cost_usd=NULL with a one-time warning. - gemini_service: call_gemini / call_gemini_multiturn log via _log_gemini_usage (feature= threaded from callers; json delegates). - routes/admin_analytics.py (+ mount): admin usage/cost rollup endpoints. - tests/test_usage_instrumentation_coverage.py: AST guard — a module that runs an agent without referencing record_agent_usage fails CI. Rebase adaptations: - Migration renumbered 0032_observability.sql -> 0035_observability.sql (main grew 0032-0034 in the meantime); content unchanged, header comment updated. - main.py lifespan: events_service.start_worker()/shutdown() coexists with #406's Logfire wiring (configure/instrument_pydantic_ai/instrument_fastapi). - Conflict resolutions keep main's guardrail semantics and add capture on top: notes.py wraps _run_note_worker results (WORKER_LIMITS + 413/500 mapping intact) and the note_chat try/except (ORCHESTRATOR_LIMITS + degraded reply intact); learn.py wraps the _prepare_chat_run-based _chat_via_agent; calendar_service keeps usage_limits=WORKER_LIMITS. - NEW run-sites landed on main since the branch: * SSE streaming tutor (#349): stream_agent_turn grows an optional on_usage(run_result) hook fed by the final AgentRunResultEvent, called once on the success path before on_complete (tokens are spent even if persistence fails); /chat/stream and /start-session/stream pass record_agent_usage(feature="chat_tutor", task="chat_tutor"). Error rungs and the Rung-1 legacy fallback don't fire it — legacy usage is captured inside call_gemini_multiturn(feature=). Covered in test_chat_stream.py. * gemini_vision_backend: per-page ocr_vision_agent run wrapped (feature="document", task="ocr_vision"; no user_id — extraction is content-addressed and user-agnostic). - SAPLING_MODEL_MODE=function: 'function:<task>' model names record with cost_usd=NULL and NO unpriced-model warning (they are the e2e/CI seam, not real spend); real unknown models keep the one-time warning. Tested. Verification: full backend suite 1275 passed / 27 skipped; ruff clean; AST guard green over all current run-sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(observability): apply #375 review fixes — range validation, truncation flag, private caching, poison-row salvage, per-call-site usage guard - admin_analytics _resolve_range: validate from/to as ISO 8601 (422 naming the bad param) and reject from > to; validated strings echoed unchanged. - Surface the 100k _SCAN_CAP: _scan_range returns (rows, truncated) and UsageSummary/UsageByUser/LLMCost carry `truncated: bool = False` (the /errors feed paginates server-side, so it has no cap to surface). - All 4 analytics GETs now send `Cache-Control: private` via a Response param. - test_default_range_is_last_30_days freezes the module clock at 2026-07-21 so the fixture window can't rot after 2026-08-09. - events_service._flush_batch: a failed bulk insert now retries rows one at a time, dropping only the rows that individually fail (per-row debug log + one warning with the drop count); never-raise contract kept, unit-tested with a 3-row batch where only the poison row is lost. - test_usage_instrumentation_coverage: upgraded the file-level substring guard to a real per-call-site AST check — every agent-run site needs an enclosing record_agent_usage, with pass-through runner helpers (return await agent.run(...)) checked at their module-local callers instead. The docstring now states the exact granularity and the cross-module blind spot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: AndresL230 <190146319+AndresL230@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Darkest-Teddy · 2026-07-22T00:46:14Z

What & why

Closes#120 — the admin-only, read-only analytics + cost-rollup API over the events / llm_usage tables. This is the backend the future admin dashboard (#121/#122) consumes.

New routes/admin_analytics.py, mounted at /api/admin/analytics, every endpoint gated by require_admin.

Endpoints

Method	Path	Returns
GET	`/usage/summary`	per-`event_type` counts, total events, distinct active users
GET	`/usage/by-user`	per-user event counts (by category) + LLM `cost_usd` + `total_tokens`, paginated
GET	`/llm/cost?group_by=user\|feature\|model`	token + cost rollups by dimension, with `totals`
GET	`/errors`	`error.*` events with path/method/status/duration from payload, paginated

All accept from/to ISO bounds (default: last 30 days) and return typed Pydantic models. No raw content — fingerprints only.

Notes

PostgREST has no GROUP BY, so the grouped endpoints scan the date-bounded rows and aggregate in Python (as the issue sanctions). Scans page via select_with_count and use its exact count both to stop and to detect the _SCAN_CAP truncation ceiling — logged, never a silent row-cap loss. /errors needs no aggregation, so it paginates server-side.
Date-range filtering uses a list-valued created_at param (gte.… + lte.…), which PostgREST ANDs.

Success criteria (#120)

All four endpoints return correct aggregates against seeded rows (tested).
Every endpoint rejected for non-admins / allowed for admins (require_admin, tested).
group_by switches user/feature/model correctly.
Date-range filtering + pagination work; default range last 30 days.
Router mounted in main.py under /api/admin/analytics.
Tests in backend/tests/, pass with a faithful mocked table().

Testing

11 cases over a fake table() that honors eq/gte/lte/like filters, ordering, and limit/offset — so aggregation, group_by switching, date filtering, default range, pagination, and admin gating are all genuinely exercised.

Stacking

Stacked on #375 (#118) — needs the 0032events/llm_usage tables. Base branch is feat/118-llm-usage-observability; retarget to main once #375 merges.

🤖 Generated with Claude Code

…120) Closes#120. Read-only, admin-only query API over the events + llm_usage tables (from #118) — the backend the future admin dashboard consumes. New routes/admin_analytics.py, mounted at /api/admin/analytics, every endpoint gated by require_admin: - GET /usage/summary — per-event_type counts, total events, distinct active users in range. - GET /usage/by-user — per-user event counts (by category) + LLM cost_usd + total_tokens; paginated. - GET /llm/cost?group_by= — token + cost rollups grouped by user|feature| model, with totals. - GET /errors — error.* events with path/method/status/duration from payload; paginated server-side. All endpoints take from/to ISO bounds (default last 30 days) and return typed Pydantic models (fingerprints only, no raw content). PostgREST has no GROUP BY, so grouped endpoints scan the date-bounded rows and aggregate in Python; scans page via select_with_count and use its exact count to stop and to detect (and log) the truncation ceiling — no silent row-cap loss. Tests: 11 cases over a faithful table() fake covering aggregation, group_by switching, date-range filtering, default range, pagination, and admin gating. Stacked on #118 (needs the 0032 events/llm_usage tables). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

coderabbitai · 2026-07-22T00:46:23Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (2)

^production$
^staging$

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Free

Run ID: c720c9b7-9665-4141-8916-1b657f906f0d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

_{Comment @coderabbitai help to get the list of available commands.}

cloudflare-workers-and-pages · 2026-07-22T00:50:09Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	frontend-staging	`7f1e776`	Commit Preview URL Branch Preview URL	Jul 22 2026, 12:52 AM

… admin analytics (#120) Rebase of PR #375 (which included stacked #376) onto current main, rebuilt as a fresh application of the branch diff plus the adaptations main now requires. Core (unchanged from #375/#376): - agents/usage.py: record_agent_usage(result, feature=, task=, user_id=) — one-line, never-raising usage capture for every Pydantic AI run. - services/events_service.py: bounded-queue fire-and-forget writer draining events/llm_usage rows through db/connection.table() off the request thread. - services/llm_pricing.py: token-field normalization + per-1K price map; unknown real models record cost_usd=NULL with a one-time warning. - gemini_service: call_gemini / call_gemini_multiturn log via _log_gemini_usage (feature= threaded from callers; json delegates). - routes/admin_analytics.py (+ mount): admin usage/cost rollup endpoints. - tests/test_usage_instrumentation_coverage.py: AST guard — a module that runs an agent without referencing record_agent_usage fails CI. Rebase adaptations: - Migration renumbered 0032_observability.sql -> 0035_observability.sql (main grew 0032-0034 in the meantime); content unchanged, header comment updated. - main.py lifespan: events_service.start_worker()/shutdown() coexists with #406's Logfire wiring (configure/instrument_pydantic_ai/instrument_fastapi). - Conflict resolutions keep main's guardrail semantics and add capture on top: notes.py wraps _run_note_worker results (WORKER_LIMITS + 413/500 mapping intact) and the note_chat try/except (ORCHESTRATOR_LIMITS + degraded reply intact); learn.py wraps the _prepare_chat_run-based _chat_via_agent; calendar_service keeps usage_limits=WORKER_LIMITS. - NEW run-sites landed on main since the branch: * SSE streaming tutor (#349): stream_agent_turn grows an optional on_usage(run_result) hook fed by the final AgentRunResultEvent, called once on the success path before on_complete (tokens are spent even if persistence fails); /chat/stream and /start-session/stream pass record_agent_usage(feature="chat_tutor", task="chat_tutor"). Error rungs and the Rung-1 legacy fallback don't fire it — legacy usage is captured inside call_gemini_multiturn(feature=). Covered in test_chat_stream.py. * gemini_vision_backend: per-page ocr_vision_agent run wrapped (feature="document", task="ocr_vision"; no user_id — extraction is content-addressed and user-agnostic). - SAPLING_MODEL_MODE=function: 'function:<task>' model names record with cost_usd=NULL and NO unpriced-model warning (they are the e2e/CI seam, not real spend); real unknown models keep the one-time warning. Tested. Verification: full backend suite 1275 passed / 27 skipped; ruff clean; AST guard green over all current run-sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(observability): capture LLM token usage + cost per call (#118) + admin analytics (#120) Rebase of PR #375 (which included stacked #376) onto current main, rebuilt as a fresh application of the branch diff plus the adaptations main now requires. Core (unchanged from #375/#376): - agents/usage.py: record_agent_usage(result, feature=, task=, user_id=) — one-line, never-raising usage capture for every Pydantic AI run. - services/events_service.py: bounded-queue fire-and-forget writer draining events/llm_usage rows through db/connection.table() off the request thread. - services/llm_pricing.py: token-field normalization + per-1K price map; unknown real models record cost_usd=NULL with a one-time warning. - gemini_service: call_gemini / call_gemini_multiturn log via _log_gemini_usage (feature= threaded from callers; json delegates). - routes/admin_analytics.py (+ mount): admin usage/cost rollup endpoints. - tests/test_usage_instrumentation_coverage.py: AST guard — a module that runs an agent without referencing record_agent_usage fails CI. Rebase adaptations: - Migration renumbered 0032_observability.sql -> 0035_observability.sql (main grew 0032-0034 in the meantime); content unchanged, header comment updated. - main.py lifespan: events_service.start_worker()/shutdown() coexists with #406's Logfire wiring (configure/instrument_pydantic_ai/instrument_fastapi). - Conflict resolutions keep main's guardrail semantics and add capture on top: notes.py wraps _run_note_worker results (WORKER_LIMITS + 413/500 mapping intact) and the note_chat try/except (ORCHESTRATOR_LIMITS + degraded reply intact); learn.py wraps the _prepare_chat_run-based _chat_via_agent; calendar_service keeps usage_limits=WORKER_LIMITS. - NEW run-sites landed on main since the branch: * SSE streaming tutor (#349): stream_agent_turn grows an optional on_usage(run_result) hook fed by the final AgentRunResultEvent, called once on the success path before on_complete (tokens are spent even if persistence fails); /chat/stream and /start-session/stream pass record_agent_usage(feature="chat_tutor", task="chat_tutor"). Error rungs and the Rung-1 legacy fallback don't fire it — legacy usage is captured inside call_gemini_multiturn(feature=). Covered in test_chat_stream.py. * gemini_vision_backend: per-page ocr_vision_agent run wrapped (feature="document", task="ocr_vision"; no user_id — extraction is content-addressed and user-agnostic). - SAPLING_MODEL_MODE=function: 'function:<task>' model names record with cost_usd=NULL and NO unpriced-model warning (they are the e2e/CI seam, not real spend); real unknown models keep the one-time warning. Tested. Verification: full backend suite 1275 passed / 27 skipped; ruff clean; AST guard green over all current run-sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(observability): apply #375 review fixes — range validation, truncation flag, private caching, poison-row salvage, per-call-site usage guard - admin_analytics _resolve_range: validate from/to as ISO 8601 (422 naming the bad param) and reject from > to; validated strings echoed unchanged. - Surface the 100k _SCAN_CAP: _scan_range returns (rows, truncated) and UsageSummary/UsageByUser/LLMCost carry `truncated: bool = False` (the /errors feed paginates server-side, so it has no cap to surface). - All 4 analytics GETs now send `Cache-Control: private` via a Response param. - test_default_range_is_last_30_days freezes the module clock at 2026-07-21 so the fixture window can't rot after 2026-08-09. - events_service._flush_batch: a failed bulk insert now retries rows one at a time, dropping only the rows that individually fail (per-row debug log + one warning with the drop count); never-raise contract kept, unit-tested with a 3-row batch where only the poison row is lost. - test_usage_instrumentation_coverage: upgraded the file-level substring guard to a real per-call-site AST check — every agent-run site needs an enclosing record_agent_usage, with pass-through runner helpers (return await agent.run(...)) checked at their module-local callers instead. The docstring now states the exact granularity and the cross-module blind spot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: AndresL230 <190146319+AndresL230@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Darkest-Teddy · 2026-07-22T00:46:14Z

What & why

Closes#120 — the admin-only, read-only analytics + cost-rollup API over the events / llm_usage tables. This is the backend the future admin dashboard (#121/#122) consumes.

New routes/admin_analytics.py, mounted at /api/admin/analytics, every endpoint gated by require_admin.

Endpoints

Method	Path	Returns
GET	`/usage/summary`	per-`event_type` counts, total events, distinct active users
GET	`/usage/by-user`	per-user event counts (by category) + LLM `cost_usd` + `total_tokens`, paginated
GET	`/llm/cost?group_by=user\|feature\|model`	token + cost rollups by dimension, with `totals`
GET	`/errors`	`error.*` events with path/method/status/duration from payload, paginated

All accept from/to ISO bounds (default: last 30 days) and return typed Pydantic models. No raw content — fingerprints only.

Notes

PostgREST has no GROUP BY, so the grouped endpoints scan the date-bounded rows and aggregate in Python (as the issue sanctions). Scans page via select_with_count and use its exact count both to stop and to detect the _SCAN_CAP truncation ceiling — logged, never a silent row-cap loss. /errors needs no aggregation, so it paginates server-side.
Date-range filtering uses a list-valued created_at param (gte.… + lte.…), which PostgREST ANDs.

Success criteria (#120)

All four endpoints return correct aggregates against seeded rows (tested).
Every endpoint rejected for non-admins / allowed for admins (require_admin, tested).
group_by switches user/feature/model correctly.
Date-range filtering + pagination work; default range last 30 days.
Router mounted in main.py under /api/admin/analytics.
Tests in backend/tests/, pass with a faithful mocked table().

Testing

11 cases over a fake table() that honors eq/gte/lte/like filters, ordering, and limit/offset — so aggregation, group_by switching, date filtering, default range, pagination, and admin gating are all genuinely exercised.

Stacking

Stacked on #375 (#118) — needs the 0032events/llm_usage tables. Base branch is feat/118-llm-usage-observability; retarget to main once #375 merges.

🤖 Generated with Claude Code

…120) Closes#120. Read-only, admin-only query API over the events + llm_usage tables (from #118) — the backend the future admin dashboard consumes. New routes/admin_analytics.py, mounted at /api/admin/analytics, every endpoint gated by require_admin: - GET /usage/summary — per-event_type counts, total events, distinct active users in range. - GET /usage/by-user — per-user event counts (by category) + LLM cost_usd + total_tokens; paginated. - GET /llm/cost?group_by= — token + cost rollups grouped by user|feature| model, with totals. - GET /errors — error.* events with path/method/status/duration from payload; paginated server-side. All endpoints take from/to ISO bounds (default last 30 days) and return typed Pydantic models (fingerprints only, no raw content). PostgREST has no GROUP BY, so grouped endpoints scan the date-bounded rows and aggregate in Python; scans page via select_with_count and use its exact count to stop and to detect (and log) the truncation ceiling — no silent row-cap loss. Tests: 11 cases over a faithful table() fake covering aggregation, group_by switching, date-range filtering, default range, pagination, and admin gating. Stacked on #118 (needs the 0032 events/llm_usage tables). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

coderabbitai · 2026-07-22T00:46:23Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (2)

^production$
^staging$

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Free

Run ID: c720c9b7-9665-4141-8916-1b657f906f0d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

_{Comment @coderabbitai help to get the list of available commands.}

cloudflare-workers-and-pages · 2026-07-22T00:50:09Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	frontend-staging	`7f1e776`	Commit Preview URL Branch Preview URL	Jul 22 2026, 12:52 AM

… admin analytics (#120) Rebase of PR #375 (which included stacked #376) onto current main, rebuilt as a fresh application of the branch diff plus the adaptations main now requires. Core (unchanged from #375/#376): - agents/usage.py: record_agent_usage(result, feature=, task=, user_id=) — one-line, never-raising usage capture for every Pydantic AI run. - services/events_service.py: bounded-queue fire-and-forget writer draining events/llm_usage rows through db/connection.table() off the request thread. - services/llm_pricing.py: token-field normalization + per-1K price map; unknown real models record cost_usd=NULL with a one-time warning. - gemini_service: call_gemini / call_gemini_multiturn log via _log_gemini_usage (feature= threaded from callers; json delegates). - routes/admin_analytics.py (+ mount): admin usage/cost rollup endpoints. - tests/test_usage_instrumentation_coverage.py: AST guard — a module that runs an agent without referencing record_agent_usage fails CI. Rebase adaptations: - Migration renumbered 0032_observability.sql -> 0035_observability.sql (main grew 0032-0034 in the meantime); content unchanged, header comment updated. - main.py lifespan: events_service.start_worker()/shutdown() coexists with #406's Logfire wiring (configure/instrument_pydantic_ai/instrument_fastapi). - Conflict resolutions keep main's guardrail semantics and add capture on top: notes.py wraps _run_note_worker results (WORKER_LIMITS + 413/500 mapping intact) and the note_chat try/except (ORCHESTRATOR_LIMITS + degraded reply intact); learn.py wraps the _prepare_chat_run-based _chat_via_agent; calendar_service keeps usage_limits=WORKER_LIMITS. - NEW run-sites landed on main since the branch: * SSE streaming tutor (#349): stream_agent_turn grows an optional on_usage(run_result) hook fed by the final AgentRunResultEvent, called once on the success path before on_complete (tokens are spent even if persistence fails); /chat/stream and /start-session/stream pass record_agent_usage(feature="chat_tutor", task="chat_tutor"). Error rungs and the Rung-1 legacy fallback don't fire it — legacy usage is captured inside call_gemini_multiturn(feature=). Covered in test_chat_stream.py. * gemini_vision_backend: per-page ocr_vision_agent run wrapped (feature="document", task="ocr_vision"; no user_id — extraction is content-addressed and user-agnostic). - SAPLING_MODEL_MODE=function: 'function:<task>' model names record with cost_usd=NULL and NO unpriced-model warning (they are the e2e/CI seam, not real spend); real unknown models keep the one-time warning. Tested. Verification: full backend suite 1275 passed / 27 skipped; ruff clean; AST guard green over all current run-sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(observability): capture LLM token usage + cost per call (#118) + admin analytics (#120) Rebase of PR #375 (which included stacked #376) onto current main, rebuilt as a fresh application of the branch diff plus the adaptations main now requires. Core (unchanged from #375/#376): - agents/usage.py: record_agent_usage(result, feature=, task=, user_id=) — one-line, never-raising usage capture for every Pydantic AI run. - services/events_service.py: bounded-queue fire-and-forget writer draining events/llm_usage rows through db/connection.table() off the request thread. - services/llm_pricing.py: token-field normalization + per-1K price map; unknown real models record cost_usd=NULL with a one-time warning. - gemini_service: call_gemini / call_gemini_multiturn log via _log_gemini_usage (feature= threaded from callers; json delegates). - routes/admin_analytics.py (+ mount): admin usage/cost rollup endpoints. - tests/test_usage_instrumentation_coverage.py: AST guard — a module that runs an agent without referencing record_agent_usage fails CI. Rebase adaptations: - Migration renumbered 0032_observability.sql -> 0035_observability.sql (main grew 0032-0034 in the meantime); content unchanged, header comment updated. - main.py lifespan: events_service.start_worker()/shutdown() coexists with #406's Logfire wiring (configure/instrument_pydantic_ai/instrument_fastapi). - Conflict resolutions keep main's guardrail semantics and add capture on top: notes.py wraps _run_note_worker results (WORKER_LIMITS + 413/500 mapping intact) and the note_chat try/except (ORCHESTRATOR_LIMITS + degraded reply intact); learn.py wraps the _prepare_chat_run-based _chat_via_agent; calendar_service keeps usage_limits=WORKER_LIMITS. - NEW run-sites landed on main since the branch: * SSE streaming tutor (#349): stream_agent_turn grows an optional on_usage(run_result) hook fed by the final AgentRunResultEvent, called once on the success path before on_complete (tokens are spent even if persistence fails); /chat/stream and /start-session/stream pass record_agent_usage(feature="chat_tutor", task="chat_tutor"). Error rungs and the Rung-1 legacy fallback don't fire it — legacy usage is captured inside call_gemini_multiturn(feature=). Covered in test_chat_stream.py. * gemini_vision_backend: per-page ocr_vision_agent run wrapped (feature="document", task="ocr_vision"; no user_id — extraction is content-addressed and user-agnostic). - SAPLING_MODEL_MODE=function: 'function:<task>' model names record with cost_usd=NULL and NO unpriced-model warning (they are the e2e/CI seam, not real spend); real unknown models keep the one-time warning. Tested. Verification: full backend suite 1275 passed / 27 skipped; ruff clean; AST guard green over all current run-sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(observability): apply #375 review fixes — range validation, truncation flag, private caching, poison-row salvage, per-call-site usage guard - admin_analytics _resolve_range: validate from/to as ISO 8601 (422 naming the bad param) and reject from > to; validated strings echoed unchanged. - Surface the 100k _SCAN_CAP: _scan_range returns (rows, truncated) and UsageSummary/UsageByUser/LLMCost carry `truncated: bool = False` (the /errors feed paginates server-side, so it has no cap to surface). - All 4 analytics GETs now send `Cache-Control: private` via a Response param. - test_default_range_is_last_30_days freezes the module clock at 2026-07-21 so the fixture window can't rot after 2026-08-09. - events_service._flush_batch: a failed bulk insert now retries rows one at a time, dropping only the rows that individually fail (per-row debug log + one warning with the drop count); never-raise contract kept, unit-tested with a 3-row batch where only the poison row is lost. - test_usage_instrumentation_coverage: upgraded the file-level substring guard to a real per-call-site AST check — every agent-run site needs an enclosing record_agent_usage, with pass-through runner helpers (return await agent.run(...)) checked at their module-local callers instead. The docstring now states the exact granularity and the cross-module blind spot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: AndresL230 <190146319+AndresL230@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Darkest-Teddy · 2026-07-22T00:46:14Z

What & why

Closes#120 — the admin-only, read-only analytics + cost-rollup API over the events / llm_usage tables. This is the backend the future admin dashboard (#121/#122) consumes.

New routes/admin_analytics.py, mounted at /api/admin/analytics, every endpoint gated by require_admin.

Endpoints

Method	Path	Returns
GET	`/usage/summary`	per-`event_type` counts, total events, distinct active users
GET	`/usage/by-user`	per-user event counts (by category) + LLM `cost_usd` + `total_tokens`, paginated
GET	`/llm/cost?group_by=user\|feature\|model`	token + cost rollups by dimension, with `totals`
GET	`/errors`	`error.*` events with path/method/status/duration from payload, paginated

All accept from/to ISO bounds (default: last 30 days) and return typed Pydantic models. No raw content — fingerprints only.

Notes

PostgREST has no GROUP BY, so the grouped endpoints scan the date-bounded rows and aggregate in Python (as the issue sanctions). Scans page via select_with_count and use its exact count both to stop and to detect the _SCAN_CAP truncation ceiling — logged, never a silent row-cap loss. /errors needs no aggregation, so it paginates server-side.
Date-range filtering uses a list-valued created_at param (gte.… + lte.…), which PostgREST ANDs.

Success criteria (#120)

All four endpoints return correct aggregates against seeded rows (tested).
Every endpoint rejected for non-admins / allowed for admins (require_admin, tested).
group_by switches user/feature/model correctly.
Date-range filtering + pagination work; default range last 30 days.
Router mounted in main.py under /api/admin/analytics.
Tests in backend/tests/, pass with a faithful mocked table().

Testing

11 cases over a fake table() that honors eq/gte/lte/like filters, ordering, and limit/offset — so aggregation, group_by switching, date filtering, default range, pagination, and admin gating are all genuinely exercised.

Stacking

Stacked on #375 (#118) — needs the 0032events/llm_usage tables. Base branch is feat/118-llm-usage-observability; retarget to main once #375 merges.

🤖 Generated with Claude Code

…120) Closes#120. Read-only, admin-only query API over the events + llm_usage tables (from #118) — the backend the future admin dashboard consumes. New routes/admin_analytics.py, mounted at /api/admin/analytics, every endpoint gated by require_admin: - GET /usage/summary — per-event_type counts, total events, distinct active users in range. - GET /usage/by-user — per-user event counts (by category) + LLM cost_usd + total_tokens; paginated. - GET /llm/cost?group_by= — token + cost rollups grouped by user|feature| model, with totals. - GET /errors — error.* events with path/method/status/duration from payload; paginated server-side. All endpoints take from/to ISO bounds (default last 30 days) and return typed Pydantic models (fingerprints only, no raw content). PostgREST has no GROUP BY, so grouped endpoints scan the date-bounded rows and aggregate in Python; scans page via select_with_count and use its exact count to stop and to detect (and log) the truncation ceiling — no silent row-cap loss. Tests: 11 cases over a faithful table() fake covering aggregation, group_by switching, date-range filtering, default range, pagination, and admin gating. Stacked on #118 (needs the 0032 events/llm_usage tables). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

coderabbitai · 2026-07-22T00:46:23Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (2)

^production$
^staging$

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Free

Run ID: c720c9b7-9665-4141-8916-1b657f906f0d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

_{Comment @coderabbitai help to get the list of available commands.}

cloudflare-workers-and-pages · 2026-07-22T00:50:09Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	frontend-staging	`7f1e776`	Commit Preview URL Branch Preview URL	Jul 22 2026, 12:52 AM

… admin analytics (#120) Rebase of PR #375 (which included stacked #376) onto current main, rebuilt as a fresh application of the branch diff plus the adaptations main now requires. Core (unchanged from #375/#376): - agents/usage.py: record_agent_usage(result, feature=, task=, user_id=) — one-line, never-raising usage capture for every Pydantic AI run. - services/events_service.py: bounded-queue fire-and-forget writer draining events/llm_usage rows through db/connection.table() off the request thread. - services/llm_pricing.py: token-field normalization + per-1K price map; unknown real models record cost_usd=NULL with a one-time warning. - gemini_service: call_gemini / call_gemini_multiturn log via _log_gemini_usage (feature= threaded from callers; json delegates). - routes/admin_analytics.py (+ mount): admin usage/cost rollup endpoints. - tests/test_usage_instrumentation_coverage.py: AST guard — a module that runs an agent without referencing record_agent_usage fails CI. Rebase adaptations: - Migration renumbered 0032_observability.sql -> 0035_observability.sql (main grew 0032-0034 in the meantime); content unchanged, header comment updated. - main.py lifespan: events_service.start_worker()/shutdown() coexists with #406's Logfire wiring (configure/instrument_pydantic_ai/instrument_fastapi). - Conflict resolutions keep main's guardrail semantics and add capture on top: notes.py wraps _run_note_worker results (WORKER_LIMITS + 413/500 mapping intact) and the note_chat try/except (ORCHESTRATOR_LIMITS + degraded reply intact); learn.py wraps the _prepare_chat_run-based _chat_via_agent; calendar_service keeps usage_limits=WORKER_LIMITS. - NEW run-sites landed on main since the branch: * SSE streaming tutor (#349): stream_agent_turn grows an optional on_usage(run_result) hook fed by the final AgentRunResultEvent, called once on the success path before on_complete (tokens are spent even if persistence fails); /chat/stream and /start-session/stream pass record_agent_usage(feature="chat_tutor", task="chat_tutor"). Error rungs and the Rung-1 legacy fallback don't fire it — legacy usage is captured inside call_gemini_multiturn(feature=). Covered in test_chat_stream.py. * gemini_vision_backend: per-page ocr_vision_agent run wrapped (feature="document", task="ocr_vision"; no user_id — extraction is content-addressed and user-agnostic). - SAPLING_MODEL_MODE=function: 'function:<task>' model names record with cost_usd=NULL and NO unpriced-model warning (they are the e2e/CI seam, not real spend); real unknown models keep the one-time warning. Tested. Verification: full backend suite 1275 passed / 27 skipped; ruff clean; AST guard green over all current run-sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(observability): capture LLM token usage + cost per call (#118) + admin analytics (#120) Rebase of PR #375 (which included stacked #376) onto current main, rebuilt as a fresh application of the branch diff plus the adaptations main now requires. Core (unchanged from #375/#376): - agents/usage.py: record_agent_usage(result, feature=, task=, user_id=) — one-line, never-raising usage capture for every Pydantic AI run. - services/events_service.py: bounded-queue fire-and-forget writer draining events/llm_usage rows through db/connection.table() off the request thread. - services/llm_pricing.py: token-field normalization + per-1K price map; unknown real models record cost_usd=NULL with a one-time warning. - gemini_service: call_gemini / call_gemini_multiturn log via _log_gemini_usage (feature= threaded from callers; json delegates). - routes/admin_analytics.py (+ mount): admin usage/cost rollup endpoints. - tests/test_usage_instrumentation_coverage.py: AST guard — a module that runs an agent without referencing record_agent_usage fails CI. Rebase adaptations: - Migration renumbered 0032_observability.sql -> 0035_observability.sql (main grew 0032-0034 in the meantime); content unchanged, header comment updated. - main.py lifespan: events_service.start_worker()/shutdown() coexists with #406's Logfire wiring (configure/instrument_pydantic_ai/instrument_fastapi). - Conflict resolutions keep main's guardrail semantics and add capture on top: notes.py wraps _run_note_worker results (WORKER_LIMITS + 413/500 mapping intact) and the note_chat try/except (ORCHESTRATOR_LIMITS + degraded reply intact); learn.py wraps the _prepare_chat_run-based _chat_via_agent; calendar_service keeps usage_limits=WORKER_LIMITS. - NEW run-sites landed on main since the branch: * SSE streaming tutor (#349): stream_agent_turn grows an optional on_usage(run_result) hook fed by the final AgentRunResultEvent, called once on the success path before on_complete (tokens are spent even if persistence fails); /chat/stream and /start-session/stream pass record_agent_usage(feature="chat_tutor", task="chat_tutor"). Error rungs and the Rung-1 legacy fallback don't fire it — legacy usage is captured inside call_gemini_multiturn(feature=). Covered in test_chat_stream.py. * gemini_vision_backend: per-page ocr_vision_agent run wrapped (feature="document", task="ocr_vision"; no user_id — extraction is content-addressed and user-agnostic). - SAPLING_MODEL_MODE=function: 'function:<task>' model names record with cost_usd=NULL and NO unpriced-model warning (they are the e2e/CI seam, not real spend); real unknown models keep the one-time warning. Tested. Verification: full backend suite 1275 passed / 27 skipped; ruff clean; AST guard green over all current run-sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(observability): apply #375 review fixes — range validation, truncation flag, private caching, poison-row salvage, per-call-site usage guard - admin_analytics _resolve_range: validate from/to as ISO 8601 (422 naming the bad param) and reject from > to; validated strings echoed unchanged. - Surface the 100k _SCAN_CAP: _scan_range returns (rows, truncated) and UsageSummary/UsageByUser/LLMCost carry `truncated: bool = False` (the /errors feed paginates server-side, so it has no cap to surface). - All 4 analytics GETs now send `Cache-Control: private` via a Response param. - test_default_range_is_last_30_days freezes the module clock at 2026-07-21 so the fixture window can't rot after 2026-08-09. - events_service._flush_batch: a failed bulk insert now retries rows one at a time, dropping only the rows that individually fail (per-row debug log + one warning with the drop count); never-raise contract kept, unit-tested with a 3-row batch where only the poison row is lost. - test_usage_instrumentation_coverage: upgraded the file-level substring guard to a real per-call-site AST check — every agent-run site needs an enclosing record_agent_usage, with pass-through runner helpers (return await agent.run(...)) checked at their module-local callers instead. The docstring now states the exact granularity and the cross-module blind spot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: AndresL230 <190146319+AndresL230@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Darkest-Teddy · 2026-07-22T00:46:14Z

What & why

Closes#120 — the admin-only, read-only analytics + cost-rollup API over the events / llm_usage tables. This is the backend the future admin dashboard (#121/#122) consumes.

New routes/admin_analytics.py, mounted at /api/admin/analytics, every endpoint gated by require_admin.

Endpoints

Method	Path	Returns
GET	`/usage/summary`	per-`event_type` counts, total events, distinct active users
GET	`/usage/by-user`	per-user event counts (by category) + LLM `cost_usd` + `total_tokens`, paginated
GET	`/llm/cost?group_by=user\|feature\|model`	token + cost rollups by dimension, with `totals`
GET	`/errors`	`error.*` events with path/method/status/duration from payload, paginated

All accept from/to ISO bounds (default: last 30 days) and return typed Pydantic models. No raw content — fingerprints only.

Notes

PostgREST has no GROUP BY, so the grouped endpoints scan the date-bounded rows and aggregate in Python (as the issue sanctions). Scans page via select_with_count and use its exact count both to stop and to detect the _SCAN_CAP truncation ceiling — logged, never a silent row-cap loss. /errors needs no aggregation, so it paginates server-side.
Date-range filtering uses a list-valued created_at param (gte.… + lte.…), which PostgREST ANDs.

Success criteria (#120)

All four endpoints return correct aggregates against seeded rows (tested).
Every endpoint rejected for non-admins / allowed for admins (require_admin, tested).
group_by switches user/feature/model correctly.
Date-range filtering + pagination work; default range last 30 days.
Router mounted in main.py under /api/admin/analytics.
Tests in backend/tests/, pass with a faithful mocked table().

Testing

11 cases over a fake table() that honors eq/gte/lte/like filters, ordering, and limit/offset — so aggregation, group_by switching, date filtering, default range, pagination, and admin gating are all genuinely exercised.

Stacking

Stacked on #375 (#118) — needs the 0032events/llm_usage tables. Base branch is feat/118-llm-usage-observability; retarget to main once #375 merges.

🤖 Generated with Claude Code

…120) Closes#120. Read-only, admin-only query API over the events + llm_usage tables (from #118) — the backend the future admin dashboard consumes. New routes/admin_analytics.py, mounted at /api/admin/analytics, every endpoint gated by require_admin: - GET /usage/summary — per-event_type counts, total events, distinct active users in range. - GET /usage/by-user — per-user event counts (by category) + LLM cost_usd + total_tokens; paginated. - GET /llm/cost?group_by= — token + cost rollups grouped by user|feature| model, with totals. - GET /errors — error.* events with path/method/status/duration from payload; paginated server-side. All endpoints take from/to ISO bounds (default last 30 days) and return typed Pydantic models (fingerprints only, no raw content). PostgREST has no GROUP BY, so grouped endpoints scan the date-bounded rows and aggregate in Python; scans page via select_with_count and use its exact count to stop and to detect (and log) the truncation ceiling — no silent row-cap loss. Tests: 11 cases over a faithful table() fake covering aggregation, group_by switching, date-range filtering, default range, pagination, and admin gating. Stacked on #118 (needs the 0032 events/llm_usage tables). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

coderabbitai · 2026-07-22T00:46:23Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (2)

^production$
^staging$

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Free

Run ID: c720c9b7-9665-4141-8916-1b657f906f0d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

_{Comment @coderabbitai help to get the list of available commands.}

cloudflare-workers-and-pages · 2026-07-22T00:50:09Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	frontend-staging	`7f1e776`	Commit Preview URL Branch Preview URL	Jul 22 2026, 12:52 AM

… admin analytics (#120) Rebase of PR #375 (which included stacked #376) onto current main, rebuilt as a fresh application of the branch diff plus the adaptations main now requires. Core (unchanged from #375/#376): - agents/usage.py: record_agent_usage(result, feature=, task=, user_id=) — one-line, never-raising usage capture for every Pydantic AI run. - services/events_service.py: bounded-queue fire-and-forget writer draining events/llm_usage rows through db/connection.table() off the request thread. - services/llm_pricing.py: token-field normalization + per-1K price map; unknown real models record cost_usd=NULL with a one-time warning. - gemini_service: call_gemini / call_gemini_multiturn log via _log_gemini_usage (feature= threaded from callers; json delegates). - routes/admin_analytics.py (+ mount): admin usage/cost rollup endpoints. - tests/test_usage_instrumentation_coverage.py: AST guard — a module that runs an agent without referencing record_agent_usage fails CI. Rebase adaptations: - Migration renumbered 0032_observability.sql -> 0035_observability.sql (main grew 0032-0034 in the meantime); content unchanged, header comment updated. - main.py lifespan: events_service.start_worker()/shutdown() coexists with #406's Logfire wiring (configure/instrument_pydantic_ai/instrument_fastapi). - Conflict resolutions keep main's guardrail semantics and add capture on top: notes.py wraps _run_note_worker results (WORKER_LIMITS + 413/500 mapping intact) and the note_chat try/except (ORCHESTRATOR_LIMITS + degraded reply intact); learn.py wraps the _prepare_chat_run-based _chat_via_agent; calendar_service keeps usage_limits=WORKER_LIMITS. - NEW run-sites landed on main since the branch: * SSE streaming tutor (#349): stream_agent_turn grows an optional on_usage(run_result) hook fed by the final AgentRunResultEvent, called once on the success path before on_complete (tokens are spent even if persistence fails); /chat/stream and /start-session/stream pass record_agent_usage(feature="chat_tutor", task="chat_tutor"). Error rungs and the Rung-1 legacy fallback don't fire it — legacy usage is captured inside call_gemini_multiturn(feature=). Covered in test_chat_stream.py. * gemini_vision_backend: per-page ocr_vision_agent run wrapped (feature="document", task="ocr_vision"; no user_id — extraction is content-addressed and user-agnostic). - SAPLING_MODEL_MODE=function: 'function:<task>' model names record with cost_usd=NULL and NO unpriced-model warning (they are the e2e/CI seam, not real spend); real unknown models keep the one-time warning. Tested. Verification: full backend suite 1275 passed / 27 skipped; ruff clean; AST guard green over all current run-sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(observability): capture LLM token usage + cost per call (#118) + admin analytics (#120) Rebase of PR #375 (which included stacked #376) onto current main, rebuilt as a fresh application of the branch diff plus the adaptations main now requires. Core (unchanged from #375/#376): - agents/usage.py: record_agent_usage(result, feature=, task=, user_id=) — one-line, never-raising usage capture for every Pydantic AI run. - services/events_service.py: bounded-queue fire-and-forget writer draining events/llm_usage rows through db/connection.table() off the request thread. - services/llm_pricing.py: token-field normalization + per-1K price map; unknown real models record cost_usd=NULL with a one-time warning. - gemini_service: call_gemini / call_gemini_multiturn log via _log_gemini_usage (feature= threaded from callers; json delegates). - routes/admin_analytics.py (+ mount): admin usage/cost rollup endpoints. - tests/test_usage_instrumentation_coverage.py: AST guard — a module that runs an agent without referencing record_agent_usage fails CI. Rebase adaptations: - Migration renumbered 0032_observability.sql -> 0035_observability.sql (main grew 0032-0034 in the meantime); content unchanged, header comment updated. - main.py lifespan: events_service.start_worker()/shutdown() coexists with #406's Logfire wiring (configure/instrument_pydantic_ai/instrument_fastapi). - Conflict resolutions keep main's guardrail semantics and add capture on top: notes.py wraps _run_note_worker results (WORKER_LIMITS + 413/500 mapping intact) and the note_chat try/except (ORCHESTRATOR_LIMITS + degraded reply intact); learn.py wraps the _prepare_chat_run-based _chat_via_agent; calendar_service keeps usage_limits=WORKER_LIMITS. - NEW run-sites landed on main since the branch: * SSE streaming tutor (#349): stream_agent_turn grows an optional on_usage(run_result) hook fed by the final AgentRunResultEvent, called once on the success path before on_complete (tokens are spent even if persistence fails); /chat/stream and /start-session/stream pass record_agent_usage(feature="chat_tutor", task="chat_tutor"). Error rungs and the Rung-1 legacy fallback don't fire it — legacy usage is captured inside call_gemini_multiturn(feature=). Covered in test_chat_stream.py. * gemini_vision_backend: per-page ocr_vision_agent run wrapped (feature="document", task="ocr_vision"; no user_id — extraction is content-addressed and user-agnostic). - SAPLING_MODEL_MODE=function: 'function:<task>' model names record with cost_usd=NULL and NO unpriced-model warning (they are the e2e/CI seam, not real spend); real unknown models keep the one-time warning. Tested. Verification: full backend suite 1275 passed / 27 skipped; ruff clean; AST guard green over all current run-sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(observability): apply #375 review fixes — range validation, truncation flag, private caching, poison-row salvage, per-call-site usage guard - admin_analytics _resolve_range: validate from/to as ISO 8601 (422 naming the bad param) and reject from > to; validated strings echoed unchanged. - Surface the 100k _SCAN_CAP: _scan_range returns (rows, truncated) and UsageSummary/UsageByUser/LLMCost carry `truncated: bool = False` (the /errors feed paginates server-side, so it has no cap to surface). - All 4 analytics GETs now send `Cache-Control: private` via a Response param. - test_default_range_is_last_30_days freezes the module clock at 2026-07-21 so the fixture window can't rot after 2026-08-09. - events_service._flush_batch: a failed bulk insert now retries rows one at a time, dropping only the rows that individually fail (per-row debug log + one warning with the drop count); never-raise contract kept, unit-tested with a 3-row batch where only the poison row is lost. - test_usage_instrumentation_coverage: upgraded the file-level substring guard to a real per-call-site AST check — every agent-run site needs an enclosing record_agent_usage, with pass-through runner helpers (return await agent.run(...)) checked at their module-local callers instead. The docstring now states the exact granularity and the cross-module blind spot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: AndresL230 <190146319+AndresL230@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Darkest-Teddy · 2026-07-22T00:46:14Z

What & why

Closes#120 — the admin-only, read-only analytics + cost-rollup API over the events / llm_usage tables. This is the backend the future admin dashboard (#121/#122) consumes.

New routes/admin_analytics.py, mounted at /api/admin/analytics, every endpoint gated by require_admin.

Endpoints

Method	Path	Returns
GET	`/usage/summary`	per-`event_type` counts, total events, distinct active users
GET	`/usage/by-user`	per-user event counts (by category) + LLM `cost_usd` + `total_tokens`, paginated
GET	`/llm/cost?group_by=user\|feature\|model`	token + cost rollups by dimension, with `totals`
GET	`/errors`	`error.*` events with path/method/status/duration from payload, paginated

All accept from/to ISO bounds (default: last 30 days) and return typed Pydantic models. No raw content — fingerprints only.

Notes

PostgREST has no GROUP BY, so the grouped endpoints scan the date-bounded rows and aggregate in Python (as the issue sanctions). Scans page via select_with_count and use its exact count both to stop and to detect the _SCAN_CAP truncation ceiling — logged, never a silent row-cap loss. /errors needs no aggregation, so it paginates server-side.
Date-range filtering uses a list-valued created_at param (gte.… + lte.…), which PostgREST ANDs.

Success criteria (#120)

All four endpoints return correct aggregates against seeded rows (tested).
Every endpoint rejected for non-admins / allowed for admins (require_admin, tested).
group_by switches user/feature/model correctly.
Date-range filtering + pagination work; default range last 30 days.
Router mounted in main.py under /api/admin/analytics.
Tests in backend/tests/, pass with a faithful mocked table().

Testing

11 cases over a fake table() that honors eq/gte/lte/like filters, ordering, and limit/offset — so aggregation, group_by switching, date filtering, default range, pagination, and admin gating are all genuinely exercised.

Stacking

Stacked on #375 (#118) — needs the 0032events/llm_usage tables. Base branch is feat/118-llm-usage-observability; retarget to main once #375 merges.

🤖 Generated with Claude Code

…120) Closes#120. Read-only, admin-only query API over the events + llm_usage tables (from #118) — the backend the future admin dashboard consumes. New routes/admin_analytics.py, mounted at /api/admin/analytics, every endpoint gated by require_admin: - GET /usage/summary — per-event_type counts, total events, distinct active users in range. - GET /usage/by-user — per-user event counts (by category) + LLM cost_usd + total_tokens; paginated. - GET /llm/cost?group_by= — token + cost rollups grouped by user|feature| model, with totals. - GET /errors — error.* events with path/method/status/duration from payload; paginated server-side. All endpoints take from/to ISO bounds (default last 30 days) and return typed Pydantic models (fingerprints only, no raw content). PostgREST has no GROUP BY, so grouped endpoints scan the date-bounded rows and aggregate in Python; scans page via select_with_count and use its exact count to stop and to detect (and log) the truncation ceiling — no silent row-cap loss. Tests: 11 cases over a faithful table() fake covering aggregation, group_by switching, date-range filtering, default range, pagination, and admin gating. Stacked on #118 (needs the 0032 events/llm_usage tables). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

coderabbitai · 2026-07-22T00:46:23Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (2)

^production$
^staging$

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Free

Run ID: c720c9b7-9665-4141-8916-1b657f906f0d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

_{Comment @coderabbitai help to get the list of available commands.}

cloudflare-workers-and-pages · 2026-07-22T00:50:09Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	frontend-staging	`7f1e776`	Commit Preview URL Branch Preview URL	Jul 22 2026, 12:52 AM

… admin analytics (#120) Rebase of PR #375 (which included stacked #376) onto current main, rebuilt as a fresh application of the branch diff plus the adaptations main now requires. Core (unchanged from #375/#376): - agents/usage.py: record_agent_usage(result, feature=, task=, user_id=) — one-line, never-raising usage capture for every Pydantic AI run. - services/events_service.py: bounded-queue fire-and-forget writer draining events/llm_usage rows through db/connection.table() off the request thread. - services/llm_pricing.py: token-field normalization + per-1K price map; unknown real models record cost_usd=NULL with a one-time warning. - gemini_service: call_gemini / call_gemini_multiturn log via _log_gemini_usage (feature= threaded from callers; json delegates). - routes/admin_analytics.py (+ mount): admin usage/cost rollup endpoints. - tests/test_usage_instrumentation_coverage.py: AST guard — a module that runs an agent without referencing record_agent_usage fails CI. Rebase adaptations: - Migration renumbered 0032_observability.sql -> 0035_observability.sql (main grew 0032-0034 in the meantime); content unchanged, header comment updated. - main.py lifespan: events_service.start_worker()/shutdown() coexists with #406's Logfire wiring (configure/instrument_pydantic_ai/instrument_fastapi). - Conflict resolutions keep main's guardrail semantics and add capture on top: notes.py wraps _run_note_worker results (WORKER_LIMITS + 413/500 mapping intact) and the note_chat try/except (ORCHESTRATOR_LIMITS + degraded reply intact); learn.py wraps the _prepare_chat_run-based _chat_via_agent; calendar_service keeps usage_limits=WORKER_LIMITS. - NEW run-sites landed on main since the branch: * SSE streaming tutor (#349): stream_agent_turn grows an optional on_usage(run_result) hook fed by the final AgentRunResultEvent, called once on the success path before on_complete (tokens are spent even if persistence fails); /chat/stream and /start-session/stream pass record_agent_usage(feature="chat_tutor", task="chat_tutor"). Error rungs and the Rung-1 legacy fallback don't fire it — legacy usage is captured inside call_gemini_multiturn(feature=). Covered in test_chat_stream.py. * gemini_vision_backend: per-page ocr_vision_agent run wrapped (feature="document", task="ocr_vision"; no user_id — extraction is content-addressed and user-agnostic). - SAPLING_MODEL_MODE=function: 'function:<task>' model names record with cost_usd=NULL and NO unpriced-model warning (they are the e2e/CI seam, not real spend); real unknown models keep the one-time warning. Tested. Verification: full backend suite 1275 passed / 27 skipped; ruff clean; AST guard green over all current run-sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(observability): capture LLM token usage + cost per call (#118) + admin analytics (#120) Rebase of PR #375 (which included stacked #376) onto current main, rebuilt as a fresh application of the branch diff plus the adaptations main now requires. Core (unchanged from #375/#376): - agents/usage.py: record_agent_usage(result, feature=, task=, user_id=) — one-line, never-raising usage capture for every Pydantic AI run. - services/events_service.py: bounded-queue fire-and-forget writer draining events/llm_usage rows through db/connection.table() off the request thread. - services/llm_pricing.py: token-field normalization + per-1K price map; unknown real models record cost_usd=NULL with a one-time warning. - gemini_service: call_gemini / call_gemini_multiturn log via _log_gemini_usage (feature= threaded from callers; json delegates). - routes/admin_analytics.py (+ mount): admin usage/cost rollup endpoints. - tests/test_usage_instrumentation_coverage.py: AST guard — a module that runs an agent without referencing record_agent_usage fails CI. Rebase adaptations: - Migration renumbered 0032_observability.sql -> 0035_observability.sql (main grew 0032-0034 in the meantime); content unchanged, header comment updated. - main.py lifespan: events_service.start_worker()/shutdown() coexists with #406's Logfire wiring (configure/instrument_pydantic_ai/instrument_fastapi). - Conflict resolutions keep main's guardrail semantics and add capture on top: notes.py wraps _run_note_worker results (WORKER_LIMITS + 413/500 mapping intact) and the note_chat try/except (ORCHESTRATOR_LIMITS + degraded reply intact); learn.py wraps the _prepare_chat_run-based _chat_via_agent; calendar_service keeps usage_limits=WORKER_LIMITS. - NEW run-sites landed on main since the branch: * SSE streaming tutor (#349): stream_agent_turn grows an optional on_usage(run_result) hook fed by the final AgentRunResultEvent, called once on the success path before on_complete (tokens are spent even if persistence fails); /chat/stream and /start-session/stream pass record_agent_usage(feature="chat_tutor", task="chat_tutor"). Error rungs and the Rung-1 legacy fallback don't fire it — legacy usage is captured inside call_gemini_multiturn(feature=). Covered in test_chat_stream.py. * gemini_vision_backend: per-page ocr_vision_agent run wrapped (feature="document", task="ocr_vision"; no user_id — extraction is content-addressed and user-agnostic). - SAPLING_MODEL_MODE=function: 'function:<task>' model names record with cost_usd=NULL and NO unpriced-model warning (they are the e2e/CI seam, not real spend); real unknown models keep the one-time warning. Tested. Verification: full backend suite 1275 passed / 27 skipped; ruff clean; AST guard green over all current run-sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(observability): apply #375 review fixes — range validation, truncation flag, private caching, poison-row salvage, per-call-site usage guard - admin_analytics _resolve_range: validate from/to as ISO 8601 (422 naming the bad param) and reject from > to; validated strings echoed unchanged. - Surface the 100k _SCAN_CAP: _scan_range returns (rows, truncated) and UsageSummary/UsageByUser/LLMCost carry `truncated: bool = False` (the /errors feed paginates server-side, so it has no cap to surface). - All 4 analytics GETs now send `Cache-Control: private` via a Response param. - test_default_range_is_last_30_days freezes the module clock at 2026-07-21 so the fixture window can't rot after 2026-08-09. - events_service._flush_batch: a failed bulk insert now retries rows one at a time, dropping only the rows that individually fail (per-row debug log + one warning with the drop count); never-raise contract kept, unit-tested with a 3-row batch where only the poison row is lost. - test_usage_instrumentation_coverage: upgraded the file-level substring guard to a real per-call-site AST check — every agent-run site needs an enclosing record_agent_usage, with pass-through runner helpers (return await agent.run(...)) checked at their module-local callers instead. The docstring now states the exact granularity and the cross-module blind spot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: AndresL230 <190146319+AndresL230@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Darkest-Teddy · 2026-07-22T00:46:14Z

What & why

Closes#120 — the admin-only, read-only analytics + cost-rollup API over the events / llm_usage tables. This is the backend the future admin dashboard (#121/#122) consumes.

New routes/admin_analytics.py, mounted at /api/admin/analytics, every endpoint gated by require_admin.

Endpoints

Method	Path	Returns
GET	`/usage/summary`	per-`event_type` counts, total events, distinct active users
GET	`/usage/by-user`	per-user event counts (by category) + LLM `cost_usd` + `total_tokens`, paginated
GET	`/llm/cost?group_by=user\|feature\|model`	token + cost rollups by dimension, with `totals`
GET	`/errors`	`error.*` events with path/method/status/duration from payload, paginated

All accept from/to ISO bounds (default: last 30 days) and return typed Pydantic models. No raw content — fingerprints only.

Notes

PostgREST has no GROUP BY, so the grouped endpoints scan the date-bounded rows and aggregate in Python (as the issue sanctions). Scans page via select_with_count and use its exact count both to stop and to detect the _SCAN_CAP truncation ceiling — logged, never a silent row-cap loss. /errors needs no aggregation, so it paginates server-side.
Date-range filtering uses a list-valued created_at param (gte.… + lte.…), which PostgREST ANDs.

Success criteria (#120)

All four endpoints return correct aggregates against seeded rows (tested).
Every endpoint rejected for non-admins / allowed for admins (require_admin, tested).
group_by switches user/feature/model correctly.
Date-range filtering + pagination work; default range last 30 days.
Router mounted in main.py under /api/admin/analytics.
Tests in backend/tests/, pass with a faithful mocked table().

Testing

11 cases over a fake table() that honors eq/gte/lte/like filters, ordering, and limit/offset — so aggregation, group_by switching, date filtering, default range, pagination, and admin gating are all genuinely exercised.

Stacking

Stacked on #375 (#118) — needs the 0032events/llm_usage tables. Base branch is feat/118-llm-usage-observability; retarget to main once #375 merges.

🤖 Generated with Claude Code

…120) Closes#120. Read-only, admin-only query API over the events + llm_usage tables (from #118) — the backend the future admin dashboard consumes. New routes/admin_analytics.py, mounted at /api/admin/analytics, every endpoint gated by require_admin: - GET /usage/summary — per-event_type counts, total events, distinct active users in range. - GET /usage/by-user — per-user event counts (by category) + LLM cost_usd + total_tokens; paginated. - GET /llm/cost?group_by= — token + cost rollups grouped by user|feature| model, with totals. - GET /errors — error.* events with path/method/status/duration from payload; paginated server-side. All endpoints take from/to ISO bounds (default last 30 days) and return typed Pydantic models (fingerprints only, no raw content). PostgREST has no GROUP BY, so grouped endpoints scan the date-bounded rows and aggregate in Python; scans page via select_with_count and use its exact count to stop and to detect (and log) the truncation ceiling — no silent row-cap loss. Tests: 11 cases over a faithful table() fake covering aggregation, group_by switching, date-range filtering, default range, pagination, and admin gating. Stacked on #118 (needs the 0032 events/llm_usage tables). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

coderabbitai · 2026-07-22T00:46:23Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (2)

^production$
^staging$

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Free

Run ID: c720c9b7-9665-4141-8916-1b657f906f0d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

_{Comment @coderabbitai help to get the list of available commands.}

cloudflare-workers-and-pages · 2026-07-22T00:50:09Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	frontend-staging	`7f1e776`	Commit Preview URL Branch Preview URL	Jul 22 2026, 12:52 AM

… admin analytics (#120) Rebase of PR #375 (which included stacked #376) onto current main, rebuilt as a fresh application of the branch diff plus the adaptations main now requires. Core (unchanged from #375/#376): - agents/usage.py: record_agent_usage(result, feature=, task=, user_id=) — one-line, never-raising usage capture for every Pydantic AI run. - services/events_service.py: bounded-queue fire-and-forget writer draining events/llm_usage rows through db/connection.table() off the request thread. - services/llm_pricing.py: token-field normalization + per-1K price map; unknown real models record cost_usd=NULL with a one-time warning. - gemini_service: call_gemini / call_gemini_multiturn log via _log_gemini_usage (feature= threaded from callers; json delegates). - routes/admin_analytics.py (+ mount): admin usage/cost rollup endpoints. - tests/test_usage_instrumentation_coverage.py: AST guard — a module that runs an agent without referencing record_agent_usage fails CI. Rebase adaptations: - Migration renumbered 0032_observability.sql -> 0035_observability.sql (main grew 0032-0034 in the meantime); content unchanged, header comment updated. - main.py lifespan: events_service.start_worker()/shutdown() coexists with #406's Logfire wiring (configure/instrument_pydantic_ai/instrument_fastapi). - Conflict resolutions keep main's guardrail semantics and add capture on top: notes.py wraps _run_note_worker results (WORKER_LIMITS + 413/500 mapping intact) and the note_chat try/except (ORCHESTRATOR_LIMITS + degraded reply intact); learn.py wraps the _prepare_chat_run-based _chat_via_agent; calendar_service keeps usage_limits=WORKER_LIMITS. - NEW run-sites landed on main since the branch: * SSE streaming tutor (#349): stream_agent_turn grows an optional on_usage(run_result) hook fed by the final AgentRunResultEvent, called once on the success path before on_complete (tokens are spent even if persistence fails); /chat/stream and /start-session/stream pass record_agent_usage(feature="chat_tutor", task="chat_tutor"). Error rungs and the Rung-1 legacy fallback don't fire it — legacy usage is captured inside call_gemini_multiturn(feature=). Covered in test_chat_stream.py. * gemini_vision_backend: per-page ocr_vision_agent run wrapped (feature="document", task="ocr_vision"; no user_id — extraction is content-addressed and user-agnostic). - SAPLING_MODEL_MODE=function: 'function:<task>' model names record with cost_usd=NULL and NO unpriced-model warning (they are the e2e/CI seam, not real spend); real unknown models keep the one-time warning. Tested. Verification: full backend suite 1275 passed / 27 skipped; ruff clean; AST guard green over all current run-sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(observability): capture LLM token usage + cost per call (#118) + admin analytics (#120) Rebase of PR #375 (which included stacked #376) onto current main, rebuilt as a fresh application of the branch diff plus the adaptations main now requires. Core (unchanged from #375/#376): - agents/usage.py: record_agent_usage(result, feature=, task=, user_id=) — one-line, never-raising usage capture for every Pydantic AI run. - services/events_service.py: bounded-queue fire-and-forget writer draining events/llm_usage rows through db/connection.table() off the request thread. - services/llm_pricing.py: token-field normalization + per-1K price map; unknown real models record cost_usd=NULL with a one-time warning. - gemini_service: call_gemini / call_gemini_multiturn log via _log_gemini_usage (feature= threaded from callers; json delegates). - routes/admin_analytics.py (+ mount): admin usage/cost rollup endpoints. - tests/test_usage_instrumentation_coverage.py: AST guard — a module that runs an agent without referencing record_agent_usage fails CI. Rebase adaptations: - Migration renumbered 0032_observability.sql -> 0035_observability.sql (main grew 0032-0034 in the meantime); content unchanged, header comment updated. - main.py lifespan: events_service.start_worker()/shutdown() coexists with #406's Logfire wiring (configure/instrument_pydantic_ai/instrument_fastapi). - Conflict resolutions keep main's guardrail semantics and add capture on top: notes.py wraps _run_note_worker results (WORKER_LIMITS + 413/500 mapping intact) and the note_chat try/except (ORCHESTRATOR_LIMITS + degraded reply intact); learn.py wraps the _prepare_chat_run-based _chat_via_agent; calendar_service keeps usage_limits=WORKER_LIMITS. - NEW run-sites landed on main since the branch: * SSE streaming tutor (#349): stream_agent_turn grows an optional on_usage(run_result) hook fed by the final AgentRunResultEvent, called once on the success path before on_complete (tokens are spent even if persistence fails); /chat/stream and /start-session/stream pass record_agent_usage(feature="chat_tutor", task="chat_tutor"). Error rungs and the Rung-1 legacy fallback don't fire it — legacy usage is captured inside call_gemini_multiturn(feature=). Covered in test_chat_stream.py. * gemini_vision_backend: per-page ocr_vision_agent run wrapped (feature="document", task="ocr_vision"; no user_id — extraction is content-addressed and user-agnostic). - SAPLING_MODEL_MODE=function: 'function:<task>' model names record with cost_usd=NULL and NO unpriced-model warning (they are the e2e/CI seam, not real spend); real unknown models keep the one-time warning. Tested. Verification: full backend suite 1275 passed / 27 skipped; ruff clean; AST guard green over all current run-sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(observability): apply #375 review fixes — range validation, truncation flag, private caching, poison-row salvage, per-call-site usage guard - admin_analytics _resolve_range: validate from/to as ISO 8601 (422 naming the bad param) and reject from > to; validated strings echoed unchanged. - Surface the 100k _SCAN_CAP: _scan_range returns (rows, truncated) and UsageSummary/UsageByUser/LLMCost carry `truncated: bool = False` (the /errors feed paginates server-side, so it has no cap to surface). - All 4 analytics GETs now send `Cache-Control: private` via a Response param. - test_default_range_is_last_30_days freezes the module clock at 2026-07-21 so the fixture window can't rot after 2026-08-09. - events_service._flush_batch: a failed bulk insert now retries rows one at a time, dropping only the rows that individually fail (per-row debug log + one warning with the drop count); never-raise contract kept, unit-tested with a 3-row batch where only the poison row is lost. - test_usage_instrumentation_coverage: upgraded the file-level substring guard to a real per-call-site AST check — every agent-run site needs an enclosing record_agent_usage, with pass-through runner helpers (return await agent.run(...)) checked at their module-local callers instead. The docstring now states the exact granularity and the cross-module blind spot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: AndresL230 <190146319+AndresL230@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Darkest-Teddy · 2026-07-22T00:46:14Z

What & why

Closes#120 — the admin-only, read-only analytics + cost-rollup API over the events / llm_usage tables. This is the backend the future admin dashboard (#121/#122) consumes.

New routes/admin_analytics.py, mounted at /api/admin/analytics, every endpoint gated by require_admin.

Endpoints

Method	Path	Returns
GET	`/usage/summary`	per-`event_type` counts, total events, distinct active users
GET	`/usage/by-user`	per-user event counts (by category) + LLM `cost_usd` + `total_tokens`, paginated
GET	`/llm/cost?group_by=user\|feature\|model`	token + cost rollups by dimension, with `totals`
GET	`/errors`	`error.*` events with path/method/status/duration from payload, paginated

All accept from/to ISO bounds (default: last 30 days) and return typed Pydantic models. No raw content — fingerprints only.

Notes

PostgREST has no GROUP BY, so the grouped endpoints scan the date-bounded rows and aggregate in Python (as the issue sanctions). Scans page via select_with_count and use its exact count both to stop and to detect the _SCAN_CAP truncation ceiling — logged, never a silent row-cap loss. /errors needs no aggregation, so it paginates server-side.
Date-range filtering uses a list-valued created_at param (gte.… + lte.…), which PostgREST ANDs.

Success criteria (#120)

All four endpoints return correct aggregates against seeded rows (tested).
Every endpoint rejected for non-admins / allowed for admins (require_admin, tested).
group_by switches user/feature/model correctly.
Date-range filtering + pagination work; default range last 30 days.
Router mounted in main.py under /api/admin/analytics.
Tests in backend/tests/, pass with a faithful mocked table().

Testing

11 cases over a fake table() that honors eq/gte/lte/like filters, ordering, and limit/offset — so aggregation, group_by switching, date filtering, default range, pagination, and admin gating are all genuinely exercised.

Stacking

Stacked on #375 (#118) — needs the 0032events/llm_usage tables. Base branch is feat/118-llm-usage-observability; retarget to main once #375 merges.

🤖 Generated with Claude Code

…120) Closes#120. Read-only, admin-only query API over the events + llm_usage tables (from #118) — the backend the future admin dashboard consumes. New routes/admin_analytics.py, mounted at /api/admin/analytics, every endpoint gated by require_admin: - GET /usage/summary — per-event_type counts, total events, distinct active users in range. - GET /usage/by-user — per-user event counts (by category) + LLM cost_usd + total_tokens; paginated. - GET /llm/cost?group_by= — token + cost rollups grouped by user|feature| model, with totals. - GET /errors — error.* events with path/method/status/duration from payload; paginated server-side. All endpoints take from/to ISO bounds (default last 30 days) and return typed Pydantic models (fingerprints only, no raw content). PostgREST has no GROUP BY, so grouped endpoints scan the date-bounded rows and aggregate in Python; scans page via select_with_count and use its exact count to stop and to detect (and log) the truncation ceiling — no silent row-cap loss. Tests: 11 cases over a faithful table() fake covering aggregation, group_by switching, date-range filtering, default range, pagination, and admin gating. Stacked on #118 (needs the 0032 events/llm_usage tables). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

coderabbitai · 2026-07-22T00:46:23Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (2)

^production$
^staging$

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Free

Run ID: c720c9b7-9665-4141-8916-1b657f906f0d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

_{Comment @coderabbitai help to get the list of available commands.}

cloudflare-workers-and-pages · 2026-07-22T00:50:09Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	frontend-staging	`7f1e776`	Commit Preview URL Branch Preview URL	Jul 22 2026, 12:52 AM

… admin analytics (#120) Rebase of PR #375 (which included stacked #376) onto current main, rebuilt as a fresh application of the branch diff plus the adaptations main now requires. Core (unchanged from #375/#376): - agents/usage.py: record_agent_usage(result, feature=, task=, user_id=) — one-line, never-raising usage capture for every Pydantic AI run. - services/events_service.py: bounded-queue fire-and-forget writer draining events/llm_usage rows through db/connection.table() off the request thread. - services/llm_pricing.py: token-field normalization + per-1K price map; unknown real models record cost_usd=NULL with a one-time warning. - gemini_service: call_gemini / call_gemini_multiturn log via _log_gemini_usage (feature= threaded from callers; json delegates). - routes/admin_analytics.py (+ mount): admin usage/cost rollup endpoints. - tests/test_usage_instrumentation_coverage.py: AST guard — a module that runs an agent without referencing record_agent_usage fails CI. Rebase adaptations: - Migration renumbered 0032_observability.sql -> 0035_observability.sql (main grew 0032-0034 in the meantime); content unchanged, header comment updated. - main.py lifespan: events_service.start_worker()/shutdown() coexists with #406's Logfire wiring (configure/instrument_pydantic_ai/instrument_fastapi). - Conflict resolutions keep main's guardrail semantics and add capture on top: notes.py wraps _run_note_worker results (WORKER_LIMITS + 413/500 mapping intact) and the note_chat try/except (ORCHESTRATOR_LIMITS + degraded reply intact); learn.py wraps the _prepare_chat_run-based _chat_via_agent; calendar_service keeps usage_limits=WORKER_LIMITS. - NEW run-sites landed on main since the branch: * SSE streaming tutor (#349): stream_agent_turn grows an optional on_usage(run_result) hook fed by the final AgentRunResultEvent, called once on the success path before on_complete (tokens are spent even if persistence fails); /chat/stream and /start-session/stream pass record_agent_usage(feature="chat_tutor", task="chat_tutor"). Error rungs and the Rung-1 legacy fallback don't fire it — legacy usage is captured inside call_gemini_multiturn(feature=). Covered in test_chat_stream.py. * gemini_vision_backend: per-page ocr_vision_agent run wrapped (feature="document", task="ocr_vision"; no user_id — extraction is content-addressed and user-agnostic). - SAPLING_MODEL_MODE=function: 'function:<task>' model names record with cost_usd=NULL and NO unpriced-model warning (they are the e2e/CI seam, not real spend); real unknown models keep the one-time warning. Tested. Verification: full backend suite 1275 passed / 27 skipped; ruff clean; AST guard green over all current run-sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(observability): capture LLM token usage + cost per call (#118) + admin analytics (#120) Rebase of PR #375 (which included stacked #376) onto current main, rebuilt as a fresh application of the branch diff plus the adaptations main now requires. Core (unchanged from #375/#376): - agents/usage.py: record_agent_usage(result, feature=, task=, user_id=) — one-line, never-raising usage capture for every Pydantic AI run. - services/events_service.py: bounded-queue fire-and-forget writer draining events/llm_usage rows through db/connection.table() off the request thread. - services/llm_pricing.py: token-field normalization + per-1K price map; unknown real models record cost_usd=NULL with a one-time warning. - gemini_service: call_gemini / call_gemini_multiturn log via _log_gemini_usage (feature= threaded from callers; json delegates). - routes/admin_analytics.py (+ mount): admin usage/cost rollup endpoints. - tests/test_usage_instrumentation_coverage.py: AST guard — a module that runs an agent without referencing record_agent_usage fails CI. Rebase adaptations: - Migration renumbered 0032_observability.sql -> 0035_observability.sql (main grew 0032-0034 in the meantime); content unchanged, header comment updated. - main.py lifespan: events_service.start_worker()/shutdown() coexists with #406's Logfire wiring (configure/instrument_pydantic_ai/instrument_fastapi). - Conflict resolutions keep main's guardrail semantics and add capture on top: notes.py wraps _run_note_worker results (WORKER_LIMITS + 413/500 mapping intact) and the note_chat try/except (ORCHESTRATOR_LIMITS + degraded reply intact); learn.py wraps the _prepare_chat_run-based _chat_via_agent; calendar_service keeps usage_limits=WORKER_LIMITS. - NEW run-sites landed on main since the branch: * SSE streaming tutor (#349): stream_agent_turn grows an optional on_usage(run_result) hook fed by the final AgentRunResultEvent, called once on the success path before on_complete (tokens are spent even if persistence fails); /chat/stream and /start-session/stream pass record_agent_usage(feature="chat_tutor", task="chat_tutor"). Error rungs and the Rung-1 legacy fallback don't fire it — legacy usage is captured inside call_gemini_multiturn(feature=). Covered in test_chat_stream.py. * gemini_vision_backend: per-page ocr_vision_agent run wrapped (feature="document", task="ocr_vision"; no user_id — extraction is content-addressed and user-agnostic). - SAPLING_MODEL_MODE=function: 'function:<task>' model names record with cost_usd=NULL and NO unpriced-model warning (they are the e2e/CI seam, not real spend); real unknown models keep the one-time warning. Tested. Verification: full backend suite 1275 passed / 27 skipped; ruff clean; AST guard green over all current run-sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(observability): apply #375 review fixes — range validation, truncation flag, private caching, poison-row salvage, per-call-site usage guard - admin_analytics _resolve_range: validate from/to as ISO 8601 (422 naming the bad param) and reject from > to; validated strings echoed unchanged. - Surface the 100k _SCAN_CAP: _scan_range returns (rows, truncated) and UsageSummary/UsageByUser/LLMCost carry `truncated: bool = False` (the /errors feed paginates server-side, so it has no cap to surface). - All 4 analytics GETs now send `Cache-Control: private` via a Response param. - test_default_range_is_last_30_days freezes the module clock at 2026-07-21 so the fixture window can't rot after 2026-08-09. - events_service._flush_batch: a failed bulk insert now retries rows one at a time, dropping only the rows that individually fail (per-row debug log + one warning with the drop count); never-raise contract kept, unit-tested with a 3-row batch where only the poison row is lost. - test_usage_instrumentation_coverage: upgraded the file-level substring guard to a real per-call-site AST check — every agent-run site needs an enclosing record_agent_usage, with pass-through runner helpers (return await agent.run(...)) checked at their module-local callers instead. The docstring now states the exact granularity and the cross-module blind spot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: AndresL230 <190146319+AndresL230@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Darkest-Teddy merged commit be376eb into feat/118-llm-usage-observabilityJul 27, 2026
4 checks passed

AndresL230 mentioned this pull request Jul 29, 2026
[P2] Observability: admin analytics + cost API (/api/admin/analytics) #120
Closed
6 tasks

AndresL230 deleted the feat/120-admin-analytics-api branch August 2, 2026 18:30