ai-partner: aggregate privacy-safe analytics

[CraigBuckmaster]

Parent epic: #1446 (Amicus — AI Study Partner v1)
Phase: 5 · Size: S · Depends on: #1450 (proxy), #1471 (gap capture), #1468 (audit)

Aggregate-only usage metrics for Amicus. Zero per-user tracking. Measures feature health, conversion, and content coverage to drive roadmap decisions.

---

Files to create

• ai-proxy/src/metrics.ts — emits structured metric events from the proxy
• _tools/amicus_analytics/query.py — aggregation queries against D1
• _tools/amicus_analytics/weekly_report.py — generates the weekly aggregate dashboard
• _tools/amicus_analytics/README.md

Files to modify

• ai-proxy/src/index.ts — call metric emitters on each request lifecycle event
• app/src/services/amicus/chat.ts — optional: send non-identifying client-side metrics (retrieval latency, etc.) via a dedicated /ai/metrics endpoint

---

What gets measured

Proxy-side (no per-user identification):

Per-request metrics (aggregated by hour bucket):

• Total requests (/ai/chat)
• Success rate (200 status)
• Rate-limit hits (429)
• Auth failures (401, 402)
• Model tier split (haiku vs sonnet %)
• Mean + p50 + p95 + p99 latency
• Mean + p95 token counts (input, output)
• Gap signal rate (% of responses with gap: true)

Per-day metrics:

• Daily active premium users (count of unique auth tokens with successful request; count only — never store token)
• Daily active Partner+ users
• Total LLM cost (derived from token counts + model prices)
• Upgrade conversion events (Premium → Partner+, captured via separate RevenueCat webhook, joined with Amicus usage)

Client-side (anonymous, batched):

Optional additional metrics via POST /ai/metrics endpoint:

• Retrieval latency p95
• Peek open rate (# peeks / # FAB visible)
• Home card tap rate
• Mini-conversation turn distribution (1/2/3+ turns before dismiss)
• Citation tap-through rate

Each payload is anonymous — no user ID, no query content, no response content. Just numbers + categorical tags.

What we do NOT track:

• Individual user behavior over time (no user profiles beyond auth)
• Query text or response text
• Any content that could reidentify a user
• Device fingerprints
• Location beyond coarse country (if Cloudflare headers naturally provide it; do not enrich)

D1 schema

Minimal — most metrics live in Cloudflare's built-in analytics. D1 only holds what we need for custom queries:

CREATE TABLE amicus_hourly_metrics (
 hour_bucket TEXT PRIMARY KEY, -- 'YYYY-MM-DDTHH' UTC
 total_requests INTEGER,
 success_count INTEGER,
 rate_limit_count INTEGER,
 auth_fail_count INTEGER,
 haiku_count INTEGER,
 sonnet_count INTEGER,
 p50_latency_ms REAL,
 p95_latency_ms REAL,
 mean_input_tokens REAL,
 mean_output_tokens REAL,
 gap_signal_count INTEGER
);

CREATE TABLE amicus_daily_users (
 date TEXT PRIMARY KEY,
 dau_premium INTEGER,
 dau_partner_plus INTEGER,
 hashed_token_fingerprints TEXT -- bloom filter or similar; NOT raw tokens
);

The DAU calculation uses a hashed token fingerprint (SHA256 of auth token, truncated to 12 chars). We never store the raw token. The hash is per-day — expires nightly — so cross-day correlation is impossible even server-side.

Weekly report

python _tools/amicus_analytics/weekly_report.py

Produces markdown report:

# Amicus Weekly Report — Week of 2026-06-01

## Usage
- Total requests: 32,482
- Daily active premium users (peak): 3,247 (+8% vs prior week)
- Daily active Partner+ users: 89 (+12% vs prior week)

## Performance
- p95 latency: 1,724ms (target: <2000ms ✓)
- Success rate: 98.7%
- Rate-limit hits: 412 (1.3% of requests)

## Cost
- Total LLM cost: $3,421 (Haiku: $487, Sonnet: $2,934)
- Cost per premium user: $0.31

## Quality signals
- Gap signal rate: 3.9% (target: <5% ✓)
- Classifier needs-review rate: 3.2% (from #1468)

## Content gaps
- Top 5 unresolved corpus-gap clusters:
 1. Orthodox perspectives on Romans 9 (47 hits)
 2. Septuagint translation decisions in Isaiah 7:14 (31 hits)
 ...

Report is a markdown file suitable for pasting into a GitHub discussion or commit message.

---

Acceptance criteria

• Proxy emits hourly metrics to D1
• DAU calculation uses hashed token fingerprints; raw tokens never persisted
• Client-side metrics endpoint works and writes aggregate-only data
• Weekly report script generates markdown covering usage, perf, cost, quality, content gaps
• No per-user tracking anywhere in the pipeline
• README covers weekly ops + privacy posture
• Privacy assertion test: search the proxy code for any path that stores raw user data → zero hits

Out of scope

• Real-time dashboards → weekly cadence for v1
• A/B testing infrastructure → future consideration
• Per-user metrics → intentionally excluded on privacy grounds