feat(onboarding): first-run funnel telemetry

docs/docs/reference/telemetry.md

@@ -47,15 +47,43 @@ We collect the following categories of events:
 | `schema_complexity` | Warehouse schema structural metrics from introspection — bucketed table, column, and schema counts plus average columns per table. No schema names or content. |
 | `validator_check` | A completion-gate validator ran on session end — validator name, `ok` boolean, step, retry count, `enforced` flag (false in shadow mode), and structured `details` (model counts, elapsed time, concurrency limit — no SQL or model content). Only emitted when `ALTIMATE_VALIDATORS_ENABLED=1` or `ALTIMATE_VALIDATORS_SHADOW=1`. See [Validators](../data-engineering/validators.md). |
 | `validator_retries_exhausted` | A session terminated with unresolved validator failures after exhausting the synthetic-retry budget — names of the failing validators (no failure body content). |
+| `onboarding_started` | The first-run setup gate opened (fresh launch with no usable model). |
+| `model_picker_shown` | The provider picker was displayed. `trigger` distinguishes the first run from `/connect`, from declining Big Pickle, and from the prompt gate. |
+| `provider_selected` | A provider row was chosen — `altimate_gateway`, `anthropic`, `openai`, `google`, `big_pickle`, `search_all`, or `other` for anything outside the curated five. `provider_id` carries the raw id only for publicly-known providers, so a provider you named yourself in config is reported as `other` with no name attached. `via_search` marks a pick made inside the full catalogue after choosing "Search all providers…". **Choosing search emits this event twice for one user** — once as `search_all`, then again with the provider actually chosen — so count distinct users or filter on `via_search`, not raw event count. Recorded at the moment of choice, so a sign-in that is then cancelled still counts. |
+| `big_pickle_confirm_shown` / `big_pickle_choice` | The Big Pickle interstitial was shown, and what the user decided (`accept`/`cancel`). |
+| `gateway_device_code_issued` | The Altimate Gateway authorize URL was built and the browser open attempted. **Name note:** the flow is a browser loopback OAuth — there is no device code. The name follows the original event spec. |
+| `gateway_auth_completed` / `gateway_auth_failed` | Gateway sign-in outcome. `reason` is `timeout`, `denied`, or `error` — never the underlying message, which can contain the instance name. An unrecognised callback state does not reject the pending attempt, so a CSRF mismatch surfaces as `timeout`. |
+| `instance_connected` | Credentials received and saved. `time_to_connect_ms` runs from the start of the authorize call, so it includes the browser launch. No instance or tenant name is sent. |
+| `onboarding_completed` | A model is ready and chat is live. |
+| `scan_gate_shown` / `scan_gate_choice` | The "scan your environment?" gate appeared, and what the user did — `scan`, `skip`, or `dismissed` (esc / click-away). |
+| `environment_scan_completed` | A `project_scan` finished during onboarding — `has_dbt`, `has_warehouse`, `is_repo`, `connections_found`, and a bounded list of short `degraded` detection keys. No paths, hostnames, or connection details. Emitted only inside an onboarding session, and only once per session, so scans from `/discover` or a model-initiated call are excluded. |
+| `sample_setup_completed` | The sample dbt project was materialised. `success`, `models`, `tables`, and `reused` — the tool is deliberately re-callable, so this is per invocation. The target path is never sent. |
+| `activation_menu_shown` | The activation menu was (very likely) rendered. `variant` is `warehouse` or `no_data`. **Derived** — see the note below. |
+| `activation_job_selected` / `first_job_completed` | Which activation job the user started and, where observable, finished. Completion is reported only for the job that was actually selected, so the two form a coherent pair. **Derived** — see the note below. |
+| `first_prompt_sent` | The user's first typed message in an onboarding session. Slash commands are excluded, so the hidden `/onboard-connect` submission does not count. |
+| `onboarding_abandoned` | The CLI exited during a first run without connecting. `last_stage` is the furthest point reached: `started`, `model_picker`, `provider_setup`, `big_pickle_confirm`, or `gateway_auth`. (`connected` is a funnel position but never a `last_stage` — reaching it means the run completed, which is not an abandonment.) Only emitted for a genuine first run — opening `/connect` as an existing user does not enter the funnel, and abandonment after setup completes is out of scope by definition. Emitted on the exit path under a bounded flush, so the measured rate is a lower bound — see [Delivery & Reliability](#delivery--reliability). |
 
-Each event includes a timestamp, anonymous session ID, CLI version, and an anonymous machine ID (a random UUID stored in `~/.altimate/machine-id`, generated once and never tied to any personal information).
+### A note on the derived activation events
+
+`activation_menu_shown`, `activation_job_selected`, and `first_job_completed` are **inferred, not observed**. The activation menu is not a UI element: it is text the model writes from a prompt template, and the user picks a job by replying in free text. Nothing in the CLI can see either moment directly.
+
+They are therefore inferred from the closest deterministic signals — the menu from the command dispatch or the completed environment scan, the job from the first matching tool or skill invocation that follows. Treat the counts as **lower bounds**, and note two specific gaps:
+
+- The "something else" branch has no tool signature at all and is never counted.
+- `first_job_completed` only fires for jobs with a real completion signal. Skill-driven jobs (downstream impact, SQL review, cost) load an instruction bundle and then do their work through other tools, so their completion is not observable and they are absent from this event rather than wrongly counted in it.
+
+Each event includes a timestamp, anonymous session ID, a per-launch correlation ID (`launch_id` — a random value regenerated every process start, not persisted and not derived from your machine or identity; it exists only to group events from the same run), CLI version, and an anonymous machine ID (a random UUID stored in `~/.altimate/machine-id`, generated once and never tied to any personal information).
 
 ## Delivery & Reliability
 
 Telemetry events are buffered in memory and flushed periodically. If a flush fails (e.g., due to a transient network error), events are re-added to the buffer for one retry. On process exit, the CLI performs a final flush to avoid losing events from the current session.
 
 No events are ever written to disk. If the process is killed before the final flush, buffered events are lost. This is by design to minimize on-disk footprint.
 
+The final flush is **time-bounded** so that quitting never hangs the shell: 2 seconds on the main thread and 5 seconds in the TUI worker. When that budget expires the in-flight request is aborted and its events are dropped rather than retried — a retry would only re-queue them into a buffer that is cleared moments later, to be shipped under the next launch's correlation id.
+
+The practical consequence is a known bias, not a silent one: events emitted **on the exit path** are the most likely to be lost on a slow or unreachable network, and `onboarding_abandoned` is emitted *only* on that path. So a measured abandonment rate is a **lower bound** — under-reporting is the failure mode, never over-reporting, since a dropped event can only remove an abandonment from the count. Read drop-off numbers as a floor, and treat a change in them as meaningful only if network conditions are comparable.
+
 ## Why We Collect Telemetry
 
 Telemetry helps us:

packages/opencode/src/altimate/plugin/altimate.ts

@@ -3,6 +3,28 @@ import { createServer } from "http"
 import { randomBytes } from "crypto"
 import open from "open"
 import { AltimateApi } from "../api/client"
+// altimate_change — onboarding telemetry for the gateway sign-in funnel
+import * as OnboardingTelemetry from "../telemetry/onboarding"
+
+/**
+ * Why a failure reason is attached at the rejection site rather than inferred from the message:
+ * `gateway_auth_failed.reason` is a closed enum, and the callback's catch sees only an Error.
+ * Matching on message text would silently drift the moment any of these strings is reworded, and
+ * the `error` query param is attacker-influenced text we must not parse or forward. Tagging the
+ * error where the cause is known keeps the classification deterministic — and note that an
+ * unknown/invalid `state` never rejects a pending flow at all (the handler 400s without touching
+ * the map), so a CSRF mismatch legitimately surfaces later as `timeout`, not `denied`.
+ */
+type GatewayFailureReason = "timeout" | "denied" | "error"
+
+function markReason(err: Error, reason: GatewayFailureReason): Error {
+ return Object.assign(err, { altimateGatewayReason: reason })
+}
+
+function reasonOf(err: unknown): GatewayFailureReason {
+ const tagged = (err as { altimateGatewayReason?: GatewayFailureReason } | undefined)?.altimateGatewayReason
+ return tagged ?? "error"
+}
 
 // Loopback port range the CLI listens on for the browser to deliver the gateway
 // credential after sign-in. We prefer 7317 (mnemonic + otherwise unused in this
@@ -153,7 +175,8 @@ async function doStartCallbackServer(): Promise<void> {
 
  const error = url.searchParams.get("error")
  if (error) {
- entry.reject(new Error(error))
+ // altimate_change — the browser reported an explicit failure: the only true `denied` signal
+ entry.reject(markReason(new Error(error), "denied"))
  html(200, HTML_ERROR(error))
  return
  }
@@ -257,7 +280,8 @@ function stopCallbackServer() {
 function registerPending(state: string, timeoutMs = 15 * 60 * 1000): Promise<CallbackResult> {
  return new Promise<CallbackResult>((resolve, reject) => {
  const timeout = setTimeout(() => {
- if (pending.delete(state)) reject(new Error("Timed out waiting for browser sign-in"))
+ // altimate_change — tag as `timeout` for gateway_auth_failed classification
+ if (pending.delete(state)) reject(markReason(new Error("Timed out waiting for browser sign-in"), "timeout"))
  // If the dialog was dismissed and callback() never ran its finally, the
  // loopback server would otherwise stay bound past the timeout. Free the
  // port once nothing is waiting on it.
@@ -286,7 +310,19 @@ export async function AltimateAuthPlugin(_input: PluginInput): Promise<Hooks> {
  label: "Altimate LLM Gateway",
  async authorize() {
  const state = randomBytes(16).toString("hex")
- await startCallbackServer()
+ // altimate_change start — the attempt starts here, before the callback server and the
+ // browser open. startCallbackServer() throws when port 7317 is taken, which is a real
+ // and reasonably common gateway-auth failure that happens before any callback object
+ // exists — without this catch it would never appear in the funnel.
+ const startedAt = Date.now()
+ try {
+ await startCallbackServer()
+ } catch (err) {
+ if (OnboardingTelemetry.isFunnelActive())
+ void OnboardingTelemetry.emit({ type: "gateway_auth_failed", reason: "error" })
+ throw err
+ }
+ // altimate_change end
  // Register the pending flow BEFORE opening the browser so an instant
  // redirect can be matched by state rather than dropped as CSRF.
  const result = registerPending(state)
@@ -321,6 +357,27 @@ export async function AltimateAuthPlugin(_input: PluginInput): Promise<Hooks> {
  // Removed to stop leaking state-bearing authorize URLs into logs.
  await open(authorizeUrl).catch(() => undefined)
 
+ // altimate_change start — onboarding funnel: gateway sign-in started.
+ // Spec name is `gateway_device_code_issued`; this flow is a browser loopback OAuth
+ // with no device code, so the event means "authorize URL built, browser open
+ // attempted". open() failures are swallowed above (the URL is also printed for the
+ // user to paste), so this fires even when no browser actually launched.
+ // The URL is never sent — it carries the CSRF `state`.
+ if (OnboardingTelemetry.isFunnelActive()) void OnboardingTelemetry.emit({ type: "gateway_device_code_issued" })
+
+ // One outcome per attempt. callback() closes over `result` and re-runs its whole body
+ // on every invocation, so a repeated call would otherwise re-emit completion/failure
+ // (and re-report a connect time measured from the original attempt).
+ //
+ // Tri-state rather than a boolean because concurrent invocations race for it and a
+ // plain latch let the WRONG one win: two callbacks exchanging the same one-time token
+ // means one fails fast, claims the latch, and reports gateway_auth_failed while the
+ // other goes on to save valid credentials — the user is connected and the funnel says
+ // they are not. Success is authoritative, so it emits even after a failure was
+ // reported; failure only reports when nothing else has.
+ let outcome: "none" | "failed" | "completed" = "none"
+ // altimate_change end
+
  return {
  url: authorizeUrl,
  instructions: "Complete sign-in in your browser to connect Altimate LLM Gateway.",
@@ -344,11 +401,33 @@ export async function AltimateAuthPlugin(_input: PluginInput): Promise<Hooks> {
  altimateInstanceName: creds.instance,
  altimateApiKey: authToken,
  })
+ // altimate_change start — onboarding funnel: auth succeeded and the instance
+ // is live. The instance name is the customer's tenant identifier and is never
+ // sent. time_to_connect_ms runs from the start of authorize() — before the
+ // callback server and browser open, both of which are part of the wait the user
+ // actually experiences — and lives in this attempt's closure, so a concurrent
+ // attempt cannot overwrite it.
+ if (outcome !== "completed" && OnboardingTelemetry.isFunnelActive()) {
+ outcome = "completed"
+ void OnboardingTelemetry.emit({ type: "gateway_auth_completed" })
+ void OnboardingTelemetry.emit({
+ type: "instance_connected",
+ time_to_connect_ms: Date.now() - startedAt,
+ })
+ }
+ // altimate_change end
  return { type: "success", key: authToken, provider: "altimate-backend" }
  } catch (err) {
  // Log the reason (CSRF / timeout / invalid instance / …). Runs in the
  // server process, so this goes to the log, not the TUI display.
  console.error("[altimate] gateway sign-in failed:", err instanceof Error ? err.message : err)
+ // altimate_change — onboarding funnel: only the classified enum is sent. The
+ // message can embed the instance name (see the invalid-instance throw above),
+ // so it never reaches telemetry.
+ if (outcome === "none" && OnboardingTelemetry.isFunnelActive()) {
+ outcome = "failed"
+ void OnboardingTelemetry.emit({ type: "gateway_auth_failed", reason: reasonOf(err) })
+ }
  return { type: "failed" }
  } finally {
  // Keep the shared server up while another flow is still waiting.