diff --git a/docs/adr/0062-external-datasource-runtime.md b/docs/adr/0062-external-datasource-runtime.md new file mode 100644 index 0000000000..637b9d682f --- /dev/null +++ b/docs/adr/0062-external-datasource-runtime.md @@ -0,0 +1,132 @@ +# ADR-0062: External Datasource Runtime — connection lifecycle, credentials, visibility & query completeness + +**Status**: Proposed — recommended for acceptance; consolidates the runtime gaps surfaced while implementing ADR-0015 federation. Some requirements are already shipped (marked ✅ below); the open decisions are D1–D8 (2026-06-22). + +**Supersedes the runtime portions of**: ADR-0015 §18 addendum (kept as the historical record). ADR-0015 remains the canonical spec/binding decision; this ADR is the canonical *runtime* decision. + +## TL;DR + +ADR-0015 declared external datasource federation. The declaration surface (`schemaMode`, `external.{remoteName,remoteSchema,columnMap,writable}`), the boot validation gate, the write gate, introspection + the runtime "Sync objects" wizard all shipped — and the query path now honors `remoteName`/`remoteSchema` (#2138), `columnMap` (#2149), the federated objects are visible in the datasource admin/meta REST (#2157), and there is a runnable showcase example + dogfood coverage (#2139). + +**But "declare a datasource and it just works" is still not true.** A declared, non-`default` datasource is registered as *metadata* (so it is now *visible*) but is **not connected as a queryable ObjectQL engine driver** — querying its federated objects requires the app to hand-register a live driver via an `onEnable` bridge. Three further runtime concerns are unowned: credential resolution at connect, native-analytics SQL over external objects, and the `columnMap`↔`field.columnName` duplication. + +This ADR defines the **complete external-datasource runtime contract** and the decisions to close those gaps with **one connection mechanism for code- and runtime-origin datasources**, a **backward-compatible, opt-in rollout** (existing managed apps unchanged), and an explicit phase plan. + +## Context + +### What ADR-0015 promised vs. what is live + +| # | Requirement | Status | Where | +|---|---|---|---| +| R1 | Declaration + per-object binding (`schemaMode`, `external.*`) | ✅ shipped | ADR-0015; `spec/data/{datasource,object}.zod.ts` | +| R2 | **Declared datasource → queryable engine driver (no app boilerplate)** | ❌ **gap** | needs `app-plugin`/`standalone` + a connection service | +| R3 | Credentials/secrets resolved at connect (`credentialsRef`) | ⚠️ partial | `SecretBinder`/`ICryptoProvider` exist; not wired into a generic connect path | +| R4 | Visibility in `GET /api/v1/datasources` + `/meta/datasource` + Setup | ✅ shipped | #2157 (in-memory `registerInMemory` fallback) | +| R5a | Read path honors `remoteName`/`remoteSchema` | ✅ shipped | #2138 | +| R5b | Read path honors `columnMap` (remote col ≠ local field) | ✅ shipped | #2149 | +| R5c | Write gate (double opt-in) | ✅ shipped | ADR-0015 §5.3 | +| R5d | Read coercion for external (best-effort, no DDL) | ✅ shipped | #2138 `registerExternalObject` | +| R5e | **Native-analytics SQL over external objects** | ❌ **gap** | analytics service compiles its own `FROM` | +| R6 | Boot schema-validation gate + background drift | ✅ shipped | ADR-0015 §5.2 `external-validation-plugin` | +| R7 | Introspection (`remote-tables`, `object-draft`) + runtime Sync wizard | ✅ shipped | ADR-0015 addendum; `service-datasource` | +| R8 | **Connection lifecycle/health/pooling for N datasources** | ❌ **gap** | only the single `default` driver's lifecycle is managed today | +| R9 | dogfood/verify handles read-only external; canonical example | ✅ shipped | #2139 (`verify` skip + `app-showcase`) | +| R10 | **`columnMap` ↔ `field.columnName` single source of truth** | ❌ **gap** | two inverse mechanisms coexist | + +### The structural gap, precisely (R2) + +`os dev` (standalone) connects exactly **one** driver — the `default` library (`.objectstack/data/standalone.db`), built in `packages/runtime/src/standalone-stack.ts` by detecting the driver kind from the DB URL. A `defineStack({ datasources: [...] })` entry is only registered as metadata by `packages/runtime/src/app-plugin.ts` (`registerInMemory('datasource', …)`). ObjectQL routes queries by **driver name** (`engine.getDriver(object) → this.drivers.get(object.datasource)`) and throws `Datasource 'x' is not registered` when absent. So the only way to make a federated object queryable today is the app's `onEnable` hook calling `ctx.drivers.register(driver)` (= `engine.registerDriver`) — see `examples/app-showcase/src/datasources/external-fixture.ts`. That bridge is framework plumbing leaking into every federation app. + +### Why one consolidating ADR (not N small PRs) + +R2/R3/R8 are the same change seen from three angles — you cannot "auto-connect" without owning **connection lifecycle** (R8) and **credential resolution** (R3). R4 (just shipped) and R2 are two faces of "declared datasource as a runtime first-class citizen" — visible *and* usable. And the connection mechanism must **converge** the code-origin path (this ADR) with the runtime-origin path (the Sync wizard already builds drivers via the injected factory for test/introspection) so we don't ship two divergent connect paths. These decisions share contracts (the injectable driver factory, the `SecretBinder`, boot ordering vs. the `kernel:ready` validation gate); deciding them piecemeal risks incoherent seams. + +## Goals / Non-goals + +**Goals.** A declared external datasource is, with no app code: visible (R4 ✅), connected + queryable (R2), credential-resolved (R3), validated at boot (R6 ✅), and lifecycle-managed (R8) — through **one** mechanism shared by code- and runtime-origin datasources, **without changing the behavior of existing managed apps**. + +**Non-goals.** New driver dialects; replacing `DataSync`/`ExternalLookup` (ADR-0015 §9 coexistence stands); cross-datasource JOINs in one query (federated objects are queried per-datasource); multi-tenant credential vaulting beyond the existing `SecretBinder` seam. + +## Decision + +### D1 — One `DatasourceConnectionService`; declared datasources auto-connect + +Introduce a single service that, given a datasource definition, builds a driver via the **injected** driver factory (the same `createDefaultDatasourceDriverFactory` used by the runtime-admin path), connects it, and registers it into the ObjectQL engine under the datasource name. `app-plugin` calls it for every declared datasource (in addition to today's `registerInMemory` for visibility); `standalone-stack`'s single-`default`-driver bootstrap is refactored to go through the same service so there is exactly one "definition → live driver" code path. `engine.registerDriver` remains the sink. The `onEnable` bridge becomes unnecessary for the common case (kept as an escape hatch — D8). + +### D2 — Connect is opt-in-safe: existing managed apps are byte-for-byte unchanged + +Auto-connect must not change apps that today declare datasources that are *decorative* or routed via `datasourceMapping` (e.g. `examples/app-crm`'s `crm_primary`/`crm_analytics`). Gate auto-connect so a declared datasource is only connected when it is meaningfully addressed: **(a)** it is `external` (`schemaMode !== 'managed'`), or **(b)** an object/`datasourceMapping` actually routes to it, or **(c)** it sets an explicit `autoConnect: true`. A managed datasource that nothing routes to stays metadata-only (today's behavior). The `default` datasource keeps its current dedicated bootstrap. This is the load-bearing backward-compat decision. + +### D3 — Credentials resolved at connect via `SecretBinder`/`ICryptoProvider` + +`DatasourceConnectionService` resolves `external.credentialsRef` (and any `secret` config fields) through the host-provided `SecretBinder` over `ICryptoProvider` **before** building the driver. Open-core default is `InMemoryCryptoProvider`; a datasource that needs a secret the host cannot decrypt **fails closed** (clear boot error, datasource left unconnected — not a silent skip). This reuses the exact mechanism the runtime-admin "Add Datasource" wizard already uses, so code- and runtime-origin secrets converge. + +### D4 — Visibility converges on the metadata registry (shipped, ratified here) + +Code-defined datasources surface in `GET /api/v1/datasources`, `GET /api/v1/meta/datasource` and **Setup → Datasources** via the single in-memory metadata registry, stamped `origin: 'code'` (read-only in the admin UI). Shipped in #2157 (the in-memory metadata fallback was missing `registerInMemory`, silently skipping registration on the host-config boot path). This ADR ratifies that the metadata registry is the single source for datasource listing across all boot paths; the admin `listDatasources` and `protocol.getMetaItems` both read it. + +### D5 — Lifecycle, health & ordering for N datasources + +`DatasourceConnectionService` owns connect/disconnect (graceful shutdown), pool config per datasource, and an optional health probe surfaced in the admin list (`status`). **Ordering**: all declared datasources connect **before** the `kernel:ready` external-validation gate (ADR-0015 §5.2) and before first query — i.e. during plugin init/start, not in a `kernel:ready` handler. Connect failure policy is **fail-fast for `external` with `validation.onMismatch: 'fail'`**, **degrade-with-warning** otherwise (a connectivity blip on an optional analytics replica should not brick boot). + +### D6 — Native-analytics SQL honors the remote table/columns + +The analytics native-SQL strategy compiles its own `FROM "