diff --git a/docs/qa/platform-checklist/FOLLOW-UPS.md b/docs/qa/platform-checklist/FOLLOW-UPS.md index 12e285bd90..21274bd319 100644 --- a/docs/qa/platform-checklist/FOLLOW-UPS.md +++ b/docs/qa/platform-checklist/FOLLOW-UPS.md @@ -1,4 +1,8 @@ -# Follow-ups — open items from the capability-coverage sweep (2026-08-08) +# Follow-ups — open items from the capability-coverage sweeps + +Standing decision register, one section per sweep (append, never rewrite — a resolved +row stays with its resolution so the next sweep can see what was already decided). +Sections §1–§4 are the **2026-08-08** sweep; §5 is the **2026-08-17** re-audit (R4). Decision register from the capability-coverage sweep. The gap items found by the five-angle sweep have all been authored into `areas/*.json` (checklist grew 84 → 170 @@ -67,3 +71,68 @@ these to the showcase would make them runnable: - The checklist itself (`areas/*.json`, `coverage.json`, `README.md`, `RUNNER.md`, `scripts/check-platform-checklist.mjs`) ships in this branch; this file carries the decisions that remain with the maintainer. + +## 5. Sweep 2026-08-17 (R4) — re-audit against the window since the 2026-08-08 ledger + +Ledger 182 → 190 items; `coverage.json` 28 mapped / 2 waived → **30 mapped / 0 waived**. + +### 5a. Docs drift (PD#10 class) + +| # | drift | evidence | captured in | +|---|---|---|---| +| E1 | **Stale defect note in `packages/spec/liveness/doc.json`.** Its `_note` records "DocSchema declares no `tags`, yet the book-side `include: { tag }` rule and the REST corpus both expect one — the tag rule can currently never match". That defect is **fixed**: `DocSchema` now declares `tags` (`packages/spec/src/system/doc.zod.ts:126`, with the history spelled out in the surrounding comment). The ledger note now describes a bug that no longer exists, which is the same failure class this sweep is correcting in `SWEEP.md`. Outside this card's file surface (`docs/qa/platform-checklist/**`), so it is reported, not edited. | `packages/spec/liveness/doc.json` `_note` vs `packages/spec/src/system/doc.zod.ts:111-127` | — (liveness ledger prose, not a checklist item) | + +### 5b. Checked and CLEAN (recorded so the next sweep does not re-derive it) + +- **The blank template's `dev` script omits `--ui`, and that is CORRECT — the console is + served anyway.** This sweep first read `"dev": "objectstack dev"` against the + quick-start's `npx os dev --ui` and inferred that a newcomer running `npm run dev` + would land on a server with no console. **Source-checking the chain refuted it**, and + the refutation is recorded here because the inference is an easy one to make twice: + - `packages/cli/src/commands/serve.ts:221` — `ui: Flags.boolean({ …, default: true, + allowNo: true })`. The console is **default-ON** at `serve`; `--no-ui` is the off + switch. + - `packages/cli/src/commands/dev.ts:370` — `...(flags.ui ? ['--ui'] : [])`. `dev` only + ever **adds** `--ui`; it never forwards `--no-ui`. With `dev.ts:69` declaring `ui` + with no `default`, an unflagged `dev` spawns `serve` with no ui flag at all, so + serve's own default takes over — on. + - `content/docs/deployment/cli.mdx:139` says it outright: "`--ui` | Force Console UI on + (**already on by default in dev**)", and `:196` documents `--ui / --no-ui … (default + on)`. + + So `--ui` on `dev` is a **no-op forwarder**, there is no divergence between the + template script and quick-start, and the newcomer is not stranded. What made the wrong + reading tempting is `dev.ts:69`'s own flag description ("Enable the bundled Console + portal at /_console/"), which reads like the mechanism when it is only a forwarder — + not worth a change on its own, but worth knowing before inferring from it. + `cli.scaffold-console-first-paint` clause 3 now asserts the two invocations **agree**, + with a difference between them as the failure. +- **No published doc prescribes a retired scaffolder template.** Grepped all of + `content/docs/` for `todo` / `compliance` / `content` / `contracts` / `procurement` as + `-t` / `--template` operands: zero hits. Every documented invocation is a bare + `npx create-objectstack my-app`, which resolves to `blank`, the only entry in + `TEMPLATES`. The delisting is cleanly reflected on the docs side. +- **`npm run validate` and `npx os validate` are the same binary** (the blank template's + `validate` script is `objectstack validate`), so the spelling difference between + `cli.scaffold-first-run` and the quick-start is not a divergence. +- **The first-run *commands* are covered** by `cli.scaffold-first-run` (P1, CI-automated) + and `studio-authoring.first-run-loop` (P0). Only the seam between them was open, and + that is now `cli.scaffold-console-first-paint`. No duplicates were authored. + +### 5c. For the maintainer + +- **Both surviving coverage waivers were stale** (`book`, `doc`) and are retired in this + PR — see `SWEEP.md` "Discipline". Running total: 6 of 6 waivers ever written turned out + stale. Worth considering whether a waiver should carry a mandatory re-audit date. +- **Cost figures in the corrected `SWEEP.md` sentence are this run's own measurements**, + which came out higher than the ones supplied with the card (validator ~0.12–0.18 s vs + ~80 ms; `pnpm check:platform-checklist` ~2.9 s wall vs 0.165 s). The container was under + load from a parallel cold monorepo install, and the `pnpm` wrapper's startup dominates + the wall figure. The corrected text therefore records the *direct-node* cost (~0.25 s + for self-test + validator) and explicitly warns that timing through `pnpm` measures the + wrapper — a durable statement rather than a number that goes stale. +- **No security-sensitive finding to withhold from this PR.** The two highest-severity + items authored (`access-security.no-active-org-session-semantics`, + `integration-system.datasource-credential-refusal-matrix`) assert guards that are + already shipped and already public in their ADRs/issues; nothing here discloses an + unfixed hole. diff --git a/docs/qa/platform-checklist/SWEEP.md b/docs/qa/platform-checklist/SWEEP.md index 64e0ffe5e3..a6148c35ea 100644 --- a/docs/qa/platform-checklist/SWEEP.md +++ b/docs/qa/platform-checklist/SWEEP.md @@ -6,11 +6,28 @@ executable by the AI session itself. Expected cadence: before each major release after any large platform surface lands. This is the AI-participation half of keeping the checklist current. The other half is -automatic and needs no human at all: `scripts/check-platform-checklist.mjs` (CI, every -PR) fails when a new metadata **kind** is unmapped (coverage ratchet) or a spec **enum** -grows a value a matrix item was pinned against (`enumSource` freshness ratchet). Those -catch drift on the PR that causes it. This sweep catches the harder class — a whole -surface or behavior nobody wrote an item for — which no deterministic gate can find. +**mechanical but still operator-triggered**: `scripts/check-platform-checklist.mjs` fails +when a new metadata **kind** is unmapped (coverage ratchet) or a spec **enum** grows a +value a matrix item was pinned against (`enumSource` freshness ratchet). + +⚠️ Those ratchets are **NOT wired into per-PR CI** — by maintainer decision they run on a +periodic **manual** cadence (see this dir's [README.md](./README.md) "Operating cadence", +and the recorded note at `.github/workflows/lint.yml:426`). So they do **not** catch drift +on the PR that causes it; they catch it at the **next manual run**. Do not read a green +ledger as "drift is already gated" — between runs, an unmapped kind or a grown enum sits +undetected on `main`, which is precisely why a sweep must re-derive from source rather +than trust the last green check. Cost is not the reason it stays out of CI: measured +2026-08-17 on a 190-item ledger, the validator alone is ~0.12–0.18 s and the full +`pnpm check:platform-checklist` (selector self-test + validator) is ~0.25 s of actual +work, zero-dependency and token-free — the `pnpm` wrapper's own startup dominates any +figure you measure through it, so time the scripts directly if the number matters. The +reason is PR independence: the checklist is a QA ledger, not a code gate, so an unrelated +PR is never blocked by checklist drift. + +This sweep catches the harder class — a whole surface or behavior nobody wrote an item +for — which no deterministic gate can find. Note what that leaves uncovered even in +principle: **new behavior inside an existing kind** is invisible to BOTH ratchets (no new +kind, no new enum member), so it is found only here. ## What a sweep is @@ -58,6 +75,14 @@ single reader would not have covered. corrected them against source rather than parroting. - **Stale waivers are the highest-value find.** Four of six coverage waivers turned out false (api/datasource/mapping/hook all ship stock fixtures). Re-audit every waiver - each sweep — a waiver is a claim that ages. + each sweep — a waiver is a claim that ages. **Confirmed again 2026-08-17**: the two + waivers that survived that sweep (`book`, `doc`) were *also* false. Both claimed the + kinds are "display-only, no independent runtime behavior to gate beyond serving"; the + REST read layer in fact runs a three-member audience vocabulary with an ADR-0049 + fail-closed branch, a two-layer gate (book audience, then per-entry doc-audience + union), a deliberate orphan-group non-leak rule, and locale resolution — an + access-control surface. `coverage.json` now carries **zero** waivers. Running total: + **6 of 6** waivers ever written turned out stale, which is the strongest argument in + this file for auditing them against the runtime rather than against their own prose. - **Defects found while grounding go to FOLLOW-UPS.md as expected-fail probes**, not silent passes; security-sensitive ones are not filed publicly without the maintainer. diff --git a/docs/qa/platform-checklist/areas/access-security.json b/docs/qa/platform-checklist/areas/access-security.json index 0ca3f8e07d..e72ef052f0 100644 --- a/docs/qa/platform-checklist/areas/access-security.json +++ b/docs/qa/platform-checklist/areas/access-security.json @@ -1910,6 +1910,115 @@ "ref": "claude/issue-7680-package-capability-doctrine" } ] + }, + { + "id": "access-security.no-active-org-session-semantics", + "title": "A session with no active organization is a LEGAL state with fail-closed semantics: tenant-scoped reads resolve to nothing, tenant-scoped writes are refused loudly naming the missing org, and nothing is ever NULL-stamped", + "since": "v17", + "status": "active", + "revision": 1, + "priority": "P0", + "surface": "api", + "personas": [ + "a freshly signed-up user on their FIRST session (structurally predates their sys_member row)", + "a user removed from their organization", + "seeded admin (control — has an active organization)" + ], + "fixtures": { + "app": "showcase", + "requires": [ + "a tenancy-walled deployment (ADR-0095 Layer 0 active on at least one tenant-scoped object) — confirm the posture before scoring, since on an unwalled deployment every clause here is vacuous", + "the ability to complete a signup and use the resulting first session before the membership reconciler settles, OR to remove a member and reuse their live session" + ], + "knownGaps": [ + "The state is timing-derived on the signup path (better-auth defers the membership write past the signup transaction, ADR-0093), so reproducing it by racing signup is flaky. The DURABLE reproduction is the removed-member path, or a multi-organization deployment whose reconciler binds nobody (ADR-0093 D1 `no-target-org`). Record WHICH producer the run used — a clause proven only on a racy producer is weaker evidence and must say so.", + "No stock showcase fixture is known to configure `invite-only` or SSO-JIT-pending; those two producers are blocked(fixture) unless the run configures them." + ] + }, + "steps": [ + "establish the state and PROVE it: obtain a session whose `activeOrganizationId` is null, and cite the evidence (session row / session endpoint payload) — do not infer the state from the symptom you are about to test", + "read a tenant-scoped object over the data API with that session", + "write (create) a row on a tenant-scoped object with that session", + "immediately read back, as the SAME caller, anything the write may have produced", + "query the underlying table as admin for rows whose tenant column is NULL, before and after the write attempt", + "repeat the read and the write against a NON-tenant-scoped object to bound the blast radius", + "sign in and out with the state in force, then inspect the audit ledger rows the session produced (plugin-audit auth-session-audit login/logout)", + "run the same read+write as the control admin persona to show the refusal is state-derived, not a blanket denial" + ], + "acceptance": [ + { + "clause": "tenant-scoped READS resolve to nothing — an empty result set, NOT an error: Layer 0's deny sentinel (RLS_DENY_FILTER) yields zero rows and a 200", + "oracle": "api", + "verify": "GET returns 200 with an empty collection; a 4xx/5xx here is a FAIL (the read side is deliberately not an error path)", + "evidence": "status + body" + }, + { + "clause": "tenant-scoped WRITES are refused LOUDLY with a 4xx whose message NAMES the missing active organization — not a generic denial, not a 500", + "oracle": "api", + "verify": "status is 4xx; the error `code` is drawn from the CLOSED ADR-0112 vocabulary (an invented code is a FAIL of D3) and the message text names the missing active organization", + "evidence": "status + full error envelope, quoted" + }, + { + "clause": "⛔ NO silent NULL stamping, anywhere: the refused write leaves NO row behind, and no subsystem writes a row whose tenant column is NULL because the caller had no organization to supply", + "oracle": "api", + "verify": "the admin-side NULL-tenant row count is IDENTICAL before and after the attempt; a row that appears and is invisible to its own creator is the #8208 defect and is a P0 FAIL", + "evidence": "both row counts + the diff" + }, + { + "clause": "the refusal does not fall OPEN in the other direction: an org-scoped permission holder with no active organization reads and writes NOTHING across tenants (the #8158 class — `scope: 'org'` grants nothing when there is no org)", + "oracle": "api", + "verify": "as the stateless caller holding an org-scoped permission (e.g. manage_sharing), attempt a cross-tenant read and write; both must be empty/refused, never another tenant's rows", + "evidence": "responses + a cross-check that another tenant's rows exist and were not returned" + }, + { + "clause": "the audit carve-out is paid for by ORDERING, not by an exemption: audit rows from a first session carry a real tenant — no audit row is NULL-tenanted and therefore permanently invisible to RLS readers (#8245)", + "oracle": "api", + "verify": "read the login/logout audit rows the session produced as a tenant-scoped reader; they must be visible. A row present in the raw table but invisible to every RLS reader is a FAIL even though nothing errored", + "evidence": "raw rows + the RLS-scoped read of the same rows" + }, + { + "clause": "the state is LEGAL, not a crash: the session authenticates, non-tenant-scoped surfaces still work, and nothing 500s", + "oracle": "api", + "verify": "the session's whoami/session endpoint answers 200; a read on a non-tenant-scoped object succeeds; no 5xx anywhere in the run", + "evidence": "the response set" + }, + { + "clause": "three subsystems agree — sharing, the Layer 0 wall + write path, and the audit ledger give the SAME answer for the SAME state (this item exists because they once gave three contradictory ones on the same working day)", + "oracle": "api", + "verify": "the run record states each subsystem's observed behavior side by side; any disagreement is the finding, whichever way it leans", + "evidence": "the three-way comparison table" + } + ], + "negative": [ + "fell OPEN: an org-scoped holder with no active organization reading or writing another tenant's rows (#8158) — P0", + "fell CLOSED SILENTLY: an HTTP-created record stamped organization_id NULL and immediately invisible to its own creator (#8208) — P0, and the reason the NULL-count clause is measured rather than inferred", + "wrote INVISIBLE rows: audit rows carrying a NULL tenant, permanently unreadable through RLS (#8245)", + "a 500 or an unhandled rejection on any of the three paths — the state is declared legal, so crashing is a defect", + "a refusal whose message does NOT name the missing active organization: it is indistinguishable from an ordinary permission denial and sends the debugger to the wrong layer", + "an error code invented for this refusal instead of drawn from the ADR-0112 closed vocabulary" + ], + "traps": ["auth-state-leak", "eventual-consistency"], + "source": [ + "ADR-0123 (an authenticated session with no active organization — reads resolve to nothing, tenant-scoped writes refused loudly), accepted 2026-08-13", + "ADR-0095 D1/W1/W2 (Layer 0 independent always-first AND-composed tenant filter)", + "ADR-0105 D1/D2 (posture spectrum, group union wall, empty/absent scope → deny)", + "ADR-0093 D1/D2 (membership reconciler; session.create.before resolving activeOrganizationId from sys_member)", + "ADR-0112 D3 (closed error-code vocabulary this refusal draws from)", + "ADR-0049 (a declared wall the write path does not meet is the defect)", + "#8247 (the ruling), from #8158 (fell open) · #8208 (fell closed silently) · #8245 (wrote invisible rows)", + "PR #8471 (ADR-0123 + enforcement)", + "@objectstack/plugin-security security-plugin.ts step 3.7 + tenant-layer.ts (computeTenantLayer0Filter, RLS_DENY_FILTER)", + "@objectstack/plugin-auth auth-manager.ts composeDatabaseHooks + reconcile-membership.ts", + "@objectstack/plugin-audit auth-session-audit.ts (login/logout rows)" + ], + "history": [ + { + "revision": 1, + "date": "2026-08-17", + "change": "new — the ADR-0123 tenant-semantics wall had ZERO checklist coverage. Neither ratchet could see it: no new metadata kind (it is new behavior across the already-mapped permission/object kinds) and no new enum member. Authored P0 because the ADR records three independently measured production-shaped defects in three subsystems for one state, two of which are silent (a row invisible to its own creator; audit rows invisible to every RLS reader) and therefore invisible to any test that only asserts 'no error'", + "ref": "#9299" + } + ] } ] } \ No newline at end of file diff --git a/docs/qa/platform-checklist/areas/api-backend.json b/docs/qa/platform-checklist/areas/api-backend.json index 0c00890dbc..2ce5e328d8 100644 --- a/docs/qa/platform-checklist/areas/api-backend.json +++ b/docs/qa/platform-checklist/areas/api-backend.json @@ -1037,6 +1037,272 @@ "ref": "claude/platform-test-checklist-ocwugl" } ] + }, + { + "id": "api-backend.filter-comparand-conformance", + "title": "Filter comparand conformance: only the accepted comparand TYPES pass the where-door, a bigint past the exact-integer limit is refused, and a dotted head is classified rather than guessed", + "since": "v17", + "status": "active", + "revision": 1, + "priority": "P1", + "surface": "api", + "personas": ["seeded admin"], + "fixtures": { + "app": "showcase", + "requires": [ + "an object with a scalar field, a relation (lookup/master-detail) field, and — if the showcase ships one — a formula/virtual field, so all three dotted-head classes are reachable" + ], + "knownGaps": [ + "If no formula/virtual field exists on a filterable object in the stock seeds, the `virtual` head class is blocked(fixture) — record it rather than scoring the class from the relation case." + ] + }, + "steps": [ + "read ACCEPTED_FILTER_COMPARAND_TYPES in packages/spec/src/data/filter-comparand-type.ts and enumerate the current members — the variants matrix below is pinned to that export, so a mismatch is a validator failure, not a judgement call", + "for EACH accepted comparand type, issue a data-API read whose `where` carries a comparand of that type against a compatible field, and record the answer set", + "issue a read whose comparand is a type NOT in the accepted set (e.g. a plain object, an array where the operator takes a scalar, undefined, a function-shaped value) and capture the refusal", + "probe the bigint boundary: a bigint comparand at FILTER_COMPARAND_BIGINT_EXACT_LIMIT (2n**53n) and one past it", + "issue reads with dotted filter heads of each class — a relation head, a virtual/formula head, and a scalar head (a dot on a field that is not traversable) — and record how each is classified", + "cross-check the SAME filters through a second door (the engine/objectql path or a view filter) so the refusal is proven at the contract, not at one transport" + ], + "acceptance": [ + { + "clause": "every member of ACCEPTED_FILTER_COMPARAND_TYPES is accepted at the where-door and returns a known answer set — the matrix is exhaustive over the pinned export, not a sample", + "oracle": "api", + "verify": "one request per member; each returns 200 and the answer set reconciles against a directly-computed expectation", + "evidence": "per-member request + answer set" + }, + { + "clause": "a comparand whose type is NOT in the accepted set is REFUSED with a located, prescriptive error — never coerced, never silently dropped, never answered as if the clause were absent", + "oracle": "api", + "verify": "status is 4xx and the message names the offending path and the accepted set (ACCEPTED_FILTER_COMPARAND_TYPES_SENTENCE exists precisely so the refusal can quote it); an unrefused bad comparand that returns 200 with a FULL table is the dangerous direction — a dropped predicate silently widens the result set", + "evidence": "status + message + the returned row count vs the unfiltered count" + }, + { + "clause": "the bigint exact-integer limit is enforced at the boundary: a value inside the limit is accepted, one past it is refused rather than silently losing precision", + "oracle": "api", + "verify": "both requests; the past-limit case must refuse — a 200 whose comparand was rounded is a silent-wrong-answer FAIL", + "evidence": "both responses" + }, + { + "clause": "a dotted filter head is CLASSIFIED (relation | virtual | scalar | null) and each class is handled per contract — a scalar head with a dot is refused rather than treated as a relation traversal", + "oracle": "api", + "verify": "one request per reachable class; classifyDottedFilterHead in packages/spec/src/data/filter-dotted-head.ts is the contract, and the observed behavior must match its classification", + "evidence": "per-class request + observed handling" + }, + { + "clause": "the refusal is a CONTRACT-level refusal, not a transport quirk: the same bad comparand is refused through a second door with the same code", + "oracle": "api", + "verify": "compare the two doors' error codes; a comparand refused at REST but accepted through the engine is a FAIL of the single-contract property", + "evidence": "both refusals side by side" + } + ], + "negative": [ + "a rejected predicate that is DROPPED rather than refused — the request answers 200 over an unfiltered set, which reads as success and silently leaks every row the filter was supposed to exclude (the load-bearing negative for this item)", + "a comparand coerced across types (string '5' silently becoming number 5) where the contract refuses it — coercion at the filter door is how an authored filter stops meaning what it says", + "a past-limit bigint answered with a rounded comparand", + "a dotted head on a scalar field traversed as if it were a relation" + ], + "variants": [ + "string", + "number", + "bigint", + "boolean", + "null", + "Date" + ], + "enumSource": { + "file": "packages/spec/src/data/filter-comparand-type.ts", + "export": "ACCEPTED_FILTER_COMPARAND_TYPES", + "expect": 6 + }, + "traps": ["silent-coercion"], + "source": [ + "packages/spec/src/data/filter-comparand-type.ts (ACCEPTED_FILTER_COMPARAND_TYPES, ACCEPTED_FILTER_COMPARAND_TYPES_SENTENCE, FILTER_COMPARAND_BIGINT_EXACT_LIMIT, isAcceptedFilterComparand, normalizeFilterComparandTypes)", + "packages/spec/src/data/filter-comparand-shape.ts", + "packages/spec/src/data/filter-comparand-type-conformance.ts", + "packages/spec/src/data/filter-dotted-head.ts (classifyDottedFilterHead, DottedFilterHeadClass)", + "packages/spec/src/data/filter-subtree-provenance.ts", + "packages/spec/src/migrations/entries/semantic/18.engine-dotted-filter-refused.ts", + "packages/spec/src/migrations/entries/semantic/17.engine-dotted-projection-refused.ts", + "packages/spec/src/migrations/entries/semantic/17.engine-find-formula-filter-refused.ts" + ], + "history": [ + { + "revision": 1, + "date": "2026-08-17", + "change": "new — the comparand-typing contract landed as a cluster of seven spec modules plus three semantic migrations at 17.0.0-rc.6 and had no item. api-backend.query-contract-matrix covers OPERATORS ('every filter operator gives known answers') but says nothing about the TYPE of the right-hand side, which is where the silent-widening failure lives. Variants pinned to ACCEPTED_FILTER_COMPARAND_TYPES so a seventh accepted type flags the matrix stale", + "ref": "#9299" + } + ] + }, + { + "id": "api-backend.date-range-preset-matrix", + "title": "Every date-range preset resolves to its declared macro window, and a bare preset name used as a raw comparand is refused with the prescription", + "since": "v17", + "status": "active", + "revision": 1, + "priority": "P2", + "surface": "api", + "personas": ["seeded admin"], + "fixtures": { + "app": "showcase", + "requires": [ + "an object with a date/datetime field carrying rows spread across the preset windows (today, this week, last quarter, last 90 days) so each preset has a distinguishable answer set" + ], + "knownGaps": [ + "Stock seeds are unlikely to place rows in every one of the 13 windows. Presets whose window contains no seeded row prove only that the window RESOLVED, not that it selected correctly — the run must record which presets were answer-discriminating and which were merely well-formed. Seeding rows at known offsets during the run is the honest way to close this; record it if done.", + "Presets are relative to run time, so a run near a period boundary (month/quarter/year rollover, or a run spanning midnight) can legitimately shift an answer set. Capture the run's clock alongside the results." + ] + }, + "steps": [ + "read DATE_RANGE_PRESETS and DATE_RANGE_PRESET_MACRO_WINDOWS in packages/spec/src/data/date-range-presets.ts and enumerate the current members", + "record the run's wall clock and timezone BEFORE issuing any request — every expectation below is derived from it", + "for EACH preset, issue a data-API read filtered by that preset against the date field and capture the answer set", + "for each preset, independently compute the expected window from DATE_RANGE_PRESET_MACRO_WINDOWS and the recorded clock, then reconcile the answer set against a direct comparison-operator query over that literal window", + "issue a read whose comparand is the BARE preset name as a raw string (e.g. where: { created: 'today' }) and capture the refusal", + "issue a read with an unknown preset name and capture the refusal" + ], + "acceptance": [ + { + "clause": "every preset resolves and returns the SAME answer set as an explicit literal-window query computed from the same clock — the preset is a macro, not an approximation", + "oracle": "api", + "verify": "per preset: preset query answer set === literal-window query answer set; any divergence is a FAIL naming the preset", + "evidence": "per-preset answer-set pair" + }, + { + "clause": "the matrix is exhaustive over DATE_RANGE_PRESETS — all pinned members exercised, none sampled away", + "oracle": "api", + "verify": "the run record lists one row per pinned member; a member with no row is blocked, not passed", + "evidence": "the 13-row table" + }, + { + "clause": "a BARE preset name used as a raw comparand is REFUSED with the prescription telling the author the correct spelling — not treated as a literal string comparison that silently matches nothing", + "oracle": "api", + "verify": "status 4xx and the message matches bareDateRangePresetComparandMessage; a 200 with zero rows is the FAIL mode (it is indistinguishable from a correct empty answer)", + "evidence": "status + message" + }, + { + "clause": "an unknown preset name is refused rather than resolving to an empty or unbounded window", + "oracle": "api", + "verify": "status 4xx; specifically NOT a 200 over the full table, which would be an unbounded window", + "evidence": "status + returned row count vs unfiltered count" + } + ], + "negative": [ + "a bare preset name answered as a literal string comparison returning 200 + zero rows — silently correct-looking and completely wrong", + "an unknown preset resolving to an unbounded window (full table) instead of refusing", + "two presets returning identical answer sets where the seeded data should distinguish them (e.g. this_month === last_month) — a sign the window is not actually being computed" + ], + "variants": [ + "today", + "yesterday", + "this_week", + "last_week", + "this_month", + "last_month", + "this_quarter", + "last_quarter", + "this_year", + "last_year", + "last_7_days", + "last_30_days", + "last_90_days" + ], + "enumSource": { + "file": "packages/spec/src/data/date-range-presets.ts", + "export": "DATE_RANGE_PRESETS", + "expect": 13 + }, + "traps": ["clock-skew", "timezone-boundary"], + "source": [ + "packages/spec/src/data/date-range-presets.ts (DATE_RANGE_PRESETS, DATE_RANGE_PRESET_MACRO_WINDOWS, isDateRangePresetName, bareDateRangePresetComparandMessage)", + "packages/spec/src/migrations/entries/semantic/18.filter-preset-ordering-comparand-refused.ts", + "packages/spec/src/data/temporal-comparand.ts + packages/core/src/utils/temporal-comparand.ts", + "packages/objectql/src/temporal-comparand-door.ts" + ], + "history": [ + { + "revision": 1, + "date": "2026-08-17", + "change": "new — DATE_RANGE_PRESETS landed at 17.0.0 (2026-08-15) with 13 members and no matrix item; the enum is exported as a const array so it is enumSource-pinnable, closing it against a 14th preset landing unnoticed. The bare-preset-comparand refusal is authored as its own clause because the spec ships a dedicated message for it, which means the failure mode was observed rather than imagined", + "ref": "#9299" + } + ] + }, + { + "id": "api-backend.retired-def-refusal", + "title": "A retired metadata DEFINITION (whole schema type withdrawn) refuses at every authoring door — distinct from a retired key, and the door that must not fall through to 'unknown type, ignored'", + "since": "v17", + "status": "active", + "revision": 1, + "priority": "P1", + "surface": "build", + "personas": ["seeded admin", "build-time author (no session)"], + "fixtures": { + "app": "showcase", + "requires": [ + "a scratch authoring source tree (os build target) plus a live boot for the metadata-write door" + ], + "knownGaps": [ + "Some retired defs were removed by ADR-0087 Route 3 ('no tombstone, no D2 conversion — this table plus the D3 semantic entry are the declaration', e.g. data/ExternalDataSource per #8075). For those, the correct behavior may be the GENERIC unknown-type refusal rather than a def-specific prescription. The run must read each sampled entry's route BEFORE deciding what a pass looks like — scoring a Route-3 def against a prescription it was never given is a false fail, and scoring it against 'anything 4xx' is a false pass." + ] + }, + "steps": [ + "enumerate packages/spec/src/migrations/entries/retired-defs/ and record the count and the version prefixes present (the registry currently carries entries at both the current major and the next)", + "sample at least four entries spanning BOTH version prefixes and at least three different domains (e.g. data/, system/, identity/, ui/)", + "for each sampled def, read its entry file and record which ADR-0087 route retired it and whether a prescription was authored", + "author a stack that declares metadata of each sampled retired type and run os build", + "attempt the same through the runtime metadata-write door (POST/PUT /meta/) against a live boot", + "as a control, author metadata of a type that was NEVER declared (a typo'd/invented type name) and compare its refusal to the retired-def refusals" + ], + "acceptance": [ + { + "clause": "every sampled retired definition is REFUSED at os build — non-zero exit, never a green build that silently drops the whole metadata family", + "oracle": "build", + "verify": "exit code non-zero per sampled def; a green build with the family absent from the artifact is the P1 FAIL (it is the shape that debugs from the far end)", + "evidence": "exit codes + the build output per def" + }, + { + "clause": "the refusal is DISTINGUISHABLE from the generic unknown-type refusal wherever a prescription was authored — an author who used a real, now-retired type is told what replaced it, not merely that the type is unknown", + "oracle": "build", + "verify": "diff each retired-def refusal against the control (invented type) refusal; for entries whose ADR-0087 route authored a prescription, the texts must differ and the retired one must name the successor. For Route-3 entries with no prescription by design, an identical generic refusal is the CORRECT answer — record which route each sampled def took", + "evidence": "the refusal texts side by side + the route per def" + }, + { + "clause": "the runtime metadata-write door refuses the same types as the build door — a type refused at build that a live POST /meta accepts is a two-door disagreement and a FAIL", + "oracle": "api", + "verify": "status per sampled type at the runtime door; compare with the build verdict", + "evidence": "the per-door verdict table" + }, + { + "clause": "the sample spans both version prefixes present in the registry, so the mechanism is proven for the retirements already staged for the NEXT major and not only for those of the shipped one", + "oracle": "build", + "verify": "the run record lists each sampled def with its version prefix; a single-prefix sample does not satisfy this clause", + "evidence": "the sample table" + } + ], + "negative": [ + "a retired definition that parses CLEAN and is silently dropped from the artifact — the whole-family version of the silent-strip defect, and worse than the key-level case because an entire metadata family goes missing", + "a retired definition refused with a bare 'unknown type' where a prescription WAS authored — the author is sent to look for a typo instead of to the successor", + "a build refusal that the runtime metadata-write door does not honor (or the reverse)", + "a refusal that names the wrong successor, which is worse than naming none" + ], + "source": [ + "ADR-0087 (retirement registries and conversion routes)", + "ADR-0049 (enforce-or-remove)", + "packages/spec/src/migrations/entries/retired-defs/ (the registry this item samples)", + "packages/spec/src/migrations/entries/semantic/ (the D3 declarations that accompany Route-3 removals, e.g. 17.external-lookup-message-queue-families-retired, 18.identity-api-key-schema-retired)", + "#8075 (the external-lookup census whose fork (b) retired data/ExternalDataSource, ExternalFieldMapping, ExternalLookup whole)", + "sibling item api-backend.enforce-or-remove-authoring-gates (retired KEYS — this item is the DEFINITION-level door it does not reach)" + ], + "history": [ + { + "revision": 1, + "date": "2026-08-17", + "change": "new — api-backend.enforce-or-remove-authoring-gates samples the retired-KEY registries and explicitly requires three key-level surfaces, but nothing exercised the retired-DEF registry, which is a materially different door: a withdrawn KEY loses one property, a withdrawn DEFINITION loses an entire metadata family, and the fall-through failure mode ('unknown type, ignored') is different from the strip-mode failure the key item guards. The registry has grown substantially since the ledger was built and now carries entries for the next major as well as the current one", + "ref": "#9299" + } + ] } ] } \ No newline at end of file diff --git a/docs/qa/platform-checklist/areas/cli.json b/docs/qa/platform-checklist/areas/cli.json index d0c5277f20..ac5fa89ce8 100644 --- a/docs/qa/platform-checklist/areas/cli.json +++ b/docs/qa/platform-checklist/areas/cli.json @@ -645,6 +645,182 @@ "history": [ { "revision": 1, "date": "2026-08-07", "change": "new item: flag/command error UX with the full registered command surface enumerated from src/commands/ as variants, and never-executed-anyway as the load-bearing negative", "ref": "claude/platform-test-checklist-ocwugl" } ] + }, + { + "id": "cli.migrate-meta-codemod", + "title": "os migrate meta actually rewrites authored sources across a spec major — the prescription hundreds of retirement messages point at is honest, idempotent, and refuses below its floor", + "since": "v14.7", + "status": "active", + "revision": 1, + "priority": "P1", + "surface": "cli", + "personas": ["build-time author (no session)"], + "fixtures": { + "app": "showcase", + "requires": [ + "a scratch copy of an authored source tree (never the repo's own working tree — this command REWRITES files in place)", + "authored sources deliberately carrying retired spellings from the previous major, so the codemod has something to rewrite" + ], + "knownGaps": [ + "Constructing sources that carry the OLD spellings requires authoring against a retired surface, which the current schemas refuse at parse. Author the fixture files as raw text (they are the input a real upgrader has on disk — files written before the retirement, which is exactly why they do not parse now); do NOT try to produce them through the current schema. If a retired spelling cannot be reconstructed for a sampled entry, record that entry as blocked(fixture) rather than declaring the codemod clean over it.", + "The registry carries entries for both the shipped major and the next one. Rewrites for the NEXT major may be staged but not yet reachable through a released --from/--to pair; record which sampled entries were actually exercised." + ] + }, + "steps": [ + "enumerate the migration registries and record their sizes: packages/spec/src/migrations/entries/{retired-defs,retired-keys,semantic}", + "collect the codemod's advertised contract: grep the spec's retirement messages for the exact command string they prescribe (many read `Run \\`os migrate meta --from \\` to rewrite existing sources automatically`) and record which majors are named", + "copy an authored source tree to a scratch dir and plant retired spellings sampled from the retired-keys registry (at minimum one ui/ key and one data/ key)", + "run `os migrate meta --from ` and capture stdout + the exit code", + "diff the tree before/after and confirm each planted spelling was rewritten to its declared successor", + "run `os build` (or `os validate`) on the rewritten tree — the whole point is that it now passes", + "run the SAME migrate command a second time on the already-migrated tree and diff again (idempotence)", + "run it with a --from BELOW the supported floor and capture the MigrationFloorError path", + "run it with --json where supported and check the exit code in the same way #4873 required of every other migrate subcommand", + "confirm the semantic-only entries behave as ADVISORIES: they are reported to the operator, not silently 'fixed' (a semantic change has no mechanical rewrite by definition)" + ], + "acceptance": [ + { + "clause": "every planted retired spelling is REWRITTEN to the successor the registry declares — the codemod does what the retirement messages promise, for each sampled entry", + "oracle": "log", + "verify": "per-entry before/after diff; an entry left untouched while a message prescribes this command is a FAIL naming that entry", + "evidence": "the per-entry diff" + }, + { + "clause": "the rewritten tree BUILDS — the migration is complete, not partial: running the codemod is sufficient to clear the retirements it claims to handle", + "oracle": "build", + "verify": "os build exits 0 on the rewritten tree; if it still refuses, the refusal text names something the codemod did not rewrite, and that gap is the finding", + "evidence": "build exit code + any residual refusal" + }, + { + "clause": "the codemod is IDEMPOTENT: a second run over an already-migrated tree changes nothing and still exits 0", + "oracle": "log", + "verify": "second-run diff is empty and exit is 0 — a codemod that double-rewrites (or errors on clean input) cannot be safely re-run, and upgraders re-run it", + "evidence": "second-run diff + exit code" + }, + { + "clause": "a --from below the supported floor is REFUSED loudly (MigrationFloorError) rather than silently doing a partial or wrong rewrite", + "oracle": "log", + "verify": "non-zero exit and a message naming the floor and the supported range; a silent no-op here would leave an upgrader believing they had migrated", + "evidence": "message + exit code" + }, + { + "clause": "semantic-registry entries are surfaced as ADVISORIES the operator must act on — reported, never silently marked handled", + "oracle": "log", + "verify": "the output distinguishes mechanically-rewritten entries from advisory ones; a run that reports 'migrated' over a semantic change nobody applied is the dangerous direction", + "evidence": "the output, with the two classes identified" + }, + { + "clause": "the exit-code honesty rule holds here as on every other migrate subcommand: --json exits 0 on success (#4873)", + "oracle": "log", + "verify": "`os migrate meta --json ...; echo $?` is 0 on a successful run", + "evidence": "exit code" + } + ], + "negative": [ + "a retirement message prescribing `os migrate meta --from N` for a spelling the codemod does not actually rewrite — the prescription is then a dead end, and the author who follows it is worse off than one who was told nothing", + "a codemod that rewrites into a spelling that ALSO does not parse (rewriting to a second retired name)", + "a non-idempotent second run (double-rewrite, or an error on clean input)", + "a below-floor --from that silently no-ops and exits 0", + "in-place rewriting with no backup/confirmation on a tree the operator did not expect to be mutated — record the actual safety posture (--yes / prompt / backup) whatever it is" + ], + "traps": ["destructive-in-place"], + "source": [ + "packages/cli/src/commands/migrate/meta.ts (applyMetaMigrations, composeSpecChanges, normalizeStackInput, MigrationFloorError, PROTOCOL_MAJOR)", + "packages/spec/src/migrations/entries/retired-defs/ · retired-keys/ · semantic/ (the registries it applies)", + "the retirement messages across packages/spec that prescribe this command by name (e.g. the BOOK_TRANSLATIONS_RETIRED text in packages/spec/src/system/book.zod.ts)", + "ADR-0087 (retirement registries and conversion routes)", + "#4873 (--json exit-code honesty across migrate subcommands)", + "sibling item cli.migrate-plan-apply-json (the DATABASE migrate lane — lists `meta` as a variant but asserts nothing about it)" + ], + "history": [ + { + "revision": 1, + "date": "2026-08-17", + "change": "new — cli.migrate-plan-apply-json lists `meta` among its eight subcommand variants, but every one of its acceptance clauses is about the DATABASE lane (plan/apply/destructive/busy/--json), so the source-rewriting codemod was named but never exercised. It is load-bearing well beyond its own command: retirement messages throughout the spec prescribe it by name as THE way to clear a retired spelling, so if it does not rewrite what it claims, every one of those prescriptions is a dead end. Authored against a registry that now holds nearly 200 entries across the two most recent majors", + "ref": "#9299" + } + ] + }, + { + "id": "cli.scaffold-console-first-paint", + "title": "The documented newcomer verification loop closes on a FRESHLY SCAFFOLDED project: os validate passes and the Console actually paints at /_console/ — the seam between the scaffold item and the showcase boot item", + "since": "v17", + "status": "active", + "revision": 1, + "priority": "P1", + "surface": "mixed", + "personas": ["a brand-new developer following content/docs/getting-started/quick-start.mdx, with no prior project"], + "fixtures": { + "app": "scaffold", + "requires": [ + "a scratch dir and network access for `npx create-objectstack` (same prerequisite as cli.scaffold-first-run)" + ], + "knownGaps": [ + "This item deliberately does NOT re-prove what two existing items already cover. The scaffold→install→validate→build→artifact-boot chain belongs to cli.scaffold-first-run (automated by .github/workflows/scaffold-e2e.yml); the console-on-a-populated-app half belongs to cli.dev-boot-contract, which drives examples/app-showcase. What is untested is the SEAM: the Console rendering against a freshly scaffolded BLANK project — one object, no seed data, and no seeded admin unless one is requested. Score only the seam; cite the two siblings for the halves.", + "Whether a blank scaffold has any loginable identity out of the box is exactly what clause 4 measures — do not assume either answer, and do not import showcase credentials to make the console open." + ] + }, + "steps": [ + "scaffold a fresh project exactly as the published docs instruct: `npx -y create-objectstack@latest qa-console-first-paint` (no -t flag — the docs never pass one), then `npm install`", + "run the documented gate: `npx os validate; echo $?`", + "run `npm run dev` — the scaffold's OWN script, which is what a newcomer reaches for after `npm install` — and probe http://localhost:3000/_console/", + "load the console in a browser and record what a newcomer actually sees: does the shell render, and is there any identity they can sign in with?", + "stop; run the documented spelling `npx os dev --ui` and probe /_console/ again — the two must agree (see clause 3; `--ui` is a forwarder for a serve-side flag that is already default-on, so this is a no-op equivalence check, not a divergence hunt)", + "capture the boot log for both invocations" + ], + "acceptance": [ + { + "clause": "`npx os validate` passes on the untouched fresh scaffold — the documented gate is green on the thing the documented scaffolder just produced", + "oracle": "log", + "verify": "exit code 0; any refusal on a pristine scaffold is a P1 finding (the first command in the docs fails on the first project)", + "evidence": "output + exit code" + }, + { + "clause": "the Console PAINTS at /_console/ on a fresh blank scaffold under the DOCUMENTED command (`os dev --ui`) — not merely on the showcase", + "oracle": "dom", + "verify": "the console shell renders (not a 404, not an empty body, not a stack trace); confirm against the served HTML and a rendered screenshot, since a 200 that paints nothing is the failure this clause exists to catch", + "evidence": "status + screenshot of the rendered console" + }, + { + "clause": "the scaffold's OWN `dev` script serves the console with NO extra flag — `npm run dev` (bare `objectstack dev`) reaches /_console/ exactly as `os dev --ui` does", + "oracle": "network", + "verify": "probe /_console/ under both invocations; both must serve. The console is default-ON downstream: `dev` spawns `serve` (packages/cli/src/commands/dev.ts:370) forwarding `--ui` only when set and never forwarding `--no-ui`, and serve declares `ui` with `default: true, allowNo: true` (packages/cli/src/commands/serve.ts:221) — so an unflagged `dev` lands on serve's own default. `--ui` on `dev` is a no-op forwarder, which is what content/docs/deployment/cli.mdx:139 means by 'Force Console UI on (already on by default in dev)'. A difference between the two invocations is the FAILURE here, not the expectation", + "evidence": "both probe statuses" + }, + { + "clause": "a newcomer can complete the documented visual check, or the run states precisely what stops them — the docs promise 'create a record, watch an action show/hide, confirm the nav and filters' on this project", + "oracle": "dom", + "verify": "attempt the documented visual check; if no identity exists to sign in with on a blank scaffold, that is the finding — record it rather than substituting showcase credentials", + "evidence": "what the newcomer sees, including any sign-in wall" + } + ], + "negative": [ + "the console 404s on a fresh scaffold while working on the showcase — a first-run-only failure that every showcase-based test is blind to", + "`npm run dev` and `os dev --ui` disagreeing about /_console/ — that would mean the forwarding in dev.ts:370 or the default in serve.ts:221 changed, and the newcomer path silently regressed", + "a future edit giving `dev` its own `ui` default of false, or forwarding `--no-ui`: either would strand the newcomer, and neither would be caught by any showcase-driven item", + "os validate refusing on a pristine scaffold", + "a console that returns 200 and paints nothing (white screen) — 200 is not the oracle here, the rendered DOM is" + ], + "traps": ["hydration-race", "first-boot-cold-start"], + "source": [ + "content/docs/getting-started/quick-start.mdx ('How you verify' — `npx os validate` and `npx os dev --ui`, then open http://localhost:3000/_console/)", + "packages/create-objectstack/src/templates/blank/package.json (scripts.dev = `objectstack dev`, with no --ui — correct, see the chain below)", + "packages/cli/src/commands/serve.ts:221 (`ui: Flags.boolean({ ..., default: true, allowNo: true })` — the console is default-ON here; `--no-ui` is the off switch)", + "packages/cli/src/commands/dev.ts:69 + :370 (dev declares `ui` with NO default and forwards `...(flags.ui ? ['--ui'] : [])` — it only ever ADDS the flag, never forwards --no-ui, so an unflagged dev inherits serve's default-on)", + "content/docs/deployment/cli.mdx:139 ('--ui | Force Console UI on (already on by default in dev)') and :196 ('--ui / --no-ui — Toggle Console UI at /_console/ (default on)')", + "packages/create-objectstack/src/template-registry.ts (TEMPLATES = { blank } only; RETIRED_TEMPLATES named for a named refusal)", + "sibling item cli.scaffold-first-run (scaffold→install→validate→build→`os start --artifact`→health/ready; automated by .github/workflows/scaffold-e2e.yml)", + "sibling item cli.dev-boot-contract (os dev --ui + /_console/ sign-in, driven against examples/app-showcase)", + "sibling item studio-authoring.first-run-loop (the in-console authoring loop, once the console is reachable)" + ], + "history": [ + { + "revision": 1, + "date": "2026-08-17", + "change": "new — angle-5 (docs claims) probe of the published first-run. Covers a real seam between two existing items: cli.scaffold-first-run boots the scaffold headless via `os start --artifact`, cli.dev-boot-contract drives the console against the fully-populated showcase, and nobody drives the console against a fresh BLANK scaffold — where there is no seed data and no seeded admin unless one is requested. Checked and CLEAN on three questions the sweep raised: no doc anywhere in content/docs/ prescribes a retired template (`todo`/`compliance`/`content`/`contracts`/`procurement`), every documented invocation is a bare `npx create-objectstack my-app` resolving to the only catalog entry, and `npm run validate` is the same binary as `npx os validate`. REFUTED during authoring, recorded here so it is not re-derived: the sweep first read the blank template's bare `\"dev\": \"objectstack dev\"` against the quick-start's `os dev --ui` and inferred that `npm run dev` would strand a newcomer with no console. Source-checking the chain disproves it — dev.ts:370 forwards `--ui` only when set and never forwards `--no-ui`, serve.ts:221 declares `ui` `default: true`, and cli.mdx:139 states outright that the console is 'already on by default in dev'. The bare script is CORRECT; `--ui` on `dev` is a no-op forwarder. Clause 3 accordingly asserts the two invocations AGREE, and a difference is the failure", + "ref": "#9299" + } + ] } ] } diff --git a/docs/qa/platform-checklist/areas/integration-system.json b/docs/qa/platform-checklist/areas/integration-system.json index 0c80879216..22c0489321 100644 --- a/docs/qa/platform-checklist/areas/integration-system.json +++ b/docs/qa/platform-checklist/areas/integration-system.json @@ -1230,6 +1230,119 @@ "ref": "claude/platform-test-checklist-ocwugl" } ] + }, + { + "id": "integration-system.datasource-credential-refusal-matrix", + "title": "A datasource credential can never be authored inline NOR read back: every inline-credential door refuses at publish, and BOTH read doors redact — including legacy alias spellings already sitting in stored rows", + "since": "v17", + "status": "active", + "revision": 1, + "priority": "P0", + "surface": "mixed", + "personas": ["seeded admin", "authenticated non-admin", "anonymous"], + "fixtures": { + "app": "showcase", + "requires": [ + "the ability to author/publish a datasource (os build and/or POST /api/v1/datasources)", + "the ability to write a stored sys_metadata datasource row directly, to plant a legacy alias spelling that no current parse would produce" + ], + "knownGaps": [ + "The legacy-alias clause tests a row shape the CURRENT parse refuses to create — a stored row from before the aliases moved to `guidance`. It is only reachable by writing the row directly (the wizard persists through metadata.register, whose validation is a structural name/label check, so that path can plant one). If the run cannot write such a row, that clause is blocked(fixture) and MUST be recorded — a redaction that has never been tested against the shape it exists for is an untested redaction.", + "`encryptionKey` (turso) is deliberately STILL WRITABLE while being redacted on read (#8081 scope item 4). Scoring it as a write refusal is a false fail; the correct expectation is write-accepted + read-redacted." + ] + }, + "steps": [ + "enumerate the inline-credential refusal doors from packages/spec/src/migrations/entries/semantic/ — at minimum the config-key door, the URL userinfo door, the URL query-parameter door, the driver-options door (mongo), the placeholder door, and the connector inline-authentication door", + "for EACH door, author a datasource carrying the credential in that position and attempt to publish it (os build, then the runtime admin door)", + "author the same datasource with the credential bound through `external.credentialsRef` (or the connection form) and confirm it publishes AND connects", + "read the published datasource back through the datasource-admin door (GET /api/v1/datasources)", + "read the SAME datasource back through the metadata door (GET /meta/) — a different code path with its own redaction hook", + "plant a stored row carrying a legacy alias spelling (passwd / pwd / token / jwt / auth_token / authtoken) and read it back through both doors", + "author a datasource for a driver the platform ships no contract for, carrying a canonically-spelled credential key, and read it back", + "for turso specifically, write `encryptionKey`, then read it back through both doors", + "grep the full response bodies of every read above for the planted cleartext values" + ], + "acceptance": [ + { + "clause": "EVERY enumerated inline-credential door refuses at publish — the matrix is exhaustive over the doors the semantic registry declares, not a sample of one", + "oracle": "build", + "verify": "one publish attempt per door; each refuses with a message naming the offending key/position and pointing at external.credentialsRef. The run record lists one row per door", + "evidence": "the per-door refusal table" + }, + { + "clause": "the credential-bound alternative WORKS: the same datasource authored with external.credentialsRef publishes and actually connects — the refusal has a paved path, it is not a dead end", + "oracle": "api", + "verify": "publish succeeds and a health/connect probe succeeds", + "evidence": "publish response + connect probe" + }, + { + "clause": "BOTH read doors redact — the datasource-admin path AND the metadata path — and they agree. A credential visible through either door is a P0 leak", + "oracle": "api", + "verify": "grep each full response body for the planted cleartext; both must be absent, and the two doors' redacted shapes must be consistent with each other", + "evidence": "both response bodies, quoted in full for the config object" + }, + { + "clause": "redaction is presence-preserving, not existence-erasing: a SET credential reads back as the mask (or hasSecret-style marker) while an UNSET one reads back null — the distinction a console renders 'configured vs not configured' from", + "oracle": "api", + "verify": "read one datasource with the credential set and one without; the two must be distinguishable, and the set one must not read back null", + "evidence": "both config objects" + }, + { + "clause": "legacy ALIAS spellings sitting in a stored row are redacted too — the spellings no current parse produces are exactly the ones a schema-shape-derived redactor would miss", + "oracle": "api", + "verify": "the planted alias value is absent from both doors' response bodies; if the row could not be planted, record blocked(fixture) rather than passing", + "evidence": "the planted row + both reads" + }, + { + "clause": "an UNKNOWN driver's canonically-spelled credential keys are redacted by NAME — an unrecognized driver means 'nothing to check against', never 'nothing to protect'", + "oracle": "api", + "verify": "read back the unknown-driver datasource; the canonical credential key must be redacted despite no shipped contract for that driver", + "evidence": "the response body" + }, + { + "clause": "the echoed-mask write guard holds: writing the mask value back verbatim is treated as 'unchanged' and does NOT overwrite the stored credential", + "oracle": "api", + "verify": "read a datasource, PUT the response back unmodified, then probe connectivity — the connection must still work. A form round-trip that destroys the stored credential is the ADR-0100 §B3 defect", + "evidence": "the round-trip + the post-round-trip connect probe" + }, + { + "clause": "the redaction is not a privilege artifact: a non-admin and an anonymous caller get no MORE than the admin does through either door (they should get less, or be refused outright)", + "oracle": "api", + "verify": "repeat both reads as non-admin and anonymous; neither may reveal a value the admin read redacted", + "evidence": "all four responses" + } + ], + "negative": [ + "a cleartext credential appearing anywhere in a read response — P0, and the reason every clause greps the FULL body rather than inspecting the fields it expects", + "a door that refuses at build but accepts through the runtime admin path (or the reverse) — the two-door disagreement that lets a credential in the back way", + "a redaction applied on the datasource-admin door but not the metadata door: the two share no dependency except spec, which is precisely why the credential-key definition was centralized there, and precisely how they could drift", + "an alias-spelled credential surviving redaction because the redactor derived its key set from the current schema shape only", + "an unknown driver's credential served in cleartext on the grounds that no contract declares it a credential", + "an echoed mask overwriting the stored credential with the literal mask string", + "a refusal that names no replacement — an author blocked from the inline spelling with no pointer to credentialsRef will reach for a workaround" + ], + "traps": ["auth-state-leak", "cache-staleness"], + "source": [ + "ADR-0100 (credential read mask; §B3 echoed-mask write guard)", + "ADR-0015 / ADR-0062 (credentials route through datasource config)", + "packages/spec/src/data/secret-mask.ts (the one mask literal both readers import)", + "packages/spec/src/data/datasource-credential-redaction.ts (the ONE definition of 'what is a credential key' — schema-derived z.never() refusals, FORMER_CREDENTIAL_ALIASES, STILL_WRITABLE_CREDENTIAL_KEYS)", + "packages/spec/src/kernel/metadata-type-redaction.ts (the per-type redaction hook — the metadata read door)", + "packages/services/service-datasource/src/datasource-config-redaction.ts + datasource-credential-migration.ts (the datasource-admin read door)", + "packages/spec/src/migrations/entries/semantic/17.datasource-config-inline-credential-refused.ts · 17.datasource-config-placeholder-refused.ts · 17.datasource-config-url-userinfo-refused.ts · 17.connector-inline-authentication-publish-refused.ts", + "packages/spec/src/migrations/entries/semantic/18.datasource-config-url-query-credential-refused.ts · 18.datasource-config-mongo-options-credential-refused.ts · 18.datasource-credentialsref-mongo-url-no-user-refused.ts · 18.datasource-config-postgres-url-unparseable-refused.ts", + "packages/spec/src/migrations/entries/retired-keys/17.data__MongoConfig__password.ts · 17.data__MysqlConfig__password.ts · 17.data__PostgresConfig__password.ts · 17.data__TursoConfig__authToken.ts", + "#7990 / #8081 / #8126 / #8154 / #8300 (the centralization of the credential-key definition into spec)", + "sibling coverage this item deliberately does NOT duplicate: integration-system.datasource-admin-lifecycle (admin-door lifecycle, 'secret never echoes'), platform-core.settings-hub-roundtrip (settings secrets), records-forms.encrypted-field-behavior (ADR-0100 field-level masking)" + ], + "history": [ + { + "revision": 1, + "date": "2026-08-17", + "change": "new — the credential campaign is the single largest behavior cluster in the window (retired keys on four driver configs, eight+ semantic refusal doors, a centralized credential-key definition, and a second redaction door on the metadata read path) and no ratchet could see any of it: `datasource` was already a mapped kind, and none of the doors is an enum member. Existing coverage reached only the datasource-ADMIN door's 'secret never echoes'; the metadata read door (#8154), the legacy-alias class, the unknown-driver rule and the echoed-mask write guard were untested. P0 because every clause failure is a cleartext credential disclosure", + "ref": "#9299" + } + ] } ] } \ No newline at end of file diff --git a/docs/qa/platform-checklist/areas/platform-core.json b/docs/qa/platform-checklist/areas/platform-core.json index 4e3061b322..366800df71 100644 --- a/docs/qa/platform-checklist/areas/platform-core.json +++ b/docs/qa/platform-checklist/areas/platform-core.json @@ -1058,6 +1058,111 @@ "history": [ { "revision": 1, "date": "2026-08-08", "change": "initial — launcher active+hidden filter and isDefault→landing proven at the metadata layer, with the AppManagementPage toggle recorded as a client-only stub (expected-fail probe) rather than faked", "ref": "claude/platform-test-checklist-ocwugl" } ] + }, + { + "id": "platform-core.docs-audience-gate", + "title": "Documentation audience gating (book + doc): the tree endpoint and the doc read layer gate on ONE semantics — public/org/permissionSet, fail-closed — and an unclaimed doc never leaks through a public book", + "since": "v14.1", + "status": "active", + "revision": 1, + "priority": "P1", + "surface": "api", + "personas": [ + "anonymous (no session)", + "authenticated member who does NOT hold the gating permission set", + "authenticated member who DOES hold the gating permission set", + "seeded admin" + ], + "fixtures": { + "app": "showcase", + "requires": [ + "at least one authored `book` and its `doc` corpus in an installed package (the showcase ships package docs; `deriveImplicitPackageBook` covers a package with docs but no authored book)", + "a session for each of the four personas above" + ], + "knownGaps": [ + "Stock showcase seeds are not known to author a book with `audience: { permissionSet: '' }` NOR one with `audience: 'public'`. Verify against the installed corpus FIRST (GET /api/v1/meta/book and read each item's `audience`). If neither gated nor public books exist, the permissionSet and public clauses are blocked(fixture) and MUST be recorded as such — do NOT score them from the `org` default alone, which is the one audience every authenticated caller passes and therefore proves nothing about the gate.", + "Authoring a book/doc at runtime to create the fixture is acceptable only if the run records that it did so and tears it down; the audience is read off the stored row, so a runtime-authored book exercises the same resolver." + ] + }, + "steps": [ + "enumerate the installed corpus: GET /meta/book and GET /meta/doc as admin; record every book's `name` and `audience`, and which docs each book claims", + "for a book whose audience is `org` (or unset — the §3.6 default): GET /meta/book//tree anonymously, then as a plain authenticated member", + "for a book whose audience is `public` (if one exists): GET the tree anonymously", + "for a book whose audience is `{ permissionSet: }` (if one exists): GET the tree as a holder and as a non-holder", + "GET /meta/book//tree — the implicit per-package book path (§6.4)", + "compare the ENTRIES of a tree fetched as admin vs as a lesser persona for the same book — the per-entry doc-audience filter, not just the book-level gate", + "identify a doc claimed by NO authored book (an orphan landing in the synthetic Uncategorized group) and check whether it appears in a `public` book's tree", + "repeat one tree fetch with a locale header (e.g. Accept-Language: zh-CN) against a doc carrying `translations`, and once with a repeated `?package=` query parameter" + ], + "acceptance": [ + { + "clause": "the book-level audience gate refuses with the CORRECT distinction: anonymous against a non-public book is 401 UNAUTHENTICATED, an authenticated non-holder against a permissionSet-gated book is 403 PERMISSION_DENIED — never a bare 404, never an empty 200 tree", + "oracle": "api", + "verify": "status + error envelope `code` on each refusal; an empty-but-200 tree where a refusal is due is a FAIL (it makes a gate indistinguishable from an empty book)", + "evidence": "status line + envelope body per persona/book pair" + }, + { + "clause": "`audience` vocabulary holds all three ways: `public` serves anonymously; `org`/unset serves any authenticated principal and refuses anonymous; `{ permissionSet }` serves a holder and refuses a non-holder", + "oracle": "api", + "verify": "one tree fetch per (audience kind × persona) cell that the fixture supports; cells the fixture cannot supply are recorded blocked(fixture), not passed", + "evidence": "the cell matrix with status per cell" + }, + { + "clause": "the permissionSet branch FAILS CLOSED when holdings cannot be resolved — an unresolvable/absent `permissionSets` denies rather than allows (ADR-0049); an unknown future audience shape also denies", + "oracle": "test", + "verify": "run the spec's own pin for audienceAllows (packages/spec/src/system/book.zod.ts) and cite the case asserting `permissionSets: undefined` → false; if no such case exists, that ABSENCE is the finding — record it and file to FOLLOW-UPS rather than ticking", + "evidence": "test output naming the fail-closed case" + }, + { + "clause": "gating is TWO-LAYERED: passing the book gate does not reveal every entry — the tree's entries are additionally filtered by each doc's effective audience, so a lesser persona's tree is a subset of admin's and contains no entry that would refuse on fetch", + "oracle": "api", + "verify": "diff the entry sets; then fetch EVERY entry the lesser persona's tree offers as that persona — each must resolve, none may 401/403 (a nav entry that refuses on click is the exact defect this filter exists to prevent)", + "evidence": "entry-set diff + the per-entry fetch statuses" + }, + { + "clause": "an UNCLAIMED doc (synthetic Uncategorized orphan) is never exposed through a public book — orphans are a rendering convenience, not an authored membership claim, and an unclaimed doc defaults to `org`", + "oracle": "api", + "verify": "the orphan doc's name is absent from the anonymous fetch of any public book's tree, and fetching it directly anonymously refuses", + "evidence": "tree body + the direct-fetch status" + }, + { + "clause": "an unknown book name resolves as the implicit per-package book rather than 404, and that implicit book's audience is `org` (so it refuses anonymous)", + "oracle": "api", + "verify": "GET the tree for a package id; expect a resolved tree for an authenticated caller and a 401 for anonymous", + "evidence": "both responses" + }, + { + "clause": "doc locale resolution runs on the tree path: a doc carrying `translations` renders its localized label under a locale header, while a book's OWN inline translations are refused at authoring (retired in 17.0.0, #4667) — the near-neighbour trap", + "oracle": "api", + "verify": "localized label present in the tree under the locale header; separately, parse a book carrying inline `translations` and confirm the refusal names `doc.translations` as the live alternative", + "evidence": "the localized tree entry + the refusal text" + } + ], + "negative": [ + "a non-public book that serves a tree to an anonymous caller — the fell-open direction, P1", + "a public book that leaks docs it does not claim — the orphan-group leak the resolver deliberately excludes", + "a refusal that discloses existence/shape (e.g. an error body echoing the book's group structure or doc names)", + "a 404 where a 401/403 is due: it hides the gate behind a not-found and makes the two states indistinguishable to a client", + "a repeated `?package=` query parameter that is silently coalesced instead of refused (refuseRepeatedQueryParams)" + ], + "traps": ["auth-state-leak", "cache-staleness"], + "source": [ + "ADR-0046 §6 (documentation spine), §6.2.1 (membership is DERIVED), §6.4 (implicit per-package book), §6.7 (read-layer audience gating)", + "ADR-0090 (audience vocabulary)", + "ADR-0049 (fail closed — unresolvable holdings deny)", + "packages/spec/src/system/book.zod.ts — audienceAllows / resolveDocAudiences / docAudienceAllows / resolveBookClaimedDocs / deriveImplicitPackageBook / resolveBookTree (UNCATEGORIZED_KEY exclusion)", + "packages/rest/src/rest-server.ts — GET /book/:name/tree (audience gate, per-entry filter, resolveDocLocale, refuseRepeatedQueryParams)", + "#2774 (feat(security)!: enforce book audience at the REST read layer)", + "packages/spec/liveness/book.json + doc.json (the ledgers whose coverage.json waivers this item retires)" + ], + "history": [ + { + "revision": 1, + "date": "2026-08-17", + "change": "new — authored to RETIRE the `book` and `doc` coverage waivers, which claimed these kinds are 'display-only, no independent runtime behavior to gate beyond serving'. Grounding disproved that: the read layer runs a three-member audience vocabulary with an ADR-0049 fail-closed branch, a two-layer gate (book, then per-entry doc audience union), a deliberate orphan-group non-leak rule, an implicit-package-book fallback, and locale resolution. That is an access-control surface, tested here like every other one. Capability predates the ledger baseline (landed at spec 14.1.0, #2774) — never authored until now, which is itself the finding", + "ref": "#9299" + } + ] } ] } diff --git a/docs/qa/platform-checklist/coverage.json b/docs/qa/platform-checklist/coverage.json index 34070f0a61..56dbaf47e2 100644 --- a/docs/qa/platform-checklist/coverage.json +++ b/docs/qa/platform-checklist/coverage.json @@ -26,7 +26,9 @@ ] }, "book": { - "waived": "docs-shaped content kind (display-only, ADR-0033 exemption class). The console DOES ship a docs/book reader (objectui apps/console DocPage/BookPage) — but the kind is authored-content-as-data with no independent runtime behavior to gate beyond serving; parse coverage exists via the spec's own tests. Waived for behavior, not for lack of a surface." + "items": [ + "platform-core.docs-audience-gate" + ] }, "dashboard": { "items": [ @@ -48,7 +50,9 @@ ] }, "doc": { - "waived": "docs-shaped content kind (display-only, ADR-0033 exemption class) — same posture as `book`: a reader surface exists in the console, but there is no runtime behavior to assert beyond serving." + "items": [ + "platform-core.docs-audience-gate" + ] }, "email_template": { "items": [