Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 84 additions & 11 deletions docs/qa/platform-checklist/RUNNER.md
Original file line numberDiff line numberDiff line change
Expand Up@@ -115,11 +115,31 @@ contradicts it, and correct it here when it does.
section at all.** `runRlsProofs` runs only behind the flag
(`packages/cli/src/commands/verify.ts`: `rls: Flags.boolean({ default: false })`, the
proofs sit inside `if (flags.rls)`, and the report prints `if (rls)`). **Check:** the
last block of a bare run is the CRUD summary — `── 15 verified, 0 gaps, 0 FAILED, 1
last block of a bare run is the CRUD summary — `── 16 verified, 0 gaps, 0 FAILED, 1
needs-fixture, 7 skipped` on stock showcase — with no `PROVEN`/`HOLES` line anywhere.
Adding `--rls` appends the RLS block: `20 PROVEN (20 consistent, 0 HOLES)` over 23
objects, plus `9 of 9 declared position(s) probed`. ⛔ Do not cite plain `verify`
output as the oracle for an RLS clause — that run never consulted one.
Adding `--rls` appends the RLS block. ⛔ Do not cite plain `verify` output as the
oracle for an RLS clause — that run never consulted one.

**The discriminator is the PRESENCE of a `PROVEN`/`HOLES` line, never its digits.**
The counts move with the seed set and are recorded here only as the shape to expect:
the CRUD tail read `15 verified` through 2026-08-18 and reads `16` on 17.1.0 because
the showcase gained one object (`client-brief.object.ts`, added between the two sweep
subjects); a run whose digits differ has a different seed, not a regression. Re-derive
the number, do not file it.

⚠️ **The `--rls` block prints TWO summary lines in DIFFERENT units — do not compare
them.** `formatRlsReport` (`packages/verify/src/rls.ts`) emits a per-persona
`── <n> PROVEN (<n> consistent, <n> HOLES) · <n> NOT PROVEN (…)` for the probe
persona, then — once position personas have run — a total line
`══ all personas: … [unit: one object × persona probe]`. The first counts OBJECTS for
one persona; the second counts object × persona PAIRS across all of them, and the
source stamps that unit into the line precisely so the two are not read as one number
moving. Measured on 17.1.0: the total line reads
`all personas: 38 PROVEN (38 consistent, 0 HOLES) · 226 NOT PROVEN`, alongside
`9 of 9 declared position(s) probed`. An older note here recorded
`20 PROVEN … over 23 objects` — that was the per-persona line, so it is **not
comparable** to the total rather than merely stale. Name which of the two lines you
are quoting whenever you cite either.

- **Playwright needs an explicit `executablePath` on these containers.**
`@playwright/test` 1.62.1 resolves chromium build **1234**; only **1194** is installed.
Expand DownExpand Up@@ -147,13 +167,45 @@ contradicts it, and correct it here when it does.
and App declares no `id` of its own. `?id=com.example.showcase` (the package id, from
`objectstack.config.ts`) returns `{"items":[]}` — which reads exactly like "the app
metadata is gone", the highest-value false P0 shape there is. Real names: `showcase_app`
(showcase), `setup` / `studio` / `account` (platform built-ins). ⚠️ **An empty
`items` has two distinct causes** — a wrong spelling, or an app that is genuinely not
installed. `studio` is defined (`packages/platform-objects/src/apps/studio.app.ts`) but
the showcase does **not** install it, so `?id=studio` is legitimately empty there; a
stock admin list is `["showcase_app","setup","account"]`. **Check:** fetch
`/api/v1/meta/app` with no query first and read the names it actually returns, then
filter.
(showcase), `setup` / `account` (the platform built-ins a stock boot registers). ⚠️
**An empty `items` has two distinct causes** — a wrong spelling, or an app that is
genuinely not registered. `studio` is the standing instance of the second cause: the
App document is defined (`packages/platform-objects/src/apps/studio.app.ts`) and
packaged (`packages/apps/studio`, `com.objectstack.studio`), but **no stock boot loads
it, by design** — both boot paths skip it with the identical note, "`@objectstack/studio`
is intentionally NOT default-loaded — the console ships a dedicated Studio surface at
`/_console/studio/<pkg>/<pillar>`, so Studio no longer needs to exist as a navigable
app tile" (`packages/plugins/plugin-dev/src/dev-plugin.ts`, the ADR-0048 app-package
loop; `packages/cli/src/commands/serve.ts`, same loop — both register only
`@objectstack/setup` and `@objectstack/account`). So `?id=studio` is legitimately empty
on every stock boot, not just on the showcase, and a stock admin list is
`["showcase_app","setup","account"]`. ⛔ Do not file the absence as a missing built-in,
and do not install the package to "restore" it — that changes what the boot ships.
**Check:** fetch `/api/v1/meta/app` with no query first and read the names it actually
returns, then filter.

- **An isolated boot on a non-3000 port breaks the showcase's self-pinging connectors
unless you export `OS_PORT` — `-p` alone is not enough, and the symptom impersonates an
egress block.** The showcase's REST/OpenAPI connector instances point at the running
server itself, resolved by `resolveShowcaseSelfUrl()`
(`examples/app-showcase/src/system/self-url.ts`) in this order: `SHOWCASE_SELF_URL` →
`OS_PORT` → `PORT` → the literal `http://127.0.0.1:3000`. Those are **environment**
reads. The CLI's `-p` flag is only a *default sourced from* `OS_PORT`
(`packages/cli/src/commands/serve.ts`: `port: Flags.string({ char: 'p', … default:
readEnvWithDeprecation('OS_PORT','PORT') ?? '3000' })`) and nothing writes the variable
back, so the dogfood §0 isolated boot — `objectstack dev … -p <port>` with the env
untouched — listens on `<port>` while every declarative connector dials **3000**. Every
dispatch then fails `connector_action(showcase_status_api.request) failed: fetch failed`.
⚠️ **That string is not evidence of a sandbox network restriction**, and reading it as
one is the expensive mistake: the module's own header records that #7516 burned a run
proving it was an address problem by putting a TCP forwarder on 3000, and #7538 fixed
the resolution but could not fix a boot that never exports the variable. **Check:**
before scoring any connector clause, `GET /api/v1/automation/connectors` and read the
instance `baseUrl` — if it says `:3000` and your server does not listen there, the
fixture is misconfigured, not blocked. **Fix:** boot with `OS_PORT=<port>` exported (or
`SHOWCASE_SELF_URL=http://127.0.0.1:<port>`) *in addition to* `-p <port>`. ⛔ Do not
record a connector clause as `blocked(environment)` on the strength of `fetch failed`
alone.

- **`ss` is not installed in these containers — read liveness with `curl`, never a socket
table.** `ss` and `netstat` are both absent (`command not found`); `lsof` and `fuser`
Expand DownExpand Up@@ -250,6 +302,27 @@ contradicts it, and correct it here when it does.
by name and exits **1**, where it used to exit **0** in silence — the entry point is
`packages/cli/bin/run.js`.

- **23 items carry an `automated.ref` that lives ENTIRELY in the `objectui` repo — from
this checkout they are neither runnable nor pin-evidenced.** Their refs name only
objectui paths (`e2e/live/*.spec.ts`, `e2e/import-console/*`, `e2e/import-harness/*`,
and unit suites under `packages/plugin-grid`, `plugin-gantt`, `plugin-calendar`,
`plugin-form`, `fields`, `core`, `app-shell`). This repository builds no console bundle
and holds none of those specs, so a run here can neither execute the pin nor cite its
output. **Every one of those items now says so in its own `fixtures.knownGaps`** — the
rediscovery this fact exists to stop (each sweep re-derived it from scratch). ⚠️ The
distinction that matters is *exclusive*: items whose `automated.ref` names an
objectui spec **alongside** a framework pin (`packages/qa/dogfood/**`,
`packages/objectql/**`) are partly runnable here — run the framework half and score the
objectui half separately, never one for the other.

**Two honest options, and the run record must say which it took:** (a) drive the item
**by hand** in the browser against a live boot and score it as a manual browser run —
the `automated` field does not make it exempt; or (b) run the pin inside an **objectui
checkout at the pinned revision** and cite that output, **naming the revision** (the
17.1.0 sweep subject was `9a3daf8d37ad`). ⛔ Never record such an item as covered on the
strength of the `automated` field alone: an unrun pin is a claim, not evidence — which
is the same "ticking on a label" failure clause-grained acceptance exists to prevent.

### Trap vocabulary (`traps` field)

| trap | what it fakes | counter |
Expand Down
33 changes: 23 additions & 10 deletions docs/qa/platform-checklist/areas/api-backend.json
Original file line numberDiff line numberDiff line change
Expand Up@@ -441,7 +441,7 @@
"title": "Sampled endpoints return the standard error envelope with ledgered codes — no invented codes, no retired keys, statuses match the map",
"since": "v16",
"status": "active",
"revision": 2,
"revision": 3,
"priority": "P1",
"surface": "api",
"personas": [
Expand All@@ -453,7 +453,7 @@
"app": "showcase",
"requires": [
"showcase_invoice.name is required:true — the cheapest deterministic VALIDATION_FAILED",
"a member persona lacking showcase_announcement create (member_default: create false) — the deterministic 403"
"a member persona lacking showcase_announcement create (member_default: create false) — the deterministic 403. Stock showcase SEEDS two loginable non-admin personas for exactly this: Mei Phone (phone.demo@example.com) and Ada Auditor (auditor.demo@example.com), both with DEMO_PERSONA_PASSWORD = 'showcase123' (examples/app-showcase/src/security/demo-personas.ts, provisioned by seed-approval-demo.ts on every dev boot). Driven as Mei, POST /api/v1/data/showcase_announcement answers 403 PERMISSION_DENIED — a ledgered code — so the 403 family is reachable with no provisioning at all. POST /api/v1/auth/sign-up/email also works on a stock dev boot if a THIRD identity is wanted"
]
},
"steps": [
Expand DownExpand Up@@ -533,7 +533,13 @@
"change": "new — error-envelope conformance sampling grounded in the two-tier code ledger, per the deep-test contract",
"ref": "claude/platform-test-checklist-ocwugl"
},
{ "revision": 2, "date": "2026-08-18", "change": "re-pointed clause 1 and steps 8/9 at ErrorCode, the canonical ADR-0112 D4 union export. The item asked the runner to hand-union StandardErrorCode with ERROR_CODE_LEDGER, but the latter is a Record of package name to code array rather than a flat code list, and three of the item's own sampled codes (VALIDATION_FAILED, UNSUPPORTED_QUERY_PARAM, BATCH_NOT_ATOMIC) are not in StandardErrorCode at all — they resolve only through REGISTERED_ERROR_CODES. error-code-ledger.zod.ts already exports ErrorCode as exactly that union, so the clause now names one export instead of prescribing a union the runner has to rebuild (#9417)", "ref": "#9386" }
{ "revision": 2, "date": "2026-08-18", "change": "re-pointed clause 1 and steps 8/9 at ErrorCode, the canonical ADR-0112 D4 union export. The item asked the runner to hand-union StandardErrorCode with ERROR_CODE_LEDGER, but the latter is a Record of package name to code array rather than a flat code list, and three of the item's own sampled codes (VALIDATION_FAILED, UNSUPPORTED_QUERY_PARAM, BATCH_NOT_ATOMIC) are not in StandardErrorCode at all — they resolve only through REGISTERED_ERROR_CODES. error-code-ledger.zod.ts already exports ErrorCode as exactly that union, so the clause now names one export instead of prescribing a union the runner has to rebuild (#9417)", "ref": "#9386" },
{
"revision": 3,
"date": "2026-08-21",
"change": "named the seeded demo personas in fixtures.requires. The 17.1.0 sweep recorded this item's 403 family as unreachable on stock fixtures and scored the item partial, having assumed a runner gets only the dev admin; the filer withdrew that in the same wave after driving Mei to a real 403 PERMISSION_DENIED (6/6 families, superseding #10233 on that point). The personas were always there — the item just never said so, which is what made the wrong assumption cheap. Requirement text now carries both persona emails, the well-known password constant and its source file, so the next runner does not repeat it (#10236 A5)",
"ref": "#10236"
}
]
},
{
Expand DownExpand Up@@ -1046,7 +1052,7 @@
"title": "Filter comparand conformance: only the accepted comparand TYPES pass the where-door, a bigint past the exact-integer limit is refused, and a dotted head is classified rather than guessed",
"since": "v17",
"status": "active",
"revision": 1,
"revision": 2,
"priority": "P1",
"surface": "api",
"personas": ["seeded admin"],
Expand All@@ -1056,14 +1062,15 @@
"an object with a scalar field, a relation (lookup/master-detail) field, and — if the showcase ships one — a formula/virtual field, so all three dotted-head classes are reachable"
],
"knownGaps": [
"If no formula/virtual field exists on a filterable object in the stock seeds, the `virtual` head class is blocked(fixture) — record it rather than scoring the class from the relation case."
"If no formula/virtual field exists on a filterable object in the stock seeds, the `virtual` head class is blocked(fixture) — record it rather than scoring the class from the relation case.",
"The bigint past-limit refusal (clause 2) has NO REST arm at all — it is not a thin fixture, it is a door property. `normalizeFilterComparandTypes` refuses on `typeof value === 'bigint'`, and JSON.parse produces `number` for every numeric literal, so the arm is unreachable through any JSON transport regardless of magnitude. Score clause 2 at the engine/driver door and record the REST verdict as not-applicable; a REST 200 there is neither evidence of enforcement nor of a leak."
]
},
"steps": [
"read ACCEPTED_FILTER_COMPARAND_TYPES in packages/spec/src/data/filter-comparand-type.ts and enumerate the current members — the variants matrix below is pinned to that export, so a mismatch is a validator failure, not a judgement call",
"for EACH accepted comparand type, issue a data-API read whose `where` carries a comparand of that type against a compatible field, and record the answer set",
"issue a read whose comparand is a type NOT in the accepted set (e.g. a plain object, an array where the operator takes a scalar, undefined, a function-shaped value) and capture the refusal",
"probe the bigint boundary: a bigint comparand at FILTER_COMPARAND_BIGINT_EXACT_LIMIT (2n**53n) and one past it",
"probe the bigint boundary AT AN IN-PROCESS DOOR: pass a real JS bigint at FILTER_COMPARAND_BIGINT_EXACT_LIMIT (2n**53n) and one past it through objectql/engine (or replay FILTER_COMPARAND_TYPE_CASES against a driver) — not over REST, where JSON.parse has already turned the literal into a number before the door is consulted",
"issue reads with dotted filter heads of each class — a relation head, a virtual/formula head, and a scalar head (a dot on a field that is not traversable) — and record how each is classified",
"cross-check the SAME filters through a second door (the engine/objectql path or a view filter) so the refusal is proven at the contract, not at one transport"
],
Expand All@@ -1081,10 +1088,10 @@
"evidence": "status + message + the returned row count vs the unfiltered count"
},
{
"clause": "the bigint exact-integer limit is enforced at the boundary: a value inside the limit is accepted, one past it is refused rather than silently losing precision",
"oracle": "api",
"verify": "both requests; the past-limit case must refuse — a 200 whose comparand was rounded is a silent-wrong-answer FAIL",
"evidence": "both responses"
"clause": "the bigint exact-integer limit is enforced at the boundarya value inside the limit narrows and is accepted, one past it is refused rather than silently losing precision. ⚠️ DOOR-SPECIFIC: assertable only at an IN-PROCESS door (objectql/engine, or a driver replaying the conformance table); it is structurally unreachable over REST",
"oracle": "test",
"verify": "run packages/objectql/src/engine-comparand-type-door.test.ts (the engine door) plus the driver suites that replay FILTER_COMPARAND_TYPE_CASES (driver-memory / driver-sql / driver-sqlite-wasm / driver-mongodb / driver-turso, each `*-comparand-type-conformance.test.ts`), and cite the past-limit case by its name in the table: 'a bigint beyond ±2^53 is refused — precision loss must not answer silently' (packages/spec/src/data/filter-comparand-type-conformance.ts). ⛔ Do NOT score this over HTTP, and do NOT read a REST 200 as either a pass or a FAIL: the refusal arm is guarded by `typeof value === 'bigint'` (judgeLiteralComparand in filter-comparand-type.ts) and JSON.parse never yields a bigint at any magnitude, so no JSON body can reach it. Measured: the literal 9007199254740993 (2^53 + 1) parses to 9007199254740992, which IS FILTER_COMPARAND_BIGINT_EXACT_LIMIT and therefore also inside the inclusive `<=` bound — two independent reasons the REST arm cannot fail. The REST door's honest verdict for this clause is not-applicable, recorded as such",
"evidence": "the engine-door + driver-conformance output naming the past-limit case; if a REST probe was issued at all, the parsed comparand value showing why its status proves nothing"
},
{
"clause": "a dotted filter head is CLASSIFIED (relation | virtual | scalar | null) and each class is handled per contract — a scalar head with a dot is refused rather than treated as a relation traversal",
Expand DownExpand Up@@ -1135,6 +1142,12 @@
"date": "2026-08-17",
"change": "new — the comparand-typing contract landed as a cluster of seven spec modules plus three semantic migrations at 17.0.0-rc.6 and had no item. api-backend.query-contract-matrix covers OPERATORS ('every filter operator gives known answers') but says nothing about the TYPE of the right-hand side, which is where the silent-widening failure lives. Variants pinned to ACCEPTED_FILTER_COMPARAND_TYPES so a seventh accepted type flags the matrix stale",
"ref": "#9299"
},
{
"revision": 2,
"date": "2026-08-21",
"change": "made clause 2 (the bigint exact-integer boundary) name the door it is actually assertable at, and marked it door-specific. As written it prescribed 'both requests' against the api oracle, but the refusal arm is guarded by `typeof value === 'bigint'` and JSON.parse never yields a bigint, so the clause was unreachable over REST at any magnitude — a runner following it collected a 200 that proves nothing in either direction. Re-derived rather than assumed: 9007199254740993 parses to 9007199254740992, which is exactly the limit and so also inside the inclusive bound, giving two independent reasons the REST arm cannot fail. Clause 2 now points at engine-comparand-type-door.test.ts and the five driver conformance suites with oracle: test, step 3 says to pass a real bigint in-process, and a knownGap records that the REST verdict is not-applicable rather than blocked (#10236 A3)",
"ref": "#10236"
}
]
},
Expand Down
Loading
Loading