Skip to content

test(dogfood): record the measured cross-organization reach of the automation toggle - #10996

Merged
os-elon merged 2 commits into
mainfrom
claude/issue-10243-toggle-tenant-scope-probe
Aug 22, 2026
Merged

test(dogfood): record the measured cross-organization reach of the automation toggle#10996
os-elon merged 2 commits into
mainfrom
claude/issue-10243-toggle-tenant-scope-probe

Conversation

@os-elon

Copy link
Copy Markdown
Collaborator

Part of #10243 — this is the measurement half of that card. It changes no behaviour and rules nothing.

What was open, and what this measures

The card splits into two facts, and only the second was ever open:

factstatus before this PR
POST /api/v1/automation/:name/toggle is reachable by any authenticated caller with no authoring capabilityalready MEASURED and PINNED by the audit block in packages/runtime/src/domains/automation-write-capability-gate.test.ts
flow enablement is environment-scoped, so one tenant's toggle reaches every organizationopen — asserted from the scoping measured for flow definitions, never reproduced for enablement

This PR adds one dogfood test that reproduces the second over HTTP and records the result.

The read-back table, as measured

Harness: bootStack(crm, { automation: true, multiTenant: 'posture-only' }) — a real, non-degraded isolated tenancy posture. Three principals: two org owners who each create their own organization (different activeOrganizationIds, asserted not assumed), plus the seeded platform admin. Flow: the CRM app's own shipped crm_convert_lead_wizard.

Tenant A is unentitled, and the gated neighbours prove it in the same session: PUT /meta/object/crm_lead answers 403 FORBIDDEN, and POST /automation and DELETE /automation/:name answer 403 PERMISSION_DENIED.

Tenant A then calls POST /automation/crm_convert_lead_wizard/toggle with { enabled: false } and gets 200:

principalbaselineafter tenant A toggled it offafter tenant A toggled it back on
tenant A (actor, org A)enabled: trueenabled: false
tenant B (unrelated tenant, org B)enabled: trueenabled: falseenabled: true
platform admin (org-less)enabled: trueenabled: falseenabled: true

It reproduces. The effect is symmetric in both directions.

Why this is not vacuous, stated up front

multiTenant: 'posture-only' activates the tenancy posture and no row wall — the enterprise multi-org runtime is cloud-private and genuinely absent from this workspace. A fixture that booted this way and asserted isolation would assert nothing. The mirror image is the trap for this file: in a stack with no wall, "tenant B saw tenant A's write" is true of everything.

What keeps it honest is that the bit under test never reaches the plane a wall operates on. An organization wall scopes rows. The enabled bit is not a row: toggleFlow(name, enabled) writes the engine's in-process flowEnabled map keyed by flow name and nothing else, and getFlowRuntimeStates() reads that same map with no caller, no organization, and no argument at all. One test leg measures exactly that discriminator over HTTP — after the toggle the flow's persisted status is still active while its runtime enabled is false, and the automation service is one instance for the whole environment. That leg is what would fail, loudly, if enablement ever became org-stamped state a wall could scope.

Separately measured and reported on the card: the override is process-local and does not survive a cold boot on the same database file.

The fence this PR keeps

  • ⛔ No production change. isFlowAuthoringWrite and every policy predicate, gate and route are untouched — the diff is one new test file.
  • ⛔ No severity grading and no argument for one. Whether toggle belongs in the manage_metadata write set is a product and security decision for the maintainer.
  • The test asserts the measured reality, not a policy opinion. When the ruling lands, this file is one of the two places it lands (the other being the existing audit pin); a ruling that toggle is an authoring write flips these expectations to a 403, and that flip is the point — an unrecorded verdict cannot be revisited.

skip-changeset: the diff is one test file in a private package and publishes nothing.

Verification

At ce6a2862f7, clean tree:

  • pnpm --filter @objectstack/dogfood exec vitest run test/automation-toggle-tenant-scope.dogfood.test.tsTest Files 1 passed (1), Tests 6 passed (6)
  • pnpm --filter @objectstack/dogfood typecheck — exit 0
  • Gate union derived at the final commit with node scripts/pm/dispatch-gates.mjs (no paths), all green, quoting each gate's own verdict line: check:slot-lookup ("baseline key set verified against 047ac86: no files added"), check:query-options-erasure ("test surface: 240 site(s) in 47 file(s) — at the ceiling"), check:where-matcher ("276 matcher(s) discovered, 276 answer the combinator battery correctly or refuse it loudly"), check:engine-double-contract ("OK — 377 pinned, 133 in the DEBT ledger, 2 exempt"), check:type-check-coverage ("OK — 64/77 workspace packages type-checked"), check:type-check-debt ("OK — 33 ledger entr(ies) re-measured in 202.8s, 1908 raw tsc error(s) total, none above its recorded number", after a full closure build), check:test-source-alias, check:type-source-resolution, check:nul-bytes, check:plugin-teardown-shape, check-ci-filter-parity, docs-audit/check-affected-docs, and the four packages/spec liveness families.

⛔ Draft on purpose. Do not merge, do not arm auto-merge, do not mark ready — the result is evidence for the maintainer's ruling, not a fix.


Generated by Claude Code

…tomation toggle
#10243 splits into an exposure half — already measured and pinned by #10145's
`automation-write-capability-gate.test.ts` audit block — and a blast-radius
half that had never been reproduced over HTTP: whether flow ENABLEMENT is
environment-scoped such that one tenant's toggle reaches every organization.
This is that measurement, recorded. On a real, non-degraded `isolated` posture
with three principals (two org owners in two different organizations, plus the
platform admin), tenant A's `POST /automation/:name/toggle {enabled:false}`
answers 200 and both tenant B and the platform admin read the flow back as
`enabled: false`; re-enabling is symmetric.
The file records reality and rules nothing. Whether toggle belongs in the
`manage_metadata` write set is the maintainer's decision; when it is made, this
file is where it lands alongside the existing audit pin.
Part of #10243
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019bmVFqoQPq63zhKrxdYG1r
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

Nothing in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 0 changed package(s)), so this run has no opinion about the docs.

@os-elon
os-elon marked this pull request as ready for review August 22, 2026 15:43
@os-elonClaude

Copy link
Copy Markdown
CollaboratorAuthor

Enqueued — marked ready for review by the domain:cli lane seat, on the maintainer's word that this one is green. ⛔ Not merged; the maintainer merges.

⚠️ The green was re-measured, because the old one was stale

The verification in the PR body ran at ce6a2862f7 on base 7ab286e44a. Main had since moved ~20 commits to 1c3a46f876, so "green" was a statement about a base that no longer existed — and a dogfood test that boots the whole CRM stack is exactly the kind of thing an intervening merge can break. Rather than report the old result, I refreshed the branch from main (merge, not rebase) and let CI re-run.

Head is now abe4da180a, base 1c3a46f876, mergeable_state: clean.

Six required checks, read BY NAME on abe4da180a — and by lane, not by roll-up

checkverdictran for
TypeScript Type Check · workspacesuccess55s
TypeScript Type Check · debt ledgersuccess4m59s
TypeScript Type Check · source gatessuccess2m04s
TypeScript Type Check · consumer gatessuccess2m49s
Lint & Repo Gatessuccess8m24s
Test Core (6 shards)success58s – 10m17s
Dogfood Regression Gate (3 shards)success2m37s / 2m38s / 9m21s
Build Coresuccess54s
Temporal Conformance (live PG + MySQL)success3m35s

The four Type Check lanes are quoted individually because the roll-up completes in ~3s whether or not the lanes did anything — reading the roll-up alone cannot distinguish a pass from a no-op. All four ran in minutes. Likewise the Dogfood Regression Gate shards ran 2–9 minutes rather than being excused by a path filter, which matters here: the diff is a dogfood test, so a filtered skip would have meant nothing exercised it.

⛔ No other red gate. One note on Check Changeset: the pre-refresh head carried a failed run of it from 03:40:47Z, before Auto Label applied skip-changeset; a later run superseded it as skipped. On the refreshed head there is a single clean skipped, so the stale red is gone rather than merely outvoted. Console Pin Gate and Build Docs are skipped by path filter on a test-only diff — stating that explicitly rather than counting them as passes.

⛔ The fence this PR set for itself still stands — enqueueing did NOT move it

This is the measurement half of #10243. It changes no behaviour: one new test file, isFlowAuthoringWrite and every policy predicate, gate and route untouched.

It grades nothing, and neither do I. Whether POST /api/v1/automation/:name/toggle belongs in the manage_metadata write set is a product and security decision, and it is the maintainer's alone. What merging this buys is that the measured reality is on the record — tenant A's toggle reaching tenant B and the org-less platform admin, symmetric in both directions — so that when the ruling lands, the file it flips is already there. An unrecorded verdict cannot be revisited.

⚠️ Note for whoever implements that ruling: if toggle is ruled an authoring write, this file's expectations flip to 403, and that flip is the point of the file rather than a regression in it.


Generated by Claude Code

@os-elon
os-elon added this pull request to the merge queueAug 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor

⛔ merge queue 构建失败 — 先分诊,再决定要不要重排

队列构建 32589382595 红了。队列跑的是全量套件(PR 侧 CI 只跑 affected 子集),
所以失败的测试可能在本 PR 没碰过的包里 —— 那不是重排能修的。每次盲目重排都会让排在后面的所有 PR 重建一轮。

失败的 job(日志抽取,best effort):

  • Console Pin Gate — 失败步骤: Build the Console SPA at the pinned objectui SHA

    ✗ Build failed in 5.84s
    

↳ 失败原因 是判读的关键:超时Test timed out in … / Hook timed out in …)多半是负载/时序,不是本 PR 的回归;
断言AssertionError: …)才指向真实的行为改变。两者的 FAIL 行长得一模一样,只有这一行能区分。

跨 PR 相同签名(24h,按失败测试文件聚合):

  • ⚠️本次没有可用的聚合签名(日志里没有能解析出测试文件名的 FAIL 行)—— 这不是「没有同签名的其他 PR」,是这一轮没测到。跨 PR 聚合本次不可用,请手工比对其他 PR 的同类评论。
  • ⚠️ 24h 评论账本没读完(超过 5 页仍未读到窗口尽头),所以上面的「不同 PR 数」是下界,不是全量。

历史信号:

  • 本 PR 过去 24h 无队列失败记录(首次)。
  • 过去 24h 队列共有 83 个失败构建(不含本次)。

分诊清单:

  1. 失败测试在本 PR 改动的包里 → 真回归,修 PR。
  2. 失败测试与本 PR 无关 → 看上面的「跨 PR 相同签名」;已有汇总 issue ⇒ flaky/环境问题实锤,去那张 issue 上谈,修好前重排只会再烧一轮全队列。
  3. 两者都不是 → 可能与同组 PR 语义冲突;等前面的 PR 落地或失败出队后再重排一次即可,不要连续重排。

Generated by Claude Code · merge-queue-triage workflow (#4859)

@github-merge-queue
github-merge-queueBot removed this pull request from the merge queue due to failed status checks Aug 22, 2026
@os-elonClaude

Copy link
Copy Markdown
CollaboratorAuthor

队列踢出 —— 签名读数收据,据此重投一次

18:15Z 前后本 PR 掉出合并队列:既不在 origin/main 上,队列分支也已清空。⛔ 未做条件反射式重投;按纪律先认签名,读数如下。

签名:队列重建的连带取消,不是红

判据读数
本 PR 名下的 merge_group run零个 —— 32 条 check run 全部是 15:18–15:29Z 的 PR-context run,队列窗口(17:57Z 之后)名下没有产生任何 run
同批次同伴的结果四条全部 success 并落地#110906d441e41f9 · #110855b39785927 · #110806e54ac09fc · #11075aea1e6467d
本 PR 自身状态opendraft: false,head 未变(abe4da180a
队列分支git ls-remote --heads origin 'refs/heads/gh-readonly-queue/*' → 空

三条同时成立 = 连带取消的签名。没有任何证据指向本 PR 的内容:它名下压根没跑过 merge_group 检查,所以不存在「红了」这回事,而同批四条走完同一条队列全部成功,排除了批次共因。

⚠️ 明确记下这不是什么:不是 flaky 重试,不是「没落地就重投」。若这次重投后再次被踢,⛔ 停止重投,按签名四分支重判 —— 同一 PR 第二次被踢不再适用本条。

入队前的门禁读数未变

六项必需检查此前已在 head abe4da180a逐名核对绿,读的是 lane 与 shard 而非 roll-up:Type Check · workspace 55s / · debt ledger 4m59s / · source gates 2m04s / · consumer gates 2m49s · Lint & Repo Gates 8m24s · Test Core 六个 shard 全绿 · Dogfood Regression Gate 三个 shard(首个 9m21s)· Build Core 54s · Temporal Conformance (live PG + MySQL) 3m35s。Check Changesetskippedskip-changeset,本 diff 只有一个测试文件、不发布任何产物)。head 自那时起未动过,读数依然成立。

⛔ 与本卡裁决无关

重投只关乎落地通路。#10243 的 severity 裁决仍然完全开放,且是维护者的 —— 本 PR 是该卡的测量半边,不改任何行为、不给任何定级:它把「一个租户的 toggle 触达全部组织、且双向对称」这一实测事实记录下来,好让日后的裁决有据可查、可被推翻。⛔ 合入它不代表该行为被接受。


Generated by Claude Code

@os-elon
os-elon added this pull request to the merge queueAug 22, 2026
Merged via the queue into main with commit 02b4123Aug 22, 2026
33 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

⛔ merge queue 构建失败 — 先分诊,再决定要不要重排

队列构建 32591821258 红了。队列跑的是全量套件(PR 侧 CI 只跑 affected 子集),
所以失败的测试可能在本 PR 没碰过的包里 —— 那不是重排能修的。每次盲目重排都会让排在后面的所有 PR 重建一轮。

失败的 job(日志抽取,best effort):

  • Console Pin Gate — 失败步骤: Build the Console SPA at the pinned objectui SHA

    ✗ Build failed in 6.26s
    

↳ 失败原因 是判读的关键:超时Test timed out in … / Hook timed out in …)多半是负载/时序,不是本 PR 的回归;
断言AssertionError: …)才指向真实的行为改变。两者的 FAIL 行长得一模一样,只有这一行能区分。

跨 PR 相同签名(24h,按失败测试文件聚合):

  • ⚠️本次没有可用的聚合签名(日志里没有能解析出测试文件名的 FAIL 行)—— 这不是「没有同签名的其他 PR」,是这一轮没测到。跨 PR 聚合本次不可用,请手工比对其他 PR 的同类评论。
  • ⚠️ 24h 评论账本没读完(超过 5 页仍未读到窗口尽头),所以上面的「不同 PR 数」是下界,不是全量。

历史信号:

  • ⚠️本 PR 过去 24h 已在队列失败 1 次(不含本次)。 内容未变而反复失败 ⇒ 高度怀疑 flaky 测试或与同组 PR 的语义冲突,重排不解决。
  • 过去 24h 队列共有 90 个失败构建(不含本次)。

分诊清单:

  1. 失败测试在本 PR 改动的包里 → 真回归,修 PR。
  2. 失败测试与本 PR 无关 → 看上面的「跨 PR 相同签名」;已有汇总 issue ⇒ flaky/环境问题实锤,去那张 issue 上谈,修好前重排只会再烧一轮全队列。
  3. 两者都不是 → 可能与同组 PR 语义冲突;等前面的 PR 落地或失败出队后再重排一次即可,不要连续重排。

Generated by Claude Code · merge-queue-triage workflow (#4859)

@os-elon
os-elon deleted the claude/issue-10243-toggle-tenant-scope-probe branch August 22, 2026 18:58
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/mskip-changesetPR has no user-facing published change; bypasses the changeset gatetests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@os-elon@claude