Skip to content

feat(objectql,cli): os migrate 新增 summary count/sum 存量 NULL 回填迁移 (#6063) - #6158

Merged
baozhoutao merged 4 commits into
mainfrom
claude/issue-6063-summary-null-backfill
Aug 7, 2026
Merged

feat(objectql,cli): os migrate 新增 summary count/sum 存量 NULL 回填迁移 (#6063)#6158
baozhoutao merged 4 commits into
mainfrom
claude/issue-6063-summary-null-backfill

Conversation

@baozhoutao

Copy link
Copy Markdown
Contributor

Fixes#6063

前提复核(P1-P3,实读 origin/main)

P1 —— #6013 的口径确如裁决描述。成立。

packages/objectql/src/engine.ts(origin/main,合并前行号):

  • 829: function summaryEmptySetValue(fn) { return fn === 'count' || fn === 'sum' ? 0 : null; } —— count/sum 的空集值是 0,min/max/avg 是 null,原文自陈是「the ONE place that list is written down」;
  • 4350: private initializeSummaryFields(...) 在 insert 路径上按该表播种,文档串 4347 行原文:「Existing rows are untouched: this is create-time only, so parents already stored with null stay null until a child write recomputes them.」
  • 4410: if (value == null) value = summaryEmptySetValue(desc.fn); —— 重算落到同一张表。
  • 测试侧:summary-rollup.test.ts:286 起的 roll-up summary seeding on the PARENT insert (#5749) 整块,含 375leaves avg/max null — undefined on an empty set

⇒ 裁决描述的「insert 起 count/sum 为 0、min/max/avg 保持 null」逐字成立。

P2 —— 存量 NULL 父行在子写入前不会被任何既有路径补齐。成立。

recomputeSummaries 的父行集合只来自子记录:idsrecs / prevsdesc.fkField 构成(engine.ts 4372 起),没有第三个来源;initializeSummaryFields 是 create-time only(上引 4347)。全仓再无别处写这些列。行为侧也钉住了:本 PR 新测 dry run reports the same rows and writes nothing 里,直写驱动 store 造出的存量父行在回填前读回仍是 null

P3 —— 迁移家族结构如裁决引用。成立。

  • 命令壳:packages/cli/src/commands/migrate/{value-shapes,files-to-references}.ts;
  • 真迁移在归属包:scanValueShapes@objectstack/objectql,runFilesToReferencesMigration@objectstack/service-storage;
  • 显式跑 + 产出证据:dry-run 默认、--apply 才写、报告函数 format*Report 由命令壳打印。

三条前提全部为真,未改道。

裁决引用(issue #6063,2026-08-07 02:07Z)

  1. 显式 os migrate,不开机自动跑 —— 落 os migrate summary-nulls,命令壳在 packages/cli/src/commands/migrate/summary-nulls.ts(薄),真迁移在 objectql;
  2. 逐行重算聚合,便宜路线默认禁用 —— 本 PR 采用便宜路线,亦尝试证伪反例:恰恰相反,新测第 1 例正是「NULL 但有子记录」的活体,便宜路线在它上面翻红(见下反向验证);
  3. 引擎层驱动无关循环 —— 走引擎 find / aggregate / update,零逐驱动 SQL,且不下推 IS NULL 谓词(理由见下)。

处置

新增

  • packages/objectql/src/summary-backfill.ts —— backfillSummaryNulls(engine, logger, options) + formatSummaryBackfillReport + summaryBackfillComplete。遍历「拥有 count/sum 汇总列」的对象,keyset 走全部父行(只取 id + 汇总列),对读回值为 null 的行逐行重算并写回。dry-run 默认;幂等;单行失败记录后继续。
  • packages/objectql/src/summary-aggregate.ts —— 从 engine.ts原样搬出的 roll-up 内核:SummaryDescriptorsummaryEmptySetValue、单描述符聚合 aggregateSummaryValue,外加派生谓词 summaryNullIsBackfillable。descriptor 新增 childObject 字段(父侧索引此前拿不到子对象名,因为子侧索引就是按它做 key)。
  • packages/cli/src/commands/migrate/summary-nulls.ts —— 照 files-to-references 的壳:occupancy gate(它改行)、--apply / --yes / --force / --object / --max-records / --json、空元数据拒跑。
  • packages/objectql/src/summary-backfill.test.ts —— 10 例。
  • changeset(objectql + cli,patch)、content/docs/deployment/cli.mdx 的数据迁移小节。

为什么不写 UPDATE ... SET col = 0 WHERE col IS NULL

它不只是粗,它是错的:升级前就有子记录的父行同样是 NULL(从没有人重算过它),正确值是真实聚合而不是 0。写 0 等于把「可见地缺失」换成「自信地错误」,而下一次子写入又会把它悄悄改回去 —— 用户什么都没动,列却变了。

为什么不下推 IS NULL

null 谓词的编译正是各驱动分歧最大的地方(sql-driver-null-operators.test.tssql-driver-out-of-contract-filter-input.test.ts 的分歧表)。一条覆盖范围取决于「碰巧跑在哪个驱动上」的迁移,等于会静默漏行的迁移。读回值在 JS 里判 == null,是唯一在所有驱动上含义相同的写法;代价见「性能口径」。

不记 migration flag

两个兄弟迁移记 flag,是因为那面 flag 是后续不可逆行为的开关。本迁移只修值、不改任何 posture,幂等重跑本身就是验证(报告归零即证),记一面没人读的 flag 反而是噪音。已写进模块与命令的文档串。

测试与反向验证

新测(10 例,均绿)

✓ gives a NULL parent WITH children its real aggregate — not 0
✓ gives a NULL parent with NO children the empty-set value 0
✓ leaves min/max/avg NULL exactly as they are, and reports them as out of scope
✓ is idempotent — the second run finds nothing and writes nothing
✓ is a no-op on a database whose rows were all created with the seed (a fresh install)
✓ dry run reports the same rows and writes nothing
✓ never overwrites a value already stored — including a deliberate 0
✓ writes the SAME value the engine's own child-write recompute would
✓ restricts to the objects it is given
✓ records a row it cannot write and carries on with the rest

存量状态是直写驱动 store 造出来的 —— 那正是原地升级库的样子(父行早于播种存在,且从没有子写入访问过它)。走引擎 insert 造反而会被 #6013 播成 0,就没得测了。

反向验证(方向先写死,再运行)

肢 A —— 把回填临时改成便宜路线 computed = 0

  • 预测:用例 1「NULL 且子记录 → 真实聚合」翻红。
  • 实测:翻红,与预测一致,并且多红一例(方向相同,幅度更大):
× gives a NULL parent WITH children its real aggregate — not 0
→ expected +0 to be 2
× writes the SAME value the engine's own child-write recompute would
→ expected 42 to be +0
Tests 2 failed | 8 passed (10)

第二例是意外收获,也是本单最该被看见的伤害形态:便宜路线写了 0 之后,下一次子写入的重算把列改回 42 —— 用户没动任何东西,列自己变了。同源同算的钉子就是钉这个。已恢复原实现,上述 10 例复绿。

肢 B —— 注册接线。

  • 预测:迁移未注册时 os migrate 列表不含它;注册后含。
  • 实测,两个方向都跑了(oclif 按 dist/commands/**/*.js glob 发现,所以「未注册」= 把编译产物移走):
# 移走 dist/commands/migrate/summary-nulls.js
$ node bin/run.js migrate --help | grep -c summary-nulls
0
$ node bin/run.js migrate summary-nulls --help
› Error: Command migrate:summary-nulls not found.
# 放回
$ node bin/run.js migrate --help | grep -c summary-nulls
1

注册后的列表项(真实输出节选):

 migrate summary-nulls Backfill roll-up count/sum summary columns still
stored as NULL on parent rows created before the
insert-time seed (#5749). Dry-run by default;
--apply recomputes and writes each affected row.

命令与真实输出

合并 origin/main(⛔ 未 rebase)后重跑,origin/main 带进了 #5979@objectstack/metadata/errors 新依赖边,worktree 补跑了一次 pnpm install(与本 PR 无关,记录在此以免下一位读者误判):

$ pnpm --filter @objectstack/objectql test -- --maxWorkers=2
Test Files 133 passed (133)
Tests 2190 passed (2190)
$ pnpm --filter @objectstack/objectql typecheck
> tsc --noEmit # 无输出即通过
$ pnpm --filter @objectstack/cli test -- --maxWorkers=2
Test Files 88 passed (88)
Tests 880 passed (880)
$ pnpm --filter @objectstack/cli typecheck
> tsc --noEmit
$ node scripts/check-engine-double-contract.mjs # exit=0
check-engine-double-contract: OK — 73 pinned, 133 in the DEBT ledger, 2 exempt.
$ node scripts/check-nul-bytes.mjs
check-nul-bytes: OK (scanned 5882 tracked text file(s); ... no raw ASCII control bytes).
$ pnpm check:query-options-erasure
✓ query-options-erasure ratchet holds: 83 unswept non-test site(s) in 19 file(s), none new.
$ pnpm check:doc-authoring
✓ doc authoring guard: 363 files clean — no bare metadata literals.
$ pnpm check:empty-changeset
✓ No empty-frontmatter changeset introduced by this diff (1 declaring changeset(s) added).
$ pnpm check:durability-log-level
✓ durability-degradation log levels: 24 durability-critical catch seam(s), all loud ...

check:query-options-erasure 中途红过一次并已修:搬运时把 as any 一起带了过来,该闸门拒绝新文件进 baseline(grandfather 列表只缩不增)。修法是把它去掉而不是加白名单 —— 新接口 SummaryAggregateEngine.aggregate 直接收该 query,本来就不需要断言。baseline 最终只记录 engine.ts 自己的计数 13 → 12。

必答项

1. #6013 口径一致性:回填值 / insert 初值 / 重算值三者同源同算?

是,且是结构性同源,不是「今天恰好相等」。 三个调用点读同一份代码:

  • insert 初值:ObjectQL.initializeSummaryFieldssummaryEmptySetValue(desc.fn);
  • 子写入重算:ObjectQL.recomputeSummariesaggregateSummaryValue(this, desc, parentId, execCtx),该函数内部空集回退同样是 summaryEmptySetValue(desc.fn);
  • 本回填:backfillSummaryNulls同一个aggregateSummaryValue(engine, desc, ...)

descriptor 也同源:回填不自己解析 summary 元数据,而是调 engine.getOwnedSummaryDescriptors(object)引擎自己的索引(同一份 FK 解析、同一份 filter、同一条 objectRevision 失效规则)。范围收窄谓词 summaryNullIsBackfillable(fn)summaryEmptySetValue(fn) !== null 派生,不另列函数名单,所以两者不可能各说各话。

行为侧还额外钉了一道:测试 writes the SAME value the engine's own child-write recompute would —— 回填后再触发一次真实子写入重算,断言列不动。肢 A 证明这枚钉子会红。

2. #5979 / #5351 / #6014 交叠

未触其面。 逐条:

engine.ts 的 hunk 逐个点名(共 4 处,全部落在 roll-up 机制内部,与上述两单的落点 seedAutonumber / buildDriverOptions 不相邻):

  1. import 段:新增 ./summary-aggregate.js 的三个符号;
  2. SummaryDescriptor + summaryEmptySetValue 定义处 → 换成一条指向新文件的说明注释(纯搬运,零语义改动);
  3. buildSummaryIndex 的 descriptor 字面量:补 childObject(值就是本来当索引 key 用的那个变量);
  4. getOwnedSummaryDescriptors 去掉 private;recomputeSummaries 内联的聚合块 → 调 aggregateSummaryValue

共享 index packages/objectql/src/index.ts 也碰了(导出新迁移面),追加式,不改既有行。

之所以不是「新文件里再写一遍聚合」:那正是 scan-value-shapes.ts 头注释「Why the predicate is imported」写下的教训 —— 第二份实现只要差一个从句,迁移写进去的值就和引擎下一次重算的值不一致。必答项 1 要求「必须同源」,搬运是唯一能给出代码证据的做法。

3. 多租户形态

value-shapes / files-to-references 完全一致,不新造机制:

  • 读写都带 { isSystem: true }不带 tenantIdbuildDriverOptions(engine.ts)只在 execCtx.tenantId !== undefined 时才下发租户作用域,所以本迁移在共享库多 org 形态下一次覆盖库内所有 org 的行 —— 这正是部署级回填需要的;
  • 按 org 分库 / 分表 形态下,与两个兄弟迁移一样:每个库跑一次,用 --database-url(或 OS_DATABASE_URL)指定。命令的 occupancy gate、确认提示与报告都是按「一次一个目标库」设计的;
  • 迁移只跑「已加载元数据里拥有 count/sum 汇总列的对象」,无对象加载时直接拒跑(空扫和干净扫不可区分)。

4. 性能口径

  • 聚合次数 = O(存量 NULL 的 父行 × count/sum 汇总列 对数),与裁决口径一致。已回填的行第二次跑不再产生任何聚合(幂等),所以量级只随「真实缺口」走,不随库大小走;
  • 读取 = O(父行总数),分页 keyset(pageSize 500,只取 id + 汇总列)。这是为了不下推 IS NULL 而付的代价,与 scanValueShapes(500/页)、backfillFileReferences(200/页)同形,先例照抄;
  • 批处理策略:先例没有批量聚合的形态,本迁移也不引入 —— 单行 aggregate + 单行 update,外加与引擎重算同款的 withTransientRetry(bug(objectql): 汇总重算失败仅 warn——子记录批量插入报成功,父 summary 静默过期且无重试 #3147)。这样一行失败只损失一行,而且重跑只会回访仍是 NULL 的行;
  • 兜底:--max-records 可为每个对象设上限,超限报告 truncated,不假装跑完;--object 可只跑已知受影响的对象;
  • 量级估算:10 万父行 / 2 个汇总列、其中 1% 为 NULL 的库,一次运行约 20 万行读(400 页)+ 2000 次聚合 + 2000 次 update;第二次运行退化为纯 400 页读、零写。真正大的库建议先 dry-run 看 N NULL row(s) 再决定窗口。

顺带

无越界发现,未新开 issue。⛔ 未碰 content/docs/releases/(known-issue 文案由发布流程从 issue 取材)。


Generated by Claude Code

…count/sum roll-ups (#6063)
PR #6013 (#5749) seeds a roll-up's empty-set value at parent INSERT, which
reaches new rows only: a database upgraded in place keeps pre-upgrade parents
at NULL, because the recompute that would fix them runs only when one of their
children is written. Those rows keep vanishing from `= 0` filters, sorts,
GROUP BY and formulas. This adds the one-off, explicit data migration for them.
- `backfillSummaryNulls` (packages/objectql/src/summary-backfill.ts): walk each
object owning a count/sum roll-up, and recompute every row whose column is
stored NULL. A pre-upgrade parent WITH children is NULL too and its correct
value is the real aggregate, so `SET col = 0 WHERE col IS NULL` is wrong, not
merely coarse. Dry run by default; idempotent; driver-agnostic (values are
read and tested in JS, no null predicate pushed down); one row's failure is
recorded and the run continues.
- min/max/avg are never touched: undefined on an empty set, so a stored null
there is the correct reading of "no child rows". The report names them as
deliberately skipped.
- `summary-aggregate.ts`: SummaryDescriptor, summaryEmptySetValue and the
single-descriptor aggregate lifted out of engine.ts unchanged, so the seed,
the recompute and the backfill share ONE computation instead of three that
agree until one is edited. The descriptor gains `childObject` so the
parent-side index is usable on its own.
- `os migrate summary-nulls`: thin oclif shell over the migration, following
the files-to-references precedent (occupancy gate, --apply/--yes, --object,
--max-records, --json). No deployment flag — nothing is gated on this run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Q7oc7ASjh8yxyS3Yz78We
…ing it (#6063)
`check:query-options-erasure` rejects a NEW file in its baseline — the
grandfather list only shrinks. The `as any` came along with the code lifted out
of engine.ts and is not needed there: `SummaryAggregateEngine.aggregate` takes
the query directly, so the cast is dropped and the baseline records only
engine.ts's own count falling 13 -> 12.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Q7oc7ASjh8yxyS3Yz78We
@vercel

vercelBot commented Aug 7, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
ProjectDeploymentActionsUpdated (UTC)
objectstackIgnoredIgnoredAug 7, 2026 4:07am

Request Review

@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/cli, @objectstack/objectql.

31 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:

  • content/docs/ai/skills-reference.mdx(via packages/cli)
  • content/docs/api/client-sdk.mdx(via @objectstack/cli)
  • content/docs/api/data-flow.mdx(via @objectstack/cli)
  • content/docs/api/environment-routing.mdx(via @objectstack/cli)
  • content/docs/api/error-catalog.mdx(via @objectstack/cli)
  • content/docs/automation/hook-bodies.mdx(via packages/cli)
  • content/docs/concepts/metadata-lifecycle.mdx(via @objectstack/objectql)
  • content/docs/data-modeling/formulas.mdx(via packages/objectql)
  • content/docs/deployment/backup-restore.mdx(via @objectstack/cli)
  • content/docs/deployment/cli.mdx(via @objectstack/cli)
  • content/docs/deployment/migration-from-objectql.mdx(via @objectstack/objectql)
  • content/docs/deployment/self-hosting.mdx(via @objectstack/cli)
  • content/docs/deployment/validating-metadata.mdx(via packages/cli)
  • content/docs/deployment/vercel.mdx(via @objectstack/objectql)
  • content/docs/getting-started/your-first-project.mdx(via @objectstack/cli)
  • content/docs/kernel/runtime-services/data-service.mdx(via @objectstack/cli)
  • content/docs/kernel/runtime-services/examples.mdx(via packages/objectql)
  • content/docs/kernel/runtime-services/index.mdx(via packages/cli)
  • content/docs/kernel/services-checklist.mdx(via @objectstack/objectql)
  • content/docs/kernel/services.mdx(via @objectstack/objectql)
  • content/docs/permissions/authentication.mdx(via @objectstack/cli, @objectstack/objectql)
  • content/docs/plugins/index.mdx(via @objectstack/cli, @objectstack/objectql)
  • content/docs/plugins/packages.mdx(via @objectstack/cli, @objectstack/objectql)
  • content/docs/protocol/kernel/index.mdx(via @objectstack/objectql)
  • content/docs/protocol/kernel/plugin-spec.mdx(via @objectstack/cli)
  • content/docs/protocol/kernel/realtime-protocol.mdx(via @objectstack/cli)
  • content/docs/protocol/objectql/query-syntax.mdx(via packages/objectql)
  • content/docs/protocol/objectql/state-machine.mdx(via @objectstack/objectql)
  • content/docs/releases/implementation-status.mdx(via @objectstack/cli, @objectstack/objectql)
  • content/docs/releases/v16.mdx(via @objectstack/cli)
  • content/docs/releases/v17.mdx(via @objectstack/cli)

Advisory only. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs origin/main → pass the list as args.docs.

@github-actionsgithub-actionsBot added size/xl documentation Improvements or additions to documentation tests tooling labels Aug 7, 2026
…g it to any (#6063)
The slot-lookup rule (#4168/#4251) rejects `const engine: any =
getService('objectql')` — the sibling migrate commands are silent only because
they are grandfathered by file, and that baseline only shrinks. The command's
real requirement is `SummaryBackfillEngine` (the slot contract plus the one
member the backfill reads), so name that type: the call site keeps its
checking and the lookup is not erased.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Q7oc7ASjh8yxyS3Yz78We
@baozhoutaoClaude

Copy link
Copy Markdown
ContributorAuthor

补记:第二道本地闸门也红过一次并已修(正文「命令与真实输出」只记了第一道)

npx eslint 对新命令壳报 slot-lookup 规则(#4168/#4251):

packages/cli/src/commands/migrate/summary-nulls.ts
175:13 error Do not erase a service-lookup result to `any` ... no-restricted-syntax

来源是照抄兄弟命令的 const engine: any = stack.kernel.getService('objectql') —— 那两个文件之所以安静,是因为它们在 scripts/slot-lookup-baseline.json按文件豁免,而该 baseline 只缩不增,新文件进不去。

修法不是加白名单,而是把这个调用点真正需要的类型写出来:命令需要的是「slot 契约 + 迁移读的那一个成员」,即 SummaryBackfillEngine(objectql 已导出),于是

importtype{SummaryBackfillEngine}from'@objectstack/objectql';
...
constengine: SummaryBackfillEngine=stack.kernel.getService('objectql');

import type 在编译期擦除,重的引擎包仍然是下面那句 await import('@objectstack/objectql') 惰性加载的。

复跑(真实输出):

$ npx eslint packages/objectql/src/summary-{aggregate,backfill}.ts \
packages/objectql/src/summary-backfill.test.ts \
packages/cli/src/commands/migrate/summary-nulls.ts
ESLINT_EXIT=0
$ pnpm check:slot-lookup
✓ slot-lookup ratchet holds: 143 unswept site(s) in 32 file(s), none new.
baseline key set verified against 811c30c: no files added.
$ pnpm --filter @objectstack/cli typecheck && pnpm --filter @objectstack/cli build
> tsc --noEmit # 均无输出即通过
$ pnpm --filter @objectstack/cli test -- --maxWorkers=2
Test Files 88 passed (88)
Tests 880 passed (880)

肢 B(注册接线)在这次重建后复验仍成立:node bin/run.js migrate --help 列表含 migrate summary-nulls

两道本地闸门红→绿的记录都留在这里,是因为它们各自指向同一条纪律:grandfather 名单不是静音键,新文件该做的是把类型写对,不是把自己加进豁免。


Generated by Claude Code

@baozhoutaoClaude

Copy link
Copy Markdown
ContributorAuthor

ACCEPT(执行席 PM 验收,冻结期收尾单)

核过:① 三取舍裁定逐条落地 —— 显式 os migrate summary-nulls(dry-run 默认 + --apply,occupancy gate 与证据报告照 files-to-references 先例)、逐行重算(便宜路线被用例 1 钉死,肢 A 还多红出一条同方向更重的证据:写 0 后下次子写入重算把列改回真值,「用户没动、列自己变了」)、引擎层驱动无关(JS 侧判 null 避开 IS NULL 下推的驱动分歧 —— 这个取舍点找得准);② 偏离 1 裁定:采纳 —— 「不触 engine.ts」与「同源同算给代码证据」在本单里真实冲突,dev 选了正确性:纯搬运抽出 summary-aggregate.ts(头注释把三个调用点与漂移后果写成了下一个读者可查证的推导),4 hunk 全在 roll-up 机制内、与在飞 #5351 落点不相邻、合并零冲突、语义零变化由 summary-rollup 全绿 + CI 双向兜底;③ 两道本地闸门中途红的修法都选了「修根因而非加白名单」(去 as any / 写出真类型),留痕完整;④ 幂等、min/max/avg 原样并报告 out of scope、租户形态照家族先例、未记 migration flag 的取舍已文档化;⑤ 必答项四条全有代码级证据。

⚠️ 与在飞 #5351 批的 import 段可能有 hunk 邻接 —— 后落地方按纪律 merge origin/main + merge-tree 预检(其派发令已含)。

翻 ready + auto-merge,进队列(rc.5 口径)。known-issue 文案的「待 #6063 迁移」从此有了具体指令名:os migrate summary-nulls --apply


Generated by Claude Code

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/xlteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

summary count/sum 存量 NULL 行的一次性回填 —— #5749 方案 1 落地后的遗留半边(原地升级的库仍漏行)

2 participants

@baozhoutao@claude