Uh oh!
There was an error while loading. Please reload this page.
fix(driver-sql): SQLite 上 Field.datetime 按存储的实例分桶,不再整体塌进一个 (null) 桶 (#3773) - #3775
Merged
Conversation
…nt (#3773) On SQLite every trend chart bucketed by day/week/month/year over a `Field.datetime` column put every record in a single `(null)` bucket — one bar carrying the whole total. The measure was right; only the bucket key was wrong. better-sqlite3 stores a `Field.datetime` as INTEGER epoch milliseconds, and `buildDateBucketExpr` emitted a flat `strftime('%Y-%m', col)`. SQLite reads a bare integer as a Julian day number, and epoch ms is far outside the legal range, so `strftime` returned NULL for every row. Nothing downstream noticed: SQLite advertises `queryDateGranularity.month`, so `engine.aggregate` pushes the bucketing down, and its in-memory fallback only engages for an unsupported granularity or a non-UTC timezone. The SQLite expression is now storage-aware, sharing one `isEpochStoredDatetime` predicate with the filter-comparand coercion added for the same root cause in #2034 — a window and a bucket that disagree about storage is exactly how an epoch column ended up correctly filtered and then entirely bucketed as NULL. Postgres and MySQL are untouched and pinned as such: `defineColumn` maps `Field.datetime` to a native timestamp there. Two details are load-bearing and each has a test that fails without it: - The conversion dispatches on each stored value's type, not just the declared one. A SQLite `Field.datetime` column is genuinely mixed-form — `formatInput` passes datetime values through, so a `Date` lands as INTEGER while an ISO string lands as TEXT. Dividing TEXT by 1000 coerces it to its leading year, filing live rows under 1970 — worse than the NULL it replaces. - Division is `/1000.0`, not `/1000`: integer division truncates toward zero, so a pre-1970 instant would surface a day late. `bucketDateValue` (the in-memory fallback) now reads a finite number as epoch ms. `new Date(String(1767225600000))` is an Invalid Date, so fixing only the driver would have traded one wrong answer for two different ones — the two paths have to label the same instant identically for a drill-down to survive crossing them. Coverage goes through `initObjects` rather than `knex.schema.createTable`, which is why the existing date-bucket suite never saw this: its fixture is ISO TEXT, the half `strftime` parses natively. Four granularities x both storage forms, plus a mixed-form column, a pre-1970 instant, and dialect gating for pg/mysql. `SqliteWasmDriver` inherits the expression, so it carried the bug and is pinned too. Co-Authored-By: Claude <noreply@anthropic.com>
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Contributor
📓 Docs Drift CheckThis PR changes 3 package(s): 18 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes#3773. 续 #3650 / #3766 —— 那个 PR 修掉的是「窗口被丢弃」,把这条分桶 bug 原样 pin 成了 KNOWN GAP 断言;本 PR 是那条断言的兑现。
验收门
sql-driver-aggregate-datetime-window.test.ts里那条断言,从坏值翻成真值:先红后绿:翻完断言、未动实现前跑,
expected { null: 330 } to deeply equal { '2026-01': 300, '2026-02': 30 }。根因
better-sqlite3 把
Field.datetime存成 INTEGER epoch 毫秒,而buildDateBucketExpr对 SQLite 产出的是裸的strftime('%Y-%m', col)。SQLite 把裸 INTEGER 当 Julian day number 解释,epoch-ms 远超合法范围 →strftime对每一行都返回 NULL。没有任何一层察觉:SQLite 声明了
queryDateGranularity.month,所以engine.aggregate必然把分桶下推给 driver;它的内存兜底只在「granularity 未被声明」或「非 UTC 时区」时才接管,两条都不成立。修法是让 SQLite 表达式感知存储形态,和 #2034 为同一个根因加的 filter 比较值强转共用同一个
isEpochStoredDatetime谓词 —— 窗口和分桶对存储形态各执一词,正是「窗口筛对了、分桶全塌成 NULL」的由来。两个承重细节(各自有一条没它就红的测试)
1. 按每行实际存储的类型分派,而不是只看声明类型。
SQLite 的
Field.datetime列是真·混合形态:formatInput不碰 datetime 值,所以 JSDate落成 INTEGER,而 ISO 字符串(含未解析的defaultValue: 'NOW()')落成 TEXT —— 这正是读路径normalizeSqliteDatetimeOutput已经在修的那套混合。对 TEXT 除以 1000 会把它强转成打头的年份,'2026-01-10T…'/1000.0= epoch 后 2.026 秒,把活数据归档到 1970 —— 比它替换掉的那个 NULL 更糟。所以CASE typeof(...)是承重的,不是保险。2.
/1000.0而非/1000。整数除法向零截断,
-1/1000 = 0,一个 1969-12-31T23:59:59.999Z 的实例会浮到 1970-01-01 —— 日、季、年三个粒度全错,且只在负 epoch 上错。改成整数除法跑,9 条测试红。选 julian day 归一(而不是每处都写
strftime(fmt, x/1000.0, 'unixepoch'))是为了得到一个可复用标量,能原样塞进任何格式串,包括要引用列两次的 quarter 表达式。40000 天的午夜往返扫过零漂移。parity:两条分桶路径必须给出同一个标签
bucketDateValue(objectql 内存兜底)对数值 epoch-ms 走new Date(String(1767225600000))→ Invalid Date →'(null)'。只修 driver 不修它,等于把「两条路一起错」换成「两条路各错各的」——sql-driver-date-bucket.test.ts顶上那句 "覆盖
新增
sql-driver-date-bucket-storage.test.ts,走driver.initObjects([{fields: {x: {type: 'datetime'}}}])—— 现有 date-bucket 套件看不见这个 bug,正是因为它用knex.schema.createTable+t.string('ts')建表,存的全是 ISO TEXT,恰好是strftime能直接解析的那一半。bucketDateValue参考实现算出(不是手打字符串),所以断的是 parity 本身;金额取 2 的幂,某个桶的和直接指认是哪几行落进去的。typeof(closed_at) ∈ {integer, real}/typeof(closed_on) = text—— 万一 better-sqlite3 哪天改了绑定方式,这条先红并解释后面所有红。/1000.0。sql-driver-temporal-dialect.test.ts补方言门:pg 保持(??)::timestamptz、mysql 保持convert_tz,都不含unixepoch//1000;外加一条「??占位符数 == bindings 数」的结构门(quarter 表达式引用列 2 次,epoch 版 6 次,错位就是 knex 静默把 binding 塞进错误的槽)。driver-sqlite-wasm继承同一个buildDateBucketExpr,所以同病同愈,单独钉一条。Postgres / MySQL
不受影响,并已用测试钉住:
defineColumn把Field.datetime映射成原生 timestamp(table.timestamp),这也正是temporalFilterValue在这两个方言上不做 epoch 强转的原因。未实测(本机无 PG/MySQL 实例,以下是从列映射推的):真碰上 bigint 列(external 表拿
datetime声明一个整数列),PG 会直接拒掉bigint::timestamptz强转报错 —— 是 SQLite 没给的那种响亮失败;MySQL 的convert_tz在这种情况下会返回 NULL,但托管表上不会出现这种列。一处未修的既有分歧
SQL 路径把 NULL 分桶产出 SQL NULL,内存路径产出字符串
'(null)'。这条在 TEXT 列上同样成立,与本 PR 无关、也未被本 PR 改变,在测试里就地标注,没有顺手扩大范围。测试
driver-sql355 ✅ /objectql1128 ✅ /driver-sqlite-wasm126 ✅ /service-analytics267 ✅;turbo build(带 dts,非OS_SKIP_DTS)三包通过;eslint 干净。