diff --git a/.changeset/calendar-day-upper-bound.md b/.changeset/calendar-day-upper-bound.md new file mode 100644 index 0000000000..53a292d78b --- /dev/null +++ b/.changeset/calendar-day-upper-bound.md @@ -0,0 +1,39 @@ +--- +"@objectstack/core": minor +"@objectstack/driver-sql": patch +"@objectstack/driver-sqlite-wasm": patch +"@objectstack/service-analytics": patch +--- + +fix(driver-sql,service-analytics): a bare-day upper bound covers the whole day on `Field.datetime` (#3777) + +A bare `YYYY-MM-DD` comparand anchors to midnight UTC. That is right for a +lower bound and was silently wrong for an upper one: the dashboard date-range +filter compiles `{ $gte: from, $lte: to }` with bare-day bounds, so on a +`datetime` column every row created after 00:00 of the `to` day vanished from +the result — no error, the chart renders, the numbers are just smaller. The +default configuration hit it: the filter's default field is `created_at` +(a system-injected `Field.datetime`) and 7 of the 13 presets end "today". + +The translation is operator-sensitive and half-open, applied at every +comparison emitter: + +- `SqlDriver` (and `SqliteWasmDriver` by inheritance): `$lte`/`<=` with a + bare-day comparand on a `datetime` column compiles to `< next-day-midnight` + in the column's storage form; `$between [min, max]` with a bare-day max + decomposes to `>= min AND < next-day(max)`. Both the plain and the + legacy-repair (mixed-storage) column paths, both `where` spellings. +- `NativeSQLStrategy`: `dateRange` windows and `lte` filters bind `< next-day` + instead of an inclusive `BETWEEN`/`<=` when the bound is a bare day. +- The `/analytics/sql` rendering and the dataset preview evaluator apply the + same rule, so the echoed SQL and drafted numbers reproduce execution. + +`@objectstack/core` gains the shared primitive `nextUtcCalendarDay(value)`: +the next calendar day of a valid bare `YYYY-MM-DD` (else `null` — instants, +`Date`s and impossible days are never widened). + +Unchanged on purpose, per the semantics table on #3777: `date`/`time` columns +(`<= day` is already whole-day-correct there), full-ISO/`Date` comparands +(instant semantics), and `$gte`/`$gt`/`$lt` (midnight anchoring is correct for +those). No authored metadata changes: a dashboard's existing +`{ $gte, $lte }` window now simply includes its final day. diff --git a/content/docs/protocol/objectql/query-syntax.mdx b/content/docs/protocol/objectql/query-syntax.mdx index cca3c6a3d3..b43acf1a61 100644 --- a/content/docs/protocol/objectql/query-syntax.mdx +++ b/content/docs/protocol/objectql/query-syntax.mdx @@ -306,6 +306,13 @@ where: { due_date: { $lt: '2024-06-01' } } where: { created_at: { $gte: '2024-01-01' } } ``` +A bare `YYYY-MM-DD` bound is a **calendar day**. As a lower bound (`$gte`) it +means the start of that day (midnight UTC); as an upper bound (`$lte`, or the +max of a `$between`) it covers the **whole** day — on a `datetime` column the +driver compiles it half-open (`< next day`), so the `$between` above includes +everything that happened on Dec 31. A full ISO timestamp keeps exact-instant +semantics on every operator. + ### Null Checks {/* os:check */} diff --git a/docs/adr/0053-date-and-datetime-semantics.md b/docs/adr/0053-date-and-datetime-semantics.md index aa07c8ab5c..8205ddc786 100644 --- a/docs/adr/0053-date-and-datetime-semantics.md +++ b/docs/adr/0053-date-and-datetime-semantics.md @@ -649,3 +649,62 @@ host's zone) is not recoverable. Regression cover: `sql-driver-time-canonical-storage.test.ts`, `sql-driver-time-of-day.test.ts` (SQLite), `sql-driver-time-live-dialects.test.ts` (live PG + MySQL, in the CI temporal-conformance job's non-UTC matrix). + +--- + +## Addendum (2026-07-30) — a bare-day UPPER bound means the whole day (#3777) + +> **Status:** landed. Extends Phase 1's calendar-day semantics with the +> operator-sensitive half the original decision left implicit — and the default +> dashboard configuration hit it: the date-range filter's default field is +> `created_at`, a system-injected `Field.datetime`, and 7 of 13 presets end +> "today", so `{ $lte: }` anchored to midnight silently dropped every +> row created after 00:00 of the final day. + +### D-D1 — Operator-sensitive translation lives at the comparison EMITTERS + +`YYYY-MM-DD` anchors to midnight UTC (D-B1). That instant is the correct +comparand for `$gte`/`$gt`/`$lt` — and the wrong one for `$lte`, whose author +means "through the whole of that day". The translation is therefore a property +of the *comparison*, not of the value: `temporalFilterValue` stays +operator-blind (form only), and each emitter that owns an operator compiles a +bare-day upper bound half-open: + +- **`SqlDriver` filter compiler** (`calendarDayUpperBoundRewrite` / + `calendarDayBetweenRewrite`): `$lte`/`<=` → `< next-day-midnight` in storage + form; `$between [min, max]` with a bare-day max decomposes to + `>= min AND < next-day(max)`. Applies on both the plain and the + CASE-normalised (D-B2) column paths, and to the Mongo-style and array + `where` spellings. `driver-sqlite-wasm` inherits. +- **`NativeSQLStrategy`** windows and `lte` filters bind `< next-day` instead + of `BETWEEN`/`<=` when the bound is a bare day. +- **`ObjectQLStrategy`** leaves its lowered `{$gte, $lte}` bounds bare — the + driver rewrite is the single execution-path authority — and renders + `/analytics/sql` half-open so the echoed SQL reproduces execution. +- **The dataset preview evaluator** applies the same rule in memory, replacing + its `'~'`-suffix string hack, so draft numbers match published numbers. + +One primitive backs all of them: `nextUtcCalendarDay` (`@objectstack/core`), +which rejects instants, `Date`s and impossible days (`2026-02-30`) rather than +inventing a bound. Half-open — never an inclusive `23:59:59.999`, which +re-opens the gap at whatever precision the dialect stores beyond milliseconds +(Postgres keeps microseconds), and is the same `[gte, lt)` shape the drill +ranges (#1752) already emit. `< next-day` is also order-equivalent to `<= day` +for `Field.date` text, which is what lets the type-blind emitters (raw SQL, +preview) apply it unconditionally; the driver, which knows the column type, +scopes the rewrite to `datetime` so `date`/`time` columns compile byte-identical +to before. + +The filter-token resolver (`filter-tokens.ts`) keeps its documented refusal to +widen: a resolver-side fix would change what a token *is*; the emitter-side fix +changes what a comparison *does* with it, per column type — which is the layer +that owns that knowledge. + +### D-D2 — Consequences for D-A3 + +The conformance matrix gains a **bound-semantics** axis (`point`, `whole-day`): +row-result coverage for the `$lte`/`$between` upper-bound cells now lives in +`sql-driver-calendar-day-upper-bound.test.ts` (canonical + legacy-mixed +storage, dialect physical forms, boundary rollovers) and the strategy/preview +suites; the full matrix program (relative-token × live-driver × timezone) +remains open under D-A3. diff --git a/packages/core/src/utils/datetime.test.ts b/packages/core/src/utils/datetime.test.ts index 577620f070..248a456ee1 100644 --- a/packages/core/src/utils/datetime.test.ts +++ b/packages/core/src/utils/datetime.test.ts @@ -6,7 +6,7 @@ // database, and midnight must land exactly on the day boundary in that zone. import { describe, it, expect } from 'vitest'; -import { zonedDateStartToUtcMs, calendarPartsInTz } from './datetime.js'; +import { zonedDateStartToUtcMs, calendarPartsInTz, nextUtcCalendarDay } from './datetime.js'; const iso = (s: string) => Date.parse(s); @@ -60,3 +60,29 @@ describe('zonedDateStartToUtcMs — round-trips to the day boundary in the zone' }); } }); + +describe('nextUtcCalendarDay — the exclusive upper bound of a bare calendar day (#3777)', () => { + it('advances one day, rolling month, year and leap boundaries', () => { + expect(nextUtcCalendarDay('2026-07-28')).toBe('2026-07-29'); + expect(nextUtcCalendarDay('2026-07-31')).toBe('2026-08-01'); + expect(nextUtcCalendarDay('2026-12-31')).toBe('2027-01-01'); + expect(nextUtcCalendarDay('2024-02-28')).toBe('2024-02-29'); // leap year + expect(nextUtcCalendarDay('2025-02-28')).toBe('2025-03-01'); + expect(nextUtcCalendarDay(' 2026-07-28 ')).toBe('2026-07-29'); // trimmed + }); + + it('returns null for anything that is not a valid bare calendar day', () => { + // Instants keep instant semantics — never widened. + expect(nextUtcCalendarDay('2026-07-28T12:00:00Z')).toBeNull(); + expect(nextUtcCalendarDay(new Date('2026-07-28T00:00:00Z'))).toBeNull(); + // Impossible days are rejected, not rolled into an invented bound. + expect(nextUtcCalendarDay('2026-02-30')).toBeNull(); + expect(nextUtcCalendarDay('2026-13-01')).toBeNull(); + // Non-strings / junk. + expect(nextUtcCalendarDay(1753660800000)).toBeNull(); + expect(nextUtcCalendarDay(null)).toBeNull(); + expect(nextUtcCalendarDay(undefined)).toBeNull(); + expect(nextUtcCalendarDay('7/28/2026')).toBeNull(); + expect(nextUtcCalendarDay('2026-7-28')).toBeNull(); // not zero-padded → not the canonical shape + }); +}); diff --git a/packages/core/src/utils/datetime.ts b/packages/core/src/utils/datetime.ts index 68939148e4..1043641a25 100644 --- a/packages/core/src/utils/datetime.ts +++ b/packages/core/src/utils/datetime.ts @@ -102,6 +102,39 @@ export function zonedDateStartToUtcMs(ymd: string, tz?: string): number { } } +/** + * The calendar day after a bare `YYYY-MM-DD` string — the exclusive upper + * bound of that day, for compiling "through day X" into the half-open + * `[X, X+1)` a `datetime` column needs (#3777). + * + * A bare calendar day used as an upper bound (`$lte`, a `dateRange` end, a + * `{current_month_end}` token) always carries whole-day intent — the author + * means "including everything that happened on X", never "up to the stroke of + * midnight that begins X". On a `date` column `<= X` already says that; on a + * `datetime` column it silently drops every instant after 00:00. The correct + * translation is the half-open pair `>= start AND < nextUtcCalendarDay(end)` + * — the same shape the analytics drill ranges already emit — rather than an + * inclusive `23:59:59.999` constant, which re-opens the gap at whatever + * precision the dialect stores beyond milliseconds. + * + * `< nextUtcCalendarDay(X)` is also *equivalent* to `<= X` for a `date` column + * (plain `YYYY-MM-DD` text ordering), so emitters that cannot see the column + * type (raw-SQL strategies, the dataset preview evaluator) can apply it + * unconditionally to a bare-day bound and be right on both column types. + * + * Returns `null` for anything that is not a valid bare calendar day — full + * ISO timestamps keep instant semantics and must NOT be widened, and an + * impossible day (`2026-02-30`) is rejected the same way + * {@link bucketKeyToCalendarRange} rejects it, so a caller falls back to the + * untranslated comparand instead of inventing a bound. + */ +export function nextUtcCalendarDay(value: unknown): string | null { + if (typeof value !== 'string') return null; + const day = value.trim(); + if (!/^\d{4}-\d{2}-\d{2}$/.test(day)) return null; + return bucketKeyToCalendarRange(day, 'day')?.end ?? null; +} + /** * Granularity of a canonical date-bucket key. Mirrors `@objectstack/spec`'s * `DateGranularity` enum but kept as a local literal union so this low-level diff --git a/packages/plugins/driver-sql/src/sql-driver-calendar-day-upper-bound.test.ts b/packages/plugins/driver-sql/src/sql-driver-calendar-day-upper-bound.test.ts new file mode 100644 index 0000000000..72292b154a --- /dev/null +++ b/packages/plugins/driver-sql/src/sql-driver-calendar-day-upper-bound.test.ts @@ -0,0 +1,273 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * Calendar-day upper bounds on `Field.datetime` (#3777) — the acceptance gate. + * + * A bare `YYYY-MM-DD` comparand anchors to midnight UTC (#3912). That is right + * for a LOWER bound and wrong for an UPPER one: the dashboard date-range + * filter compiles `{ $gte: from, $lte: to }` with bare-day bounds, so on a + * `datetime` column — and `created_at`, the filter's DEFAULT field, is a + * system-injected `Field.datetime` — every row created after 00:00 of the + * `to` day silently vanished. Seven of the thirteen presets end "today". + * + * The fix is operator-sensitive and half-open: `$lte`/`<=`/`between`-max with + * a bare-day comparand on a datetime column compiles to `< next-day-midnight` + * (`calendarDayUpperBoundRewrite`), the same `[gte, lt)` shape the analytics + * drill ranges emit. Everything else — `date` columns, full-ISO comparands, + * `$gte`/`$gt`/`$lt` — keeps its exact pre-fix behaviour, matching the + * semantics table on the issue. + * + * These tests assert ROW RESULTS (the ADR-0053 D-A3 posture), on the two + * storage states a SQLite deployment can be in: canonical text (#3912) and + * the un-backfilled mixed INTEGER-epoch / ISO-TEXT legacy column. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { SqlDriver } from '../src/index.js'; +import { LegacyStorageDriver } from '../src/legacy-datetime-storage.testkit.js'; + +const ids = (rows: any[]) => rows.map((r: any) => r.id).sort(); + +describe('bare-day $lte on Field.datetime — the #3777 repro', () => { + let driver: SqlDriver; + + beforeEach(async () => { + driver = new SqlDriver({ + client: 'better-sqlite3', + connection: { filename: ':memory:' }, + useNullAsDefault: true, + }); + await driver.initObjects([ + { + name: 'task', + fields: { + title: { type: 'string' }, + created_at: { type: 'datetime' }, + created_on: { type: 'date' }, + }, + }, + ]); + // The issue's probe, verbatim: "today" is 2026-07-28, the window is the + // last_90_days preset's expansion. `created_on` mirrors each instant's + // calendar day so every case below can run the same probe on a date column. + for (const [id, at] of [ + ['t_midnight', '2026-07-28T00:00:00Z'], + ['t_morning', '2026-07-28T09:15:00Z'], + ['t_evening', '2026-07-28T21:40:00Z'], + ['t_yesterday', '2026-07-27T14:00:00Z'], + ['t_old', '2026-04-19T10:00:00Z'], + ] as const) { + await driver.create( + 'task', + { id, title: id, created_at: new Date(at), created_on: at.slice(0, 10) }, + { bypassTenantAudit: true } as any, + ); + } + }); + + afterEach(async () => { + await driver.disconnect?.(); + }); + + it('keeps the whole final day — the dashboard default-config window', async () => { + // Exactly the shape objectui's buildFilterCondition sends for last_90_days. + const found = await driver.find('task', { + where: { created_at: { $gte: '2026-04-29', $lte: '2026-07-28' } }, + } as any); + // Pre-fix this returned only [t_midnight, t_yesterday]: the 09:15 and + // 21:40 rows fell past the midnight-anchored upper bound. + expect(ids(found)).toEqual(['t_evening', 't_midnight', 't_morning', 't_yesterday']); + }); + + it('the same probe on a Field.date column is unchanged (already whole-day)', async () => { + const found = await driver.find('task', { + where: { created_on: { $gte: '2026-04-29', $lte: '2026-07-28' } }, + } as any); + expect(ids(found)).toEqual(['t_evening', 't_midnight', 't_morning', 't_yesterday']); + }); + + it('a full-ISO $lte keeps instant semantics — only the bare day is widened', async () => { + const found = await driver.find('task', { + where: { created_at: { $lte: '2026-07-28T12:00:00.000Z' } }, + } as any); + expect(ids(found)).toEqual(['t_midnight', 't_morning', 't_old', 't_yesterday']); + }); + + it('a Date-object $lte keeps instant semantics too', async () => { + const found = await driver.find('task', { + where: { created_at: { $lte: new Date('2026-07-28T00:00:00Z') } }, + } as any); + expect(ids(found)).toEqual(['t_midnight', 't_old', 't_yesterday']); + }); + + it('$gte / $gt / $lt keep their midnight anchoring (the issue-table rows marked correct)', async () => { + const gte = await driver.find('task', { where: { created_at: { $gte: '2026-07-28' } } } as any); + expect(ids(gte)).toEqual(['t_evening', 't_midnight', 't_morning']); + + const gt = await driver.find('task', { where: { created_at: { $gt: '2026-07-28' } } } as any); + expect(ids(gt)).toEqual(['t_evening', 't_morning']); // excludes the exact-midnight row + + const lt = await driver.find('task', { where: { created_at: { $lt: '2026-07-28' } } } as any); + expect(ids(lt)).toEqual(['t_old', 't_yesterday']); + }); + + it('$between with a bare-day max covers the whole final day', async () => { + const found = await driver.find('task', { + where: { created_at: { $between: ['2026-04-29', '2026-07-28'] } }, + } as any); + expect(ids(found)).toEqual(['t_evening', 't_midnight', 't_morning', 't_yesterday']); + }); + + it('$between on a Field.date column is unchanged', async () => { + const found = await driver.find('task', { + where: { created_on: { $between: ['2026-04-29', '2026-07-28'] } }, + } as any); + expect(ids(found)).toEqual(['t_evening', 't_midnight', 't_morning', 't_yesterday']); + }); + + it('stays one grouped predicate inside an $or branch', async () => { + const found = await driver.find('task', { + where: { + $or: [ + { created_at: { $gte: '2026-07-28', $lte: '2026-07-28' } }, // "today" preset + { title: 't_old' }, + ], + }, + } as any); + expect(ids(found)).toEqual(['t_evening', 't_midnight', 't_morning', 't_old']); + }); + + it('applies to the array (`[field, op, value]`) where spelling too', async () => { + const found = await driver.find('task', { + where: [['created_at', '<=', '2026-07-28'], 'and', ['created_at', '>=', '2026-04-29']], + } as any); + expect(ids(found)).toEqual(['t_evening', 't_midnight', 't_morning', 't_yesterday']); + + const between = await driver.find('task', { + where: [['created_at', 'between', ['2026-04-29', '2026-07-28']]], + } as any); + expect(ids(between)).toEqual(['t_evening', 't_midnight', 't_morning', 't_yesterday']); + }); +}); + +describe('bare-day $lte on an un-backfilled legacy column (mixed storage)', () => { + let driver: LegacyStorageDriver; + + beforeEach(async () => { + driver = new LegacyStorageDriver({ + client: 'better-sqlite3', + connection: { filename: ':memory:' }, + useNullAsDefault: true, + }); + await driver.initObjects([ + { name: 'task', fields: { title: { type: 'string' }, created_at: { type: 'datetime' } } }, + ]); + // Both pre-#3912 storage forms of the same calendar day, in ONE column: + // an INTEGER epoch (a bound JS Date) and zone-naive TEXT (CURRENT_TIMESTAMP). + // The half-open bound must keep both — via the CASE-normalised column + // expression, the applyNormalizedComparison arm of the rewrite. + await driver.seedLegacyRows('task', 'created_at', [ + { id: 'epoch_morning', title: 'epoch', created_at: Date.parse('2026-07-28T09:15:00Z') }, + { id: 'text_evening', title: 'text', created_at: '2026-07-28 21:40:00' }, + { id: 'epoch_next_day', title: 'next', created_at: Date.parse('2026-07-29T00:00:00Z') }, + { id: 'text_old', title: 'old', created_at: '2026-04-19 10:00:00' }, + ]); + }); + + afterEach(async () => { + await driver.disconnect?.(); + }); + + it('keeps both stored forms of the final day and excludes the next midnight', async () => { + const found = await driver.find('task', { + where: { created_at: { $gte: '2026-04-29', $lte: '2026-07-28' } }, + } as any); + expect(ids(found)).toEqual(['epoch_morning', 'text_evening']); + }); + + it('$between decomposes against the normalised column the same way', async () => { + const found = await driver.find('task', { + where: { created_at: { $between: ['2026-04-29', '2026-07-28'] } }, + } as any); + expect(ids(found)).toEqual(['epoch_morning', 'text_evening']); + }); +}); + +// ── Dialect physical form of the rewritten bound (no DB connection) ───────── + +/** Test double that injects field-type metadata without a live connection. */ +class ProbeDriver extends SqlDriver { + seedDatetime(table: string, field: string): void { + (this.datetimeFields[table] ??= new Set()).add(field); + } + seedDate(table: string, field: string): void { + (this.dateFields[table] ??= new Set()).add(field); + } + rewrite(table: string, field: string, op: string, value: unknown) { + return this.calendarDayUpperBoundRewrite(table, field, op, value); + } + betweenRewrite(table: string, field: string, value: unknown) { + return this.calendarDayBetweenRewrite(table, field, value); + } +} + +function makeProbe(client: string): ProbeDriver { + return new ProbeDriver({ client, connection: { filename: ':memory:' }, useNullAsDefault: true } as any); +} + +describe('calendarDayUpperBoundRewrite — dialect and boundary matrix', () => { + it('binds next-day midnight in each dialect physical spelling', () => { + const expected: Record = { + 'better-sqlite3': '2026-07-29T00:00:00.000Z', + pg: '2026-07-29T00:00:00.000Z', + mysql2: '2026-07-29 00:00:00.000', // #3942: MySQL parses neither T nor Z + }; + for (const [client, value] of Object.entries(expected)) { + const d = makeProbe(client); + d.seedDatetime('t', 'at'); + expect(d.rewrite('t', 'at', '$lte', '2026-07-28'), client).toEqual({ op: '$lt', value }); + expect(d.rewrite('t', 'at', '<=', '2026-07-28'), client).toEqual({ op: '<', value }); + } + }); + + it('rolls month, year and leap-day boundaries as calendar arithmetic', () => { + const d = makeProbe('better-sqlite3'); + d.seedDatetime('t', 'at'); + const upper = (day: string) => (d.rewrite('t', 'at', '$lte', day) as any)?.value; + expect(upper('2026-07-31')).toBe('2026-08-01T00:00:00.000Z'); + expect(upper('2026-12-31')).toBe('2027-01-01T00:00:00.000Z'); + expect(upper('2024-02-28')).toBe('2024-02-29T00:00:00.000Z'); // leap year + expect(upper('2025-02-28')).toBe('2025-03-01T00:00:00.000Z'); + }); + + it('declines everything outside the calendar-day-on-datetime cell', () => { + const d = makeProbe('better-sqlite3'); + d.seedDatetime('t', 'at'); + d.seedDate('t', 'on'); + // date column — `<=` is already whole-day-correct there. + expect(d.rewrite('t', 'on', '$lte', '2026-07-28')).toBeNull(); + // lower bounds and strict-less keep their midnight anchor. + expect(d.rewrite('t', 'at', '$gte', '2026-07-28')).toBeNull(); + expect(d.rewrite('t', 'at', '$lt', '2026-07-28')).toBeNull(); + // instant comparands keep instant semantics. + expect(d.rewrite('t', 'at', '$lte', '2026-07-28T12:00:00Z')).toBeNull(); + expect(d.rewrite('t', 'at', '$lte', new Date('2026-07-28T00:00:00Z'))).toBeNull(); + // an impossible day is rejected, not rolled into an invented bound. + expect(d.rewrite('t', 'at', '$lte', '2026-02-30')).toBeNull(); + // non-temporal column. + expect(d.rewrite('t', 'title', '$lte', '2026-07-28')).toBeNull(); + }); + + it('between: only a [min, max] with a bare-day max on datetime decomposes', () => { + const d = makeProbe('better-sqlite3'); + d.seedDatetime('t', 'at'); + d.seedDate('t', 'on'); + expect(d.betweenRewrite('t', 'at', ['2026-04-29', '2026-07-28'])).toEqual({ + lower: '2026-04-29T00:00:00.000Z', + upper: '2026-07-29T00:00:00.000Z', + }); + expect(d.betweenRewrite('t', 'on', ['2026-04-29', '2026-07-28'])).toBeNull(); + expect(d.betweenRewrite('t', 'at', ['2026-04-29', '2026-07-28T12:00:00Z'])).toBeNull(); + expect(d.betweenRewrite('t', 'at', ['2026-04-29'])).toBeNull(); // malformed → caller's error + }); +}); diff --git a/packages/plugins/driver-sql/src/sql-driver.ts b/packages/plugins/driver-sql/src/sql-driver.ts index 5f97c28559..e97f27e21c 100644 --- a/packages/plugins/driver-sql/src/sql-driver.ts +++ b/packages/plugins/driver-sql/src/sql-driver.ts @@ -14,6 +14,7 @@ import type { IDataDriver } from '@objectstack/spec/contracts'; import { StorageNameMapping } from '@objectstack/spec/system'; import { ExternalSchemaModeViolationError } from '@objectstack/spec/shared'; import { resolveMultiOrgEnabled } from '@objectstack/types'; +import { nextUtcCalendarDay } from '@objectstack/core'; import { buildIndexName, diffManagedIndexes, @@ -4235,6 +4236,86 @@ export class SqlDriver implements IDataDriver { return this.toDateOnly(value); } + /** + * The exclusive upper-bound instant for a bare calendar-day comparand on a + * `datetime` column — next day's midnight UTC, in this dialect's storage + * form — or `null` when the calendar-day reading does not apply. + * + * This is the missing half of the calendar-day convention (#3777). A bare + * `YYYY-MM-DD` anchors to midnight UTC ({@link storageDatetimeValue}), which + * is exactly right for a LOWER bound (`>= {today}` means "from the moment + * the day starts") and exactly wrong for an UPPER bound: `<= {today}` from a + * dashboard's date-range filter means "including today", but midnight + * anchoring turns it into "up to the first instant of today", silently + * dropping every row created after 00:00 — with `created_at` (a system + * `Field.datetime`) as the filter's default field, the default dashboard + * configuration loses the current day. + * + * The translation is operator-sensitive, so it lives at the comparison + * emitters (which know their operator) rather than inside the operator-blind + * {@link coerceFilterValue}: an upper-bound `$lte`/`<=`/`between`-max with a + * bare-day comparand compiles to the half-open `< next-day-midnight` — the + * same `[gte, lt)` shape the analytics drill ranges emit — never to an + * inclusive `23:59:59.999`, which re-opens the gap at whatever precision the + * dialect stores beyond milliseconds. + * + * Deliberately narrow, mirroring the semantics table on #3777: + * - `date` / `time` / non-temporal columns → null (a bare day on a `date` + * column is already whole-day-correct under `<=`); + * - full ISO timestamps and `Date` objects → null (an instant comparand + * keeps instant semantics — only the day-granular STRING carries + * calendar-day intent); + * - `$gte` / `$gt` / `$lt` keep their midnight anchoring (correct today). + */ + protected calendarDayExclusiveUpperBound( + table: string | null, + field: string, + value: unknown, + ): unknown | null { + if (this.temporalFieldKind(table, field) !== 'datetime') return null; + const next = nextUtcCalendarDay(value); + if (next == null) return null; + return this.storageDatetimeValue(`${next}T00:00:00.000Z`); + } + + /** + * Rewrite one upper-bound comparison for calendar-day intent: `$lte`/`<=` + * with a bare `YYYY-MM-DD` on a `datetime` column becomes `$lt`/`<` against + * {@link calendarDayExclusiveUpperBound}. Returns `null` — "not applicable, + * compile as-is" — for every other operator/comparand/column combination. + */ + protected calendarDayUpperBoundRewrite( + table: string | null, + field: string, + op: string, + value: unknown, + ): { op: string; value: unknown } | null { + if (op !== '$lte' && op !== '<=') return null; + const upper = this.calendarDayExclusiveUpperBound(table, field, value); + if (upper == null) return null; + return { op: op === '$lte' ? '$lt' : '<', value: upper }; + } + + /** + * The `between` companion of {@link calendarDayUpperBoundRewrite}: a + * `[min, max]` range whose max is a bare calendar day on a `datetime` column + * decomposes into the half-open pair `>= min AND < next-day(max)` — knex's + * `whereBetween` is inclusive on both ends, so it inherits the same + * midnight-anchored upper bound `$lte` had. Returns `null` when the range is + * malformed (caller keeps its descriptive error) or the rewrite does not + * apply. + */ + protected calendarDayBetweenRewrite( + table: string | null, + field: string, + value: unknown, + ): { lower: unknown; upper: unknown } | null { + if (!Array.isArray(value) || value.length !== 2) return null; + const upper = this.calendarDayExclusiveUpperBound(table, field, value[1]); + if (upper == null) return null; + return { lower: this.coerceFilterValue(table, field, value[0]), upper }; + } + /** * Might this SQLite `Field.datetime` column still hold values written BEFORE * the canonical-UTC-text convention (#3912) — an INTEGER/REAL epoch from a @@ -4577,6 +4658,15 @@ export class SqlDriver implements IDataDriver { * This is a thin, intentionally narrow wrapper over the same `coerceFilterValue` * the driver already uses, so there is exactly one source of truth for the * storage convention and the analytics path can never drift from CRUD. + * + * Deliberately operator-blind — it translates FORM, never bound semantics. + * A caller compiling an upper bound from a bare calendar day (`<= {today}`, + * a `dateRange` end) must apply `nextUtcCalendarDay` from `@objectstack/core` + * and emit `<` — the half-open translation the driver's own `find()` path + * performs via {@link calendarDayUpperBoundRewrite} (#3777). Folding that in + * here would silently widen every `<=`-bound value whether or not the caller + * flips its operator, which is exactly the ambiguity the emitter-side rule + * avoids. */ public temporalFilterValue(objectName: string, field: string, value: any): any { return this.coerceFilterValue(objectName, field, value); @@ -4653,11 +4743,30 @@ export class SqlDriver implements IDataDriver { if (isCriterion) { const localField = this.mapSortField(fieldRaw); const field = this.remoteColumn(table, fieldRaw, localField); - const coerced = this.coerceFilterValue(table, localField, value); - this.applyAstComparison( - builder, nextJoin, field, op, value, coerced, - this.filterColumnExpr(table, localField, field), - ); + const opLower = String(op).toLowerCase(); + const columnExpr = this.filterColumnExpr(table, localField, field); + // Calendar-day upper bounds (#3777) — same translation the + // Mongo-operator path applies, for the array (`[field, op, value]`) + // spelling of the identical comparison. + const dayRange = opLower === 'between' + ? this.calendarDayBetweenRewrite(table, localField, value) : null; + if (dayRange) { + (builder as any)[nextJoin === 'or' ? 'orWhere' : 'where']((qb: any) => { + if (columnExpr) { + this.applyNormalizedComparison(qb, 'and', columnExpr, '$gte', dayRange.lower); + this.applyNormalizedComparison(qb, 'and', columnExpr, '$lt', dayRange.upper); + } else { + qb.where(field, '>=', dayRange.lower).andWhere(field, '<', dayRange.upper); + } + }); + } else { + const rewrite = this.calendarDayUpperBoundRewrite(table, localField, opLower, value); + const coerced = rewrite ? rewrite.value : this.coerceFilterValue(table, localField, value); + this.applyAstComparison( + builder, nextJoin, field, rewrite?.op ?? op, value, coerced, + columnExpr, + ); + } } else { const method = nextJoin === 'or' ? 'orWhere' : 'where'; (builder as any)[method]((qb: any) => { @@ -4895,9 +5004,29 @@ export class SqlDriver implements IDataDriver { // Non-null only for a SQLite `Field.datetime`, whose two stored forms // (INTEGER epoch / ISO TEXT) must be unified before comparing (#3912). const columnExpr = this.filterColumnExpr(table, localField, field); - for (const [op, opValue] of Object.entries(value as Record)) { + for (const [rawOp, opValue] of Object.entries(value as Record)) { const method = logicalOp === 'or' ? 'orWhere' : 'where'; - const coerced = this.coerceFilterValue(table, localField, opValue); + // Calendar-day upper bounds first (#3777): `$lte` on a bare + // `YYYY-MM-DD` against a datetime column compiles half-open, and a + // `$between` whose max is a bare day decomposes into the same pair — + // grouped, so an `$or` branch stays one predicate. + if (rawOp === '$between') { + const dayRange = this.calendarDayBetweenRewrite(table, localField, opValue); + if (dayRange) { + (builder as any)[method]((qb: any) => { + if (columnExpr) { + this.applyNormalizedComparison(qb, 'and', columnExpr, '$gte', dayRange.lower); + this.applyNormalizedComparison(qb, 'and', columnExpr, '$lt', dayRange.upper); + } else { + qb.where(field, '>=', dayRange.lower).andWhere(field, '<', dayRange.upper); + } + }); + continue; + } + } + const rewrite = this.calendarDayUpperBoundRewrite(table, localField, rawOp, opValue); + const op = rewrite?.op ?? rawOp; + const coerced = rewrite ? rewrite.value : this.coerceFilterValue(table, localField, opValue); if (columnExpr && this.applyNormalizedComparison(builder, logicalOp, columnExpr, op, coerced)) continue; switch (op) { case '$eq': diff --git a/packages/plugins/driver-sql/vitest.config.ts b/packages/plugins/driver-sql/vitest.config.ts index d8263a3634..16c0c561dc 100644 --- a/packages/plugins/driver-sql/vitest.config.ts +++ b/packages/plugins/driver-sql/vitest.config.ts @@ -16,7 +16,13 @@ export default defineConfig({ '@objectstack/spec/system': path.resolve(__dirname, '../../spec/src/system/index.ts'), // [ADR-0105 D1] Reached transitively via `@objectstack/types` (tenancy posture). '@objectstack/spec/security': path.resolve(__dirname, '../../spec/src/security/index.ts'), + // Reached transitively via `@objectstack/core` (#3777's `nextUtcCalendarDay` + // import pulls core's src barrel in, which fans out to these subpaths). + '@objectstack/spec/api': path.resolve(__dirname, '../../spec/src/api/index.ts'), + '@objectstack/spec/kernel': path.resolve(__dirname, '../../spec/src/kernel/index.ts'), + '@objectstack/spec/qa': path.resolve(__dirname, '../../spec/src/qa/index.ts'), '@objectstack/spec': path.resolve(__dirname, '../../spec/src/index.ts'), + '@objectstack/core': path.resolve(__dirname, '../../core/src/index.ts'), }, }, }); diff --git a/packages/plugins/driver-sqlite-wasm/src/sqlite-wasm-driver-calendar-day-upper-bound.test.ts b/packages/plugins/driver-sqlite-wasm/src/sqlite-wasm-driver-calendar-day-upper-bound.test.ts new file mode 100644 index 0000000000..460e4bbddc --- /dev/null +++ b/packages/plugins/driver-sqlite-wasm/src/sqlite-wasm-driver-calendar-day-upper-bound.test.ts @@ -0,0 +1,47 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * `SqliteWasmDriver extends SqlDriver`, so the calendar-day upper-bound + * translation (#3777) — a bare-day `$lte` on a `Field.datetime` column + * compiling half-open (`< next-day-midnight`) — is inherited, not + * re-implemented. This pins the inheritance the same way the date-bucket + * storage fix (#3773) pinned its own: same disease, same cure, one test that + * fails if the wasm driver ever stops sharing the seam. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { SqliteWasmDriver } from '../src/index.js'; + +describe('SqliteWasmDriver — bare-day $lte covers the whole day (#3777)', () => { + let driver: SqliteWasmDriver; + + beforeEach(async () => { + driver = new SqliteWasmDriver({ filename: ':memory:' }); + await driver.initObjects([ + { name: 'task', fields: { title: { type: 'string' }, created_at: { type: 'datetime' } } }, + ]); + for (const [id, at] of [ + ['t_midnight', '2026-07-28T00:00:00Z'], + ['t_evening', '2026-07-28T21:40:00Z'], + ['t_yesterday', '2026-07-27T14:00:00Z'], + ['t_next_day', '2026-07-29T00:00:00Z'], + ] as const) { + await driver.create( + 'task', + { id, title: id, created_at: new Date(at) }, + { bypassTenantAudit: true } as any, + ); + } + }); + + afterEach(async () => { + await (driver as any).knex.destroy(); + }); + + it('keeps the final day of a dashboard window and excludes the next midnight', async () => { + const found = await driver.find('task', { + where: { created_at: { $gte: '2026-04-29', $lte: '2026-07-28' } }, + } as any); + expect(found.map((r: any) => r.id).sort()).toEqual(['t_evening', 't_midnight', 't_yesterday']); + }); +}); diff --git a/packages/services/service-analytics/src/__tests__/native-sql-datetime-filter-column.test.ts b/packages/services/service-analytics/src/__tests__/native-sql-datetime-filter-column.test.ts index 18d9a9a583..8ba11903c5 100644 --- a/packages/services/service-analytics/src/__tests__/native-sql-datetime-filter-column.test.ts +++ b/packages/services/service-analytics/src/__tests__/native-sql-datetime-filter-column.test.ts @@ -87,8 +87,11 @@ describe('NativeSQLStrategy — datetime filter column normalisation (#3912)', ( }, withHook(), ); + // Half-open since #3777: the bare-day end means "through that whole day", + // so it binds as `< 2025-07-02`, and BOTH comparisons read the normalised + // column. expect(sql).toContain( - 'EPOCH_MS(assessed_at) BETWEEN $1 AND $2', + '(EPOCH_MS(assessed_at) >= $1 AND EPOCH_MS(assessed_at) < $2)', ); }); @@ -135,7 +138,9 @@ describe('NativeSQLStrategy — datetime filter column normalisation (#3912)', ( }; const { sql } = await gen(query, ctxWith({})); expect(sql).toContain('assessed_at >= $1'); - expect(sql).toContain('assessed_at BETWEEN $2 AND $3'); + // The window itself is half-open (#3777) even without the storage hook — + // bound semantics and storage-form normalisation are independent layers. + expect(sql).toContain('(assessed_at >= $2 AND assessed_at < $3)'); }); it('falls back to the bare column when the hook returns nothing usable', async () => { diff --git a/packages/services/service-analytics/src/__tests__/native-sql-datetime-filter.test.ts b/packages/services/service-analytics/src/__tests__/native-sql-datetime-filter.test.ts index a020591679..e7589abbaf 100644 --- a/packages/services/service-analytics/src/__tests__/native-sql-datetime-filter.test.ts +++ b/packages/services/service-analytics/src/__tests__/native-sql-datetime-filter.test.ts @@ -129,6 +129,34 @@ describe('NativeSQLStrategy — datetime filter storage coercion', () => { ]); }); + it('compiles a bare-day `lte` half-open — through the whole day (#3777)', async () => { + const strategy = new NativeSQLStrategy(); + const ctx = ctxWith({ coerceTemporalFilterValue: sqliteHook }); + const query: AnalyticsQuery = { + cube: 'compliance', + measures: ['total'], + where: { assessed: { $lte: '2025-06-18' } }, + }; + const { sql, params } = await strategy.generateSql(query, ctx); + // `<= 2025-06-18` means "including everything on June 18th": the bound is + // June 19th's midnight, compared with `<` — not midnight of the 18th. + expect(sql).toContain('assessed_at < $1'); + expect(params).toEqual([Date.parse('2025-06-19T00:00:00.000Z')]); + }); + + it('a full-ISO `lte` keeps instant semantics (no widening)', async () => { + const strategy = new NativeSQLStrategy(); + const ctx = ctxWith({ coerceTemporalFilterValue: sqliteHook }); + const query: AnalyticsQuery = { + cube: 'compliance', + measures: ['total'], + where: { assessed: { $lte: '2025-06-18T12:00:00.000Z' } }, + }; + const { sql, params } = await strategy.generateSql(query, ctx); + expect(sql).toContain('assessed_at <= $1'); + expect(params).toEqual([Date.parse('2025-06-18T12:00:00.000Z')]); + }); + it('coerces each element of an `in` set on a datetime column', async () => { const strategy = new NativeSQLStrategy(); const ctx = ctxWith({ coerceTemporalFilterValue: sqliteHook }); @@ -145,7 +173,7 @@ describe('NativeSQLStrategy — datetime filter storage coercion', () => { ]); }); - it('coerces a timeDimension dateRange (BETWEEN) on a datetime column', async () => { + it('coerces a timeDimension dateRange (half-open window) on a datetime column', async () => { const strategy = new NativeSQLStrategy(); const ctx = ctxWith({ coerceTemporalFilterValue: sqliteHook }); const query: AnalyticsQuery = { @@ -154,10 +182,15 @@ describe('NativeSQLStrategy — datetime filter storage coercion', () => { timeDimensions: [{ dimension: 'assessed', dateRange: ['2025-06-18', '2025-07-01'] }], }; const { sql, params } = await strategy.generateSql(query, ctx); - expect(sql).toContain('BETWEEN $1 AND $2'); + // Half-open since #3777: the bare-day end "2025-07-01" means through the + // whole of July 1st, so the coerced upper bound is July 2nd's midnight, + // compared with `<` — not the BETWEEN whose inclusive midnight bound + // dropped the final day's rows. + expect(sql).toContain('>= $1 AND'); + expect(sql).toContain('< $2'); expect(params).toEqual([ EPOCH_2025_06_18, - Date.parse('2025-07-01T00:00:00.000Z'), + Date.parse('2025-07-02T00:00:00.000Z'), ]); }); diff --git a/packages/services/service-analytics/src/__tests__/objectql-daterange.test.ts b/packages/services/service-analytics/src/__tests__/objectql-daterange.test.ts index 56145612b5..9e2cea2de0 100644 --- a/packages/services/service-analytics/src/__tests__/objectql-daterange.test.ts +++ b/packages/services/service-analytics/src/__tests__/objectql-daterange.test.ts @@ -345,7 +345,7 @@ describe('DatasetExecutor compareTo over the ObjectQL path (#3650)', () => { }); describe('ObjectQLStrategy.generateSql — window rendering (#3650)', () => { - it('renders the window as a parameterised BETWEEN', async () => { + it('renders the window as a parameterised half-open pair', async () => { const seen: AggOpts[] = []; const svc = makeService(seen); @@ -360,11 +360,14 @@ describe('ObjectQLStrategy.generateSql — window rendering (#3650)', () => { // The preview used to omit the window deliberately, because `execute()` // dropped it. Now that the window applies, omitting it would be the lie in - // the other direction. - expect(sql).toContain('close_date BETWEEN $1 AND $2'); + // the other direction — and since #3777 the render is half-open, because + // that is what execute()'s driver actually runs for a bare-day `$lte` on a + // datetime column; a BETWEEN would hand a debugger SQL that drops the + // final day's rows. + expect(sql).toContain('(close_date >= $1 AND close_date < $2)'); expect(sql).toContain("date_trunc('month', close_date)"); // Bounds bind as parameters — the echoed string travels to the browser. - expect(params).toEqual(['2026-01-01', '2026-02-28']); + expect(params).toEqual(['2026-01-01', '2026-03-01']); expect(sql).not.toContain('2026-01-01'); }); @@ -380,8 +383,8 @@ describe('ObjectQLStrategy.generateSql — window rendering (#3650)', () => { timeDimensions: [{ dimension: 'close_date', dateRange: ['2026-01-01', '2026-01-31'] }], }); - expect(sql).toContain('close_date BETWEEN $2 AND $3'); - expect(params).toEqual(['won', '2026-01-01', '2026-01-31']); + expect(sql).toContain('(close_date >= $2 AND close_date < $3)'); + expect(params).toEqual(['won', '2026-01-01', '2026-02-01']); }); }); diff --git a/packages/services/service-analytics/src/__tests__/preview-evaluator.test.ts b/packages/services/service-analytics/src/__tests__/preview-evaluator.test.ts index 171f41a542..a18ff6e845 100644 --- a/packages/services/service-analytics/src/__tests__/preview-evaluator.test.ts +++ b/packages/services/service-analytics/src/__tests__/preview-evaluator.test.ts @@ -151,6 +151,46 @@ describe('evaluateAnalyticsQueryOverRows', () => { expect(matchesWhere({ a: 'Hello World' }, { a: { $contains: 'world' } })).toBe(true); }); + it('a bare-day $lte covers the whole day on a timestamp value (#3777)', () => { + // Same translation the SQL paths apply — the preview must agree, or a + // drafted chart shows different numbers than the published one. + expect(matchesWhere({ at: '2026-07-28T21:40:00.000Z' }, { at: { $lte: '2026-07-28' } })).toBe(true); + expect(matchesWhere({ at: '2026-07-29T00:00:00.000Z' }, { at: { $lte: '2026-07-28' } })).toBe(false); + // A plain date value is unchanged (string ordering makes the two forms + // equivalent there). + expect(matchesWhere({ on: '2026-07-28' }, { on: { $lte: '2026-07-28' } })).toBe(true); + expect(matchesWhere({ on: '2026-07-29' }, { on: { $lte: '2026-07-28' } })).toBe(false); + // Full-ISO bounds keep instant semantics. + expect( + matchesWhere({ at: '2026-07-28T21:40:00.000Z' }, { at: { $lte: '2026-07-28T12:00:00.000Z' } }), + ).toBe(false); + }); + + it('a timeDimension dateRange keeps the final day of the window on timestamps (#3777)', () => { + const rows = [ + { spent_at: '2026-05-31T09:15:00.000Z', amount: 70 }, + { spent_at: '2026-05-31T00:00:00.000Z', amount: 5 }, + { spent_at: '2026-06-01T00:00:00.000Z', amount: 900 }, + ]; + const r = evaluateAnalyticsQueryOverRows( + { + measures: ['total_amount'], + dimensions: ['spent_at'], + timeDimensions: [{ dimension: 'spent_at', granularity: 'month', dateRange: ['2026-05-01', '2026-05-31'] }], + }, + { + name: 'expenses', + sql: 'expenses', + dimensions: { spent_at: { sql: 'spent_at', type: 'time' } }, + measures: { total_amount: { sql: 'amount', type: 'sum' } }, + } as unknown as Cube, + rows, + ); + // Pre-fix the 09:15 row survived only via the `'~'`-suffix trick; the + // half-open bound keeps it AND excludes June 1st's midnight exactly. + expect(r.rows).toEqual([{ spent_at: '2026-05', total_amount: 75 }]); + }); + it('helpers: bucketDate resolves the calendar day in a reference timezone', () => { // 2024-03-01T03:00Z is still 2024-02-29 in America/New_York. const near = '2024-03-01T03:00:00.000Z'; diff --git a/packages/services/service-analytics/src/preview-evaluator.ts b/packages/services/service-analytics/src/preview-evaluator.ts index e1c31843ea..e027c51159 100644 --- a/packages/services/service-analytics/src/preview-evaluator.ts +++ b/packages/services/service-analytics/src/preview-evaluator.ts @@ -17,7 +17,7 @@ // Anything beyond (joins via `include`, raw SQL) falls back to the caller's // normal execution path — the preview simply doesn't claim it. -import { calendarPartsInTzOrUtc } from '@objectstack/core'; +import { calendarPartsInTzOrUtc, nextUtcCalendarDay } from '@objectstack/core'; import type { AnalyticsQuery, AnalyticsResult } from '@objectstack/spec/contracts'; import type { Cube } from '@objectstack/spec/data'; @@ -37,7 +37,17 @@ function matchOp(value: unknown, op: string, expected: unknown): boolean { case '$gt': return value != null && compare(value, expected) > 0; case '$gte': return value != null && compare(value, expected) >= 0; case '$lt': return value != null && compare(value, expected) < 0; - case '$lte': return value != null && compare(value, expected) <= 0; + case '$lte': { + if (value == null) return false; + // A bare-day upper bound means "through that whole day" (#3777): the SQL + // paths compile it half-open (`< day+1`), and the preview must agree or + // a drafted chart shows different numbers than the published one. String + // ordering makes `< nextDay` equivalent to `<= day` for plain date + // values, so no type lookup is needed here either. + const nextDay = nextUtcCalendarDay(expected); + if (nextDay != null) return compare(value, nextDay) < 0; + return compare(value, expected) <= 0; + } case '$in': return Array.isArray(expected) && expected.some((e) => value === e || String(value) === String(e)); case '$nin': return Array.isArray(expected) && !expected.some((e) => value === e || String(value) === String(e)); case '$contains': return String(value ?? '').toLowerCase().includes(String(expected ?? '').toLowerCase()); @@ -133,7 +143,12 @@ export function evaluateAnalyticsQueryOverRows( const [start, end] = Array.isArray(td.dateRange) ? td.dateRange : [td.dateRange, td.dateRange]; filtered = filtered.filter((r) => { const v = String(r[field] ?? ''); - return v >= String(start) && v <= `${end}~`; // '~' > any date char: inclusive end-day + // Bare-day end → half-open `< day+1`, the same translation the SQL + // paths apply (#3777); a full-timestamp end keeps the historical + // `'~'`-suffix trick (inclusive of that instant's own sub-values). + const nextDay = nextUtcCalendarDay(end); + const inUpper = nextDay != null ? v < nextDay : v <= `${end}~`; + return v >= String(start) && inUpper; }); } diff --git a/packages/services/service-analytics/src/strategies/native-sql-strategy.ts b/packages/services/service-analytics/src/strategies/native-sql-strategy.ts index a9a35543c3..8b86d78b10 100644 --- a/packages/services/service-analytics/src/strategies/native-sql-strategy.ts +++ b/packages/services/service-analytics/src/strategies/native-sql-strategy.ts @@ -5,6 +5,7 @@ import type { Cube } from '@objectstack/spec/data'; import type { AnalyticsStrategy, StrategyContext } from './types.js'; import { normalizeAnalyticsFilters, coerceFilterValueForSql } from './filter-normalizer.js'; import { compileScopedFilterToSql } from '../read-scope-sql.js'; +import { nextUtcCalendarDay } from '@objectstack/core'; /** * NativeSQLStrategy — Priority 1 @@ -124,13 +125,24 @@ export class NativeSQLStrategy implements AnalyticsStrategy { // BOTH forms at once and coercing only the bounds still empties the // half the writer stored the other way (#3912). const td2 = this.resolveStorageTarget(cube, td.dimension, tableName); - params.push( - this.coerceTemporal(ctx, td2, range[0]), - this.coerceTemporal(ctx, td2, range[1]), - ); - whereClauses.push( - `${this.temporalColumn(ctx, td2, colExpr)} BETWEEN $${params.length - 1} AND $${params.length}`, - ); + const column = this.temporalColumn(ctx, td2, colExpr); + // A bare-day window end means "through that whole day" (#3777). A + // BETWEEN's inclusive upper bound anchors a bare `YYYY-MM-DD` to + // midnight on a datetime column, dropping the final day's rows, so + // the window compiles half-open — `>= start AND < end+1day` — the + // same `[gte, lt)` the drill ranges emit. Equivalent to the old + // BETWEEN for a `date` column (plain `YYYY-MM-DD` ordering), which + // is what lets this path stay column-type-blind. + const nextDay = nextUtcCalendarDay(range[1]); + params.push(this.coerceTemporal(ctx, td2, range[0])); + const lower = `${column} >= $${params.length}`; + if (nextDay != null) { + params.push(this.coerceTemporal(ctx, td2, nextDay)); + whereClauses.push(`(${lower} AND ${column} < $${params.length})`); + } else { + params.push(this.coerceTemporal(ctx, td2, range[1])); + whereClauses.push(`(${lower} AND ${column} <= $${params.length})`); + } } } } @@ -497,6 +509,17 @@ export class NativeSQLStrategy implements AnalyticsStrategy { return `${rawCol} ${sqlOp} $${params.length}`; } + // A bare-day `lte` bound means "through that whole day" (#3777): compile + // half-open (`< day+1`) so a datetime column keeps the final day's rows. + // Equivalent to `<=` for a `date` column, so no column-type lookup needed. + if (operator === 'lte') { + const nextDay = nextUtcCalendarDay(values[0]); + if (nextDay != null) { + params.push(this.coerceTemporal(ctx, target, nextDay)); + return `${this.temporalColumn(ctx, target, rawCol)} < $${params.length}`; + } + } + // Coerce so booleans/numbers bind as their native SQL types AND so a // relative-date / ISO-string comparand on a SQLite `Field.datetime` // column is converted to its INTEGER epoch storage form. Without this a diff --git a/packages/services/service-analytics/src/strategies/objectql-strategy.ts b/packages/services/service-analytics/src/strategies/objectql-strategy.ts index b683b63a7b..945d80b5ec 100644 --- a/packages/services/service-analytics/src/strategies/objectql-strategy.ts +++ b/packages/services/service-analytics/src/strategies/objectql-strategy.ts @@ -5,6 +5,7 @@ import type { Cube } from '@objectstack/spec/data'; import type { AnalyticsStrategy, StrategyContext } from './types.js'; import { normalizeAnalyticsFilters, coerceFilterValueForObjectQL } from './filter-normalizer.js'; import { compileScopedFilterToSql } from '../read-scope-sql.js'; +import { nextUtcCalendarDay } from '@objectstack/core'; import { rebucketCrossObject, RECOMBINABLE_METHODS, @@ -297,9 +298,16 @@ export class ObjectQLStrategy implements AnalyticsStrategy { } // Bounds bind as `$n` placeholders like every other comparand: this string // travels to the browser, and a window can carry tenant-derived dates. + // A bare-day upper bound renders half-open (`< day+1`) because that is + // what `execute()`'s driver actually runs for it on a datetime column + // (#3777) — rendering the BETWEEN would hand a debugger SQL that drops + // the final day's rows and cannot reproduce the result. for (const { field, bounds } of this.dateRangeBounds(cube, query)) { - params.push(bounds.$gte, bounds.$lte); - whereParts.push(`${field} BETWEEN $${params.length - 1} AND $${params.length}`); + const nextDay = nextUtcCalendarDay(bounds.$lte); + params.push(bounds.$gte, nextDay ?? bounds.$lte); + whereParts.push( + `(${field} >= $${params.length - 1} AND ${field} ${nextDay ? '<' : '<='} $${params.length})`, + ); } // Read scope last, so it reads as the outermost constraint. Compiled by the // same fail-closed compiler `NativeSQLStrategy` uses — it throws rather than @@ -749,9 +757,13 @@ export class ObjectQLStrategy implements AnalyticsStrategy { * HERE on every driver — and "bucketed trend" is precisely the shape that also * carries a range ("last 12 months", "this quarter"). * - * Bounds are inclusive on both ends — the same `$gte`/`$lte` pair - * `NativeSQLStrategy` binds as `BETWEEN` and the memory driver builds as a - * `$match`, so one dashboard reads the same on every driver. + * Bounds are inclusive on both ends — logically "from day X through day Y". + * The `$lte` end is left as the bare calendar day on purpose: the driver's + * filter compiler owns the calendar-day → instant translation, compiling a + * bare-day `$lte` on a `datetime` column into the half-open `< nextDay` + * (#3777) while a `date` column keeps the plain `<=`. `NativeSQLStrategy` + * performs the same half-open translation itself because it binds into raw + * SQL, so one dashboard reads the same on every driver. * * Comparands are coerced by the SAME helper the `where` path uses, so an * epoch-ms bound recovers as a number and an ISO string stays a string. No