Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 95 additions & 0 deletions .changeset/dataset-reference-resolution.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
---
'@objectstack/lint': minor
---

Resolve an ADR-0021 dataset's own references — base object, `include[]`,
`dimensions[].field` / `measures[].field`, and filter KEYS — at
`validate`/`build` (#14105)

A dataset could name a **base object that does not exist**, join a
**relationship that does not exist**, and bind every dimension and measure to
**fields that do not exist**, and `objectstack validate` exited **0** with
`✓ Validation passed`. `objectstack build` also exited 0 and wrote the dangling
dataset into `dist/objectstack.json`.

The sting was that the author-time rule pass **already walked those exact
nodes**. Measured on published 17.2.0, each mutation applied on its own and
confirmed on disk before running:

| mutation | before | after |
|:----------------------------------------------------|:-------|:------|
| dimension `field` → a base field that does not exist | passed | `dataset-field-unknown` |
| dimension `field` → a joined field that does not exist | passed | `dataset-field-unknown` |
| measure `field` → a field that does not exist | passed | `dataset-field-unknown` |
| measure filter KEY → a field that does not exist | passed | `dataset-filter-field-unknown` |
| `include[]` → a relationship that does not exist | passed | `dataset-include-unknown` |
| `object` → an object that does not exist | passed | `object-reference-unknown` |

The two controls in that measurement — a duplicate measure name
(`DatasetSchema.superRefine`) and a bad date macro in a **measure filter**
(`filter-token-unknown`) — both failed the build, so datasets were
demonstrably in the validation path the whole time. `filter-token-unknown`
already stood at `datasets[1].measures[1].filter.last_update_at.$lt` and
reasoned about the **value**; nothing standing in that same position resolved
the **key**, or the sibling `field` one level up.

This matters more for a dataset than for most metadata because a dataset is the
semantic layer: dashboards and reports bind its dimensions and measures by name
(ADR-0021), and the consumer end of that binding is already guarded
(`widget-dataset-unknown` / `widget-dimension-unknown` / `widget-measure-unknown`,
#7529/#8902). So the surviving hole was the quiet one — every binding resolves,
the board renders, and the charts are empty or subtly wrong because the dataset
underneath addresses columns that do not exist.

**Five verdicts, all `error`.** Four are new rule ids on a new suite member,
`validateDatasetReferences`:

- `dataset-include-unknown` — an `include[]` entry that resolves to nothing, or
to a field that is not a relationship, so no join can be derived from it.
- `dataset-field-unknown` — a dimension or measure `field` path that resolves to
no column, on the base object or on any joined object along the path.
- `dataset-field-not-included` — the second real check: a dotted path that
RESOLVES, but whose relationship prefix was never declared in `include`.
ADR-0021 D-C joins only declared paths, so the column is out of the query's
reach however real it is.
- `dataset-filter-field-unknown` — a filter key on `Dataset.filter` or
`measures[].filter`, in any of the three authored filter shapes.

The fifth, the base object itself, lands on `validateObjectReferences` as a new
`datasets[].object` reference site rather than as a sixth id here. That rule's
charter IS object-name references that are plain `z.string()`, and putting it
there buys the curated cross-package severity ladder: the platform's own
`system.datasets.ts` declares five datasets over `sys_*` objects, three of which
live in packages a stack compiling plugin-auth alone cannot see. All five
resolve through `PLATFORM_PROVIDED_OBJECT_NAMES`; a local "not in this stack ⇒
error" check would have reported every one of them. When the base object does
not resolve, `validateDatasetReferences` skips the dataset entirely, so one typo
yields one finding rather than one per dimension, measure and filter key.

**Skips, so a finding is never a guess** (ADR-0072 D1): an object this stack
does not define, an object with no readable field map (ADR-0015 `external` and
introspected schemas), a registry-injected system column, and any hop *through*
one — an injected `owner_id` is a lookup at the registry whose target is
invisible here, so `owner_id.name` is unanswerable rather than a miss. The
shipped `showcase_task_metrics` dimension `{ field: 'created_at' }` is that skip's
live case, and every shipped dataset in the repo is silent under the new rule.

**Two reusable seams ship with it**, newly exported, because the same two
questions are asked at a dashboard widget's filter keys and `sortBy` and at a
list view's field positions, and three independent copies of a hop-walker drift:

- `object-graph.ts` — `indexObjectGraph` / `resolveFieldPath` / `isUnjudgeable`,
answering "what does this `relationship[.relationship].field` path resolve to?"
as a discriminated **verdict** union rather than a boolean, so a caller can
tell "this hop is not a relationship" from "this leaf does not exist" and write
the right prescription. Plus `nearestName` / `suggestName` / `listNames`.
- `walkFilterFieldKeys` (`filter-walk.ts`) — the FIELD-KEY half of a filter
subtree, beside the subtree-finding half that module already owns. It handles
all three authored shapes (Mongo condition object, `{ field, operator, value }`
rules, `[field, op, value]` triples), because a reader that handles only one
shape is the exact bug #3574 was filed against, and it composes a nested
condition object into one relationship path so `{ account: { region: … } }`
reports `account.region` rather than a bare `region` resolved against the
wrong object.

Both hold mechanism only — no rule ids, no severities, no findings.
132 changes: 132 additions & 0 deletions packages/lint/src/filter-walk.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -35,8 +35,21 @@
* they resolve an additional vocabulary (`AppContextSelector` ids such as
* `{active_package}`) that is meaningless in a filter, and restricting the walk
* is what holds false positives at zero.
*
* ## The two halves of a filter, and where each one is answered
*
* `walkAuthoredFilters` finds the SUBTREES. Inside one, a rule wants either the
* VALUES (`validate-filter-tokens.ts` classifies placeholders in them;
* `validate-preset-comparands.ts` judges ordering comparands) or the FIELD KEYS
* — the names the query is filtered BY. {@link walkFilterFieldKeys} is the
* second half (#14105), and it is here rather than in its first caller because
* the shape dispatch is identical to the one `validate-preset-comparands.ts`
* already performs on values: the platform authors filters three ways, and a
* reader that handles only one shape is the exact bug #3574 was filed against.
*/

import { VALID_AST_OPERATORS } from '@objectstack/spec/data';

/** Any plain metadata record. */
type AnyRec = Record<string, unknown>;

Expand DownExpand Up@@ -162,3 +175,122 @@ export function walkAuthoredFilters(
});
}
}

// ── The FIELD-KEY half of a filter subtree (#14105) ──────────────────────────

/** One field position inside an authored filter. */
export interface FilterFieldKey {
/**
* The field the condition filters BY, exactly as authored — a bare name
* (`status`), or a dotted relationship path (`account.region`, whether
* spelled that way or reached by descending a nested condition object).
*/
field: string;
/** Config path of the position, e.g. `datasets[1].measures[1].filter.last_update_at`. */
path: string;
}

/** Recursion guard — an authored filter is a bounded document, not a graph. */
const MAX_KEY_DEPTH = 32;

function isPlainObject(v: unknown): v is AnyRec {
return !!v && typeof v === 'object' && !Array.isArray(v) && !(v instanceof Date);
}

/**
* Emit the field positions of one Mongo-style condition NODE.
*
* `prefix` carries the relationship path accumulated by descending nested
* condition objects, so `{ account: { region: { $eq: 'emea' } } }` reports the
* single position `account.region` rather than a bare `region` that would
* resolve against the wrong object. A node whose value carries `$` operators is
* a leaf condition and ends the descent; a `$`-prefixed key that is not a
* recognised combinator is skipped and NOT descended, matching
* `validate-preset-comparands.ts` — an unrecognised operator's operand shape is
* not ours to guess.
*/
function conditionFieldKeys(
node: AnyRec,
path: string,
prefix: string,
visit: (key: FilterFieldKey) => void,
depth: number,
): void {
if (depth > MAX_KEY_DEPTH) return;
for (const [key, value] of Object.entries(node)) {
const here = `${path}.${key}`;
if (key === '$and' || key === '$or') {
if (Array.isArray(value)) {
value.forEach((arm, i) => {
if (isPlainObject(arm)) conditionFieldKeys(arm, `${here}[${i}]`, prefix, visit, depth + 1);
});
}
continue;
}
if (key === '$not') {
if (isPlainObject(value)) conditionFieldKeys(value, here, prefix, visit, depth + 1);
continue;
}
if (key.startsWith('$')) continue; // unrecognised combinator
const field = prefix ? `${prefix}.${key}` : key;
if (isPlainObject(value) && !Object.keys(value).some((k) => k.startsWith('$'))) {
// Nested relation / deep equality — the field position is one level
// deeper, so descend rather than reporting the intermediate hop twice.
// An EMPTY nested object addresses nothing further; report the hop itself.
if (Object.keys(value).length > 0) {
conditionFieldKeys(value, here, field, visit, depth + 1);
continue;
}
}
visit({ field, path: here });
}
}

/**
* Emit every FIELD KEY inside one authored filter subtree, whatever shape it
* was authored in — the key half of what `validate-preset-comparands.ts` does
* for values, and the traversal `filter-token-unknown` already performs while
* reasoning only about the strings it finds.
*
* Holds no judgement: it does not know which object the filter is bound to and
* emits no findings. Resolution is {@link resolveFieldPath}'s job
* (`object-graph.ts`) and the verdict is the caller's.
*/
export function walkFilterFieldKeys(
node: unknown,
path: string,
visit: (key: FilterFieldKey) => void,
depth = 0,
): void {
if (depth > MAX_KEY_DEPTH) return;

if (Array.isArray(node)) {
// Triple: ['field', op, value] — the field position is a non-keyword
// string and the operator position is in the AST vocabulary (the
// `isFilterAST` test `validate-preset-comparands.ts` uses).
if (
typeof node[0] === 'string' && typeof node[1] === 'string'
&& !['and', 'or'].includes(node[0].toLowerCase())
&& VALID_AST_OPERATORS.has(node[1].toLowerCase())
) {
visit({ field: node[0], path: `${path}[0]` });
return;
}
// Group ['and'|'or', ...members] or a bare list — recurse the members.
node.forEach((member, i) => {
if (typeof member === 'string') return; // the leading keyword
walkFilterFieldKeys(member, `${path}[${i}]`, visit, depth + 1);
});
return;
}

if (!isPlainObject(node)) return;

// View filter rule: { field, operator[, value] }.
if (typeof node.field === 'string' && typeof node.operator === 'string') {
visit({ field: node.field, path: `${path}.field` });
return;
}

conditionFieldKeys(node, path, '', visit, depth);
}
31 changes: 31 additions & 0 deletions packages/lint/src/index.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -467,6 +467,37 @@ export {
} from './validate-chart-bindings.js';
export type { ChartBindingFinding, ChartBindingSeverity } from './validate-chart-bindings.js';

// [#14105] The layer BELOW the chart/widget binding rules above: a dataset's
// own `include[]`, `dimensions[].field`, `measures[].field` and filter KEYS,
// resolved against the object graph. Its base-object half is
// `validateObjectReferences`' `datasets[].object` site.
export {
validateDatasetReferences,
DATASET_INCLUDE_UNKNOWN,
DATASET_FIELD_UNKNOWN,
DATASET_FIELD_NOT_INCLUDED,
DATASET_FILTER_FIELD_UNKNOWN,
} from './validate-dataset-references.js';
export type { DatasetRefFinding, DatasetRefSeverity } from './validate-dataset-references.js';

// The two reusable seams the rule above is written on, exported because the
// queued siblings (#14148 widget filter keys + sortBy, #14107 list-view field
// positions) must reuse ONE mechanism rather than growing a second hop-walker
// and a second filter-key reader. Both hold mechanism only — no rule ids, no
// severities, no findings; the judgement stays with the rule that asks.
export {
indexObjectGraph,
resolveFieldPath,
isUnjudgeable,
nearestName,
suggestName,
listNames,
RELATIONSHIP_FIELD_TYPES,
} from './object-graph.js';
export type { ObjectGraph, GraphObject, GraphField, FieldPathVerdict } from './object-graph.js';
export { walkFilterFieldKeys } from './filter-walk.js';
export type { FilterFieldKey } from './filter-walk.js';

// #4762 — the two STATIC artifacts an object validation rule carries (a
// `format` rule's `regex`, a `json_schema` rule's `schema`) are fail-OPEN at
// runtime: one that does not compile is logged and skipped, so the rule is
Expand Down
Loading
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 95 additions & 0 deletions .changeset/dataset-reference-resolution.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
---
'@objectstack/lint': minor
---

Resolve an ADR-0021 dataset's own references — base object, `include[]`,
`dimensions[].field` / `measures[].field`, and filter KEYS — at
`validate`/`build` (#14105)

A dataset could name a **base object that does not exist**, join a
**relationship that does not exist**, and bind every dimension and measure to
**fields that do not exist**, and `objectstack validate` exited **0** with
`✓ Validation passed`. `objectstack build` also exited 0 and wrote the dangling
dataset into `dist/objectstack.json`.

The sting was that the author-time rule pass **already walked those exact
nodes**. Measured on published 17.2.0, each mutation applied on its own and
confirmed on disk before running:

| mutation | before | after |
|:----------------------------------------------------|:-------|:------|
| dimension `field` → a base field that does not exist | passed | `dataset-field-unknown` |
| dimension `field` → a joined field that does not exist | passed | `dataset-field-unknown` |
| measure `field` → a field that does not exist | passed | `dataset-field-unknown` |
| measure filter KEY → a field that does not exist | passed | `dataset-filter-field-unknown` |
| `include[]` → a relationship that does not exist | passed | `dataset-include-unknown` |
| `object` → an object that does not exist | passed | `object-reference-unknown` |

The two controls in that measurement — a duplicate measure name
(`DatasetSchema.superRefine`) and a bad date macro in a **measure filter**
(`filter-token-unknown`) — both failed the build, so datasets were
demonstrably in the validation path the whole time. `filter-token-unknown`
already stood at `datasets[1].measures[1].filter.last_update_at.$lt` and
reasoned about the **value**; nothing standing in that same position resolved
the **key**, or the sibling `field` one level up.

This matters more for a dataset than for most metadata because a dataset is the
semantic layer: dashboards and reports bind its dimensions and measures by name
(ADR-0021), and the consumer end of that binding is already guarded
(`widget-dataset-unknown` / `widget-dimension-unknown` / `widget-measure-unknown`,
#7529/#8902). So the surviving hole was the quiet one — every binding resolves,
the board renders, and the charts are empty or subtly wrong because the dataset
underneath addresses columns that do not exist.

**Five verdicts, all `error`.** Four are new rule ids on a new suite member,
`validateDatasetReferences`:

- `dataset-include-unknown` — an `include[]` entry that resolves to nothing, or
to a field that is not a relationship, so no join can be derived from it.
- `dataset-field-unknown` — a dimension or measure `field` path that resolves to
no column, on the base object or on any joined object along the path.
- `dataset-field-not-included` — the second real check: a dotted path that
RESOLVES, but whose relationship prefix was never declared in `include`.
ADR-0021 D-C joins only declared paths, so the column is out of the query's
reach however real it is.
- `dataset-filter-field-unknown` — a filter key on `Dataset.filter` or
`measures[].filter`, in any of the three authored filter shapes.

The fifth, the base object itself, lands on `validateObjectReferences` as a new
`datasets[].object` reference site rather than as a sixth id here. That rule's
charter IS object-name references that are plain `z.string()`, and putting it
there buys the curated cross-package severity ladder: the platform's own
`system.datasets.ts` declares five datasets over `sys_*` objects, three of which
live in packages a stack compiling plugin-auth alone cannot see. All five
resolve through `PLATFORM_PROVIDED_OBJECT_NAMES`; a local "not in this stack ⇒
error" check would have reported every one of them. When the base object does
not resolve, `validateDatasetReferences` skips the dataset entirely, so one typo
yields one finding rather than one per dimension, measure and filter key.

**Skips, so a finding is never a guess** (ADR-0072 D1): an object this stack
does not define, an object with no readable field map (ADR-0015 `external` and
introspected schemas), a registry-injected system column, and any hop *through*
one — an injected `owner_id` is a lookup at the registry whose target is
invisible here, so `owner_id.name` is unanswerable rather than a miss. The
shipped `showcase_task_metrics` dimension `{ field: 'created_at' }` is that skip's
live case, and every shipped dataset in the repo is silent under the new rule.

**Two reusable seams ship with it**, newly exported, because the same two
questions are asked at a dashboard widget's filter keys and `sortBy` and at a
list view's field positions, and three independent copies of a hop-walker drift:

- `object-graph.ts` — `indexObjectGraph` / `resolveFieldPath` / `isUnjudgeable`,
answering "what does this `relationship[.relationship].field` path resolve to?"
as a discriminated **verdict** union rather than a boolean, so a caller can
tell "this hop is not a relationship" from "this leaf does not exist" and write
the right prescription. Plus `nearestName` / `suggestName` / `listNames`.
- `walkFilterFieldKeys` (`filter-walk.ts`) — the FIELD-KEY half of a filter
subtree, beside the subtree-finding half that module already owns. It handles
all three authored shapes (Mongo condition object, `{ field, operator, value }`
rules, `[field, op, value]` triples), because a reader that handles only one
shape is the exact bug #3574 was filed against, and it composes a nested
condition object into one relationship path so `{ account: { region: … } }`
reports `account.region` rather than a bare `region` resolved against the
wrong object.

Both hold mechanism only — no rule ids, no severities, no findings.
132 changes: 132 additions & 0 deletions packages/lint/src/filter-walk.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -35,8 +35,21 @@
* they resolve an additional vocabulary (`AppContextSelector` ids such as
* `{active_package}`) that is meaningless in a filter, and restricting the walk
* is what holds false positives at zero.
*
* ## The two halves of a filter, and where each one is answered
*
* `walkAuthoredFilters` finds the SUBTREES. Inside one, a rule wants either the
* VALUES (`validate-filter-tokens.ts` classifies placeholders in them;
* `validate-preset-comparands.ts` judges ordering comparands) or the FIELD KEYS
* — the names the query is filtered BY. {@link walkFilterFieldKeys} is the
* second half (#14105), and it is here rather than in its first caller because
* the shape dispatch is identical to the one `validate-preset-comparands.ts`
* already performs on values: the platform authors filters three ways, and a
* reader that handles only one shape is the exact bug #3574 was filed against.
*/

import { VALID_AST_OPERATORS } from '@objectstack/spec/data';

/** Any plain metadata record. */
type AnyRec = Record<string, unknown>;

Expand DownExpand Up@@ -162,3 +175,122 @@ export function walkAuthoredFilters(
});
}
}

// ── The FIELD-KEY half of a filter subtree (#14105) ──────────────────────────

/** One field position inside an authored filter. */
export interface FilterFieldKey {
/**
* The field the condition filters BY, exactly as authored — a bare name
* (`status`), or a dotted relationship path (`account.region`, whether
* spelled that way or reached by descending a nested condition object).
*/
field: string;
/** Config path of the position, e.g. `datasets[1].measures[1].filter.last_update_at`. */
path: string;
}

/** Recursion guard — an authored filter is a bounded document, not a graph. */
const MAX_KEY_DEPTH = 32;

function isPlainObject(v: unknown): v is AnyRec {
return !!v && typeof v === 'object' && !Array.isArray(v) && !(v instanceof Date);
}

/**
* Emit the field positions of one Mongo-style condition NODE.
*
* `prefix` carries the relationship path accumulated by descending nested
* condition objects, so `{ account: { region: { $eq: 'emea' } } }` reports the
* single position `account.region` rather than a bare `region` that would
* resolve against the wrong object. A node whose value carries `$` operators is
* a leaf condition and ends the descent; a `$`-prefixed key that is not a
* recognised combinator is skipped and NOT descended, matching
* `validate-preset-comparands.ts` — an unrecognised operator's operand shape is
* not ours to guess.
*/
function conditionFieldKeys(
node: AnyRec,
path: string,
prefix: string,
visit: (key: FilterFieldKey) => void,
depth: number,
): void {
if (depth > MAX_KEY_DEPTH) return;
for (const [key, value] of Object.entries(node)) {
const here = `${path}.${key}`;
if (key === '$and' || key === '$or') {
if (Array.isArray(value)) {
value.forEach((arm, i) => {
if (isPlainObject(arm)) conditionFieldKeys(arm, `${here}[${i}]`, prefix, visit, depth + 1);
});
}
continue;
}
if (key === '$not') {
if (isPlainObject(value)) conditionFieldKeys(value, here, prefix, visit, depth + 1);
continue;
}
if (key.startsWith('$')) continue; // unrecognised combinator
const field = prefix ? `${prefix}.${key}` : key;
if (isPlainObject(value) && !Object.keys(value).some((k) => k.startsWith('$'))) {
// Nested relation / deep equality — the field position is one level
// deeper, so descend rather than reporting the intermediate hop twice.
// An EMPTY nested object addresses nothing further; report the hop itself.
if (Object.keys(value).length > 0) {
conditionFieldKeys(value, here, field, visit, depth + 1);
continue;
}
}
visit({ field, path: here });
}
}

/**
* Emit every FIELD KEY inside one authored filter subtree, whatever shape it
* was authored in — the key half of what `validate-preset-comparands.ts` does
* for values, and the traversal `filter-token-unknown` already performs while
* reasoning only about the strings it finds.
*
* Holds no judgement: it does not know which object the filter is bound to and
* emits no findings. Resolution is {@link resolveFieldPath}'s job
* (`object-graph.ts`) and the verdict is the caller's.
*/
export function walkFilterFieldKeys(
node: unknown,
path: string,
visit: (key: FilterFieldKey) => void,
depth = 0,
): void {
if (depth > MAX_KEY_DEPTH) return;

if (Array.isArray(node)) {
// Triple: ['field', op, value] — the field position is a non-keyword
// string and the operator position is in the AST vocabulary (the
// `isFilterAST` test `validate-preset-comparands.ts` uses).
if (
typeof node[0] === 'string' && typeof node[1] === 'string'
&& !['and', 'or'].includes(node[0].toLowerCase())
&& VALID_AST_OPERATORS.has(node[1].toLowerCase())
) {
visit({ field: node[0], path: `${path}[0]` });
return;
}
// Group ['and'|'or', ...members] or a bare list — recurse the members.
node.forEach((member, i) => {
if (typeof member === 'string') return; // the leading keyword
walkFilterFieldKeys(member, `${path}[${i}]`, visit, depth + 1);
});
return;
}

if (!isPlainObject(node)) return;

// View filter rule: { field, operator[, value] }.
if (typeof node.field === 'string' && typeof node.operator === 'string') {
visit({ field: node.field, path: `${path}.field` });
return;
}

conditionFieldKeys(node, path, '', visit, depth);
}
31 changes: 31 additions & 0 deletions packages/lint/src/index.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -467,6 +467,37 @@ export {
} from './validate-chart-bindings.js';
export type { ChartBindingFinding, ChartBindingSeverity } from './validate-chart-bindings.js';

// [#14105] The layer BELOW the chart/widget binding rules above: a dataset's
// own `include[]`, `dimensions[].field`, `measures[].field` and filter KEYS,
// resolved against the object graph. Its base-object half is
// `validateObjectReferences`' `datasets[].object` site.
export {
validateDatasetReferences,
DATASET_INCLUDE_UNKNOWN,
DATASET_FIELD_UNKNOWN,
DATASET_FIELD_NOT_INCLUDED,
DATASET_FILTER_FIELD_UNKNOWN,
} from './validate-dataset-references.js';
export type { DatasetRefFinding, DatasetRefSeverity } from './validate-dataset-references.js';

// The two reusable seams the rule above is written on, exported because the
// queued siblings (#14148 widget filter keys + sortBy, #14107 list-view field
// positions) must reuse ONE mechanism rather than growing a second hop-walker
// and a second filter-key reader. Both hold mechanism only — no rule ids, no
// severities, no findings; the judgement stays with the rule that asks.
export {
indexObjectGraph,
resolveFieldPath,
isUnjudgeable,
nearestName,
suggestName,
listNames,
RELATIONSHIP_FIELD_TYPES,
} from './object-graph.js';
export type { ObjectGraph, GraphObject, GraphField, FieldPathVerdict } from './object-graph.js';
export { walkFilterFieldKeys } from './filter-walk.js';
export type { FilterFieldKey } from './filter-walk.js';

// #4762 — the two STATIC artifacts an object validation rule carries (a
// `format` rule's `regex`, a `json_schema` rule's `schema`) are fail-OPEN at
// runtime: one that does not compile is logged and skipped, so the rule is
Expand Down
Loading
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 95 additions & 0 deletions .changeset/dataset-reference-resolution.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
---
'@objectstack/lint': minor
---

Resolve an ADR-0021 dataset's own references — base object, `include[]`,
`dimensions[].field` / `measures[].field`, and filter KEYS — at
`validate`/`build` (#14105)

A dataset could name a **base object that does not exist**, join a
**relationship that does not exist**, and bind every dimension and measure to
**fields that do not exist**, and `objectstack validate` exited **0** with
`✓ Validation passed`. `objectstack build` also exited 0 and wrote the dangling
dataset into `dist/objectstack.json`.

The sting was that the author-time rule pass **already walked those exact
nodes**. Measured on published 17.2.0, each mutation applied on its own and
confirmed on disk before running:

| mutation | before | after |
|:----------------------------------------------------|:-------|:------|
| dimension `field` → a base field that does not exist | passed | `dataset-field-unknown` |
| dimension `field` → a joined field that does not exist | passed | `dataset-field-unknown` |
| measure `field` → a field that does not exist | passed | `dataset-field-unknown` |
| measure filter KEY → a field that does not exist | passed | `dataset-filter-field-unknown` |
| `include[]` → a relationship that does not exist | passed | `dataset-include-unknown` |
| `object` → an object that does not exist | passed | `object-reference-unknown` |

The two controls in that measurement — a duplicate measure name
(`DatasetSchema.superRefine`) and a bad date macro in a **measure filter**
(`filter-token-unknown`) — both failed the build, so datasets were
demonstrably in the validation path the whole time. `filter-token-unknown`
already stood at `datasets[1].measures[1].filter.last_update_at.$lt` and
reasoned about the **value**; nothing standing in that same position resolved
the **key**, or the sibling `field` one level up.

This matters more for a dataset than for most metadata because a dataset is the
semantic layer: dashboards and reports bind its dimensions and measures by name
(ADR-0021), and the consumer end of that binding is already guarded
(`widget-dataset-unknown` / `widget-dimension-unknown` / `widget-measure-unknown`,
#7529/#8902). So the surviving hole was the quiet one — every binding resolves,
the board renders, and the charts are empty or subtly wrong because the dataset
underneath addresses columns that do not exist.

**Five verdicts, all `error`.** Four are new rule ids on a new suite member,
`validateDatasetReferences`:

- `dataset-include-unknown` — an `include[]` entry that resolves to nothing, or
to a field that is not a relationship, so no join can be derived from it.
- `dataset-field-unknown` — a dimension or measure `field` path that resolves to
no column, on the base object or on any joined object along the path.
- `dataset-field-not-included` — the second real check: a dotted path that
RESOLVES, but whose relationship prefix was never declared in `include`.
ADR-0021 D-C joins only declared paths, so the column is out of the query's
reach however real it is.
- `dataset-filter-field-unknown` — a filter key on `Dataset.filter` or
`measures[].filter`, in any of the three authored filter shapes.

The fifth, the base object itself, lands on `validateObjectReferences` as a new
`datasets[].object` reference site rather than as a sixth id here. That rule's
charter IS object-name references that are plain `z.string()`, and putting it
there buys the curated cross-package severity ladder: the platform's own
`system.datasets.ts` declares five datasets over `sys_*` objects, three of which
live in packages a stack compiling plugin-auth alone cannot see. All five
resolve through `PLATFORM_PROVIDED_OBJECT_NAMES`; a local "not in this stack ⇒
error" check would have reported every one of them. When the base object does
not resolve, `validateDatasetReferences` skips the dataset entirely, so one typo
yields one finding rather than one per dimension, measure and filter key.

**Skips, so a finding is never a guess** (ADR-0072 D1): an object this stack
does not define, an object with no readable field map (ADR-0015 `external` and
introspected schemas), a registry-injected system column, and any hop *through*
one — an injected `owner_id` is a lookup at the registry whose target is
invisible here, so `owner_id.name` is unanswerable rather than a miss. The
shipped `showcase_task_metrics` dimension `{ field: 'created_at' }` is that skip's
live case, and every shipped dataset in the repo is silent under the new rule.

**Two reusable seams ship with it**, newly exported, because the same two
questions are asked at a dashboard widget's filter keys and `sortBy` and at a
list view's field positions, and three independent copies of a hop-walker drift:

- `object-graph.ts` — `indexObjectGraph` / `resolveFieldPath` / `isUnjudgeable`,
answering "what does this `relationship[.relationship].field` path resolve to?"
as a discriminated **verdict** union rather than a boolean, so a caller can
tell "this hop is not a relationship" from "this leaf does not exist" and write
the right prescription. Plus `nearestName` / `suggestName` / `listNames`.
- `walkFilterFieldKeys` (`filter-walk.ts`) — the FIELD-KEY half of a filter
subtree, beside the subtree-finding half that module already owns. It handles
all three authored shapes (Mongo condition object, `{ field, operator, value }`
rules, `[field, op, value]` triples), because a reader that handles only one
shape is the exact bug #3574 was filed against, and it composes a nested
condition object into one relationship path so `{ account: { region: … } }`
reports `account.region` rather than a bare `region` resolved against the
wrong object.

Both hold mechanism only — no rule ids, no severities, no findings.
132 changes: 132 additions & 0 deletions packages/lint/src/filter-walk.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -35,8 +35,21 @@
* they resolve an additional vocabulary (`AppContextSelector` ids such as
* `{active_package}`) that is meaningless in a filter, and restricting the walk
* is what holds false positives at zero.
*
* ## The two halves of a filter, and where each one is answered
*
* `walkAuthoredFilters` finds the SUBTREES. Inside one, a rule wants either the
* VALUES (`validate-filter-tokens.ts` classifies placeholders in them;
* `validate-preset-comparands.ts` judges ordering comparands) or the FIELD KEYS
* — the names the query is filtered BY. {@link walkFilterFieldKeys} is the
* second half (#14105), and it is here rather than in its first caller because
* the shape dispatch is identical to the one `validate-preset-comparands.ts`
* already performs on values: the platform authors filters three ways, and a
* reader that handles only one shape is the exact bug #3574 was filed against.
*/

import { VALID_AST_OPERATORS } from '@objectstack/spec/data';

/** Any plain metadata record. */
type AnyRec = Record<string, unknown>;

Expand DownExpand Up@@ -162,3 +175,122 @@ export function walkAuthoredFilters(
});
}
}

// ── The FIELD-KEY half of a filter subtree (#14105) ──────────────────────────

/** One field position inside an authored filter. */
export interface FilterFieldKey {
/**
* The field the condition filters BY, exactly as authored — a bare name
* (`status`), or a dotted relationship path (`account.region`, whether
* spelled that way or reached by descending a nested condition object).
*/
field: string;
/** Config path of the position, e.g. `datasets[1].measures[1].filter.last_update_at`. */
path: string;
}

/** Recursion guard — an authored filter is a bounded document, not a graph. */
const MAX_KEY_DEPTH = 32;

function isPlainObject(v: unknown): v is AnyRec {
return !!v && typeof v === 'object' && !Array.isArray(v) && !(v instanceof Date);
}

/**
* Emit the field positions of one Mongo-style condition NODE.
*
* `prefix` carries the relationship path accumulated by descending nested
* condition objects, so `{ account: { region: { $eq: 'emea' } } }` reports the
* single position `account.region` rather than a bare `region` that would
* resolve against the wrong object. A node whose value carries `$` operators is
* a leaf condition and ends the descent; a `$`-prefixed key that is not a
* recognised combinator is skipped and NOT descended, matching
* `validate-preset-comparands.ts` — an unrecognised operator's operand shape is
* not ours to guess.
*/
function conditionFieldKeys(
node: AnyRec,
path: string,
prefix: string,
visit: (key: FilterFieldKey) => void,
depth: number,
): void {
if (depth > MAX_KEY_DEPTH) return;
for (const [key, value] of Object.entries(node)) {
const here = `${path}.${key}`;
if (key === '$and' || key === '$or') {
if (Array.isArray(value)) {
value.forEach((arm, i) => {
if (isPlainObject(arm)) conditionFieldKeys(arm, `${here}[${i}]`, prefix, visit, depth + 1);
});
}
continue;
}
if (key === '$not') {
if (isPlainObject(value)) conditionFieldKeys(value, here, prefix, visit, depth + 1);
continue;
}
if (key.startsWith('$')) continue; // unrecognised combinator
const field = prefix ? `${prefix}.${key}` : key;
if (isPlainObject(value) && !Object.keys(value).some((k) => k.startsWith('$'))) {
// Nested relation / deep equality — the field position is one level
// deeper, so descend rather than reporting the intermediate hop twice.
// An EMPTY nested object addresses nothing further; report the hop itself.
if (Object.keys(value).length > 0) {
conditionFieldKeys(value, here, field, visit, depth + 1);
continue;
}
}
visit({ field, path: here });
}
}

/**
* Emit every FIELD KEY inside one authored filter subtree, whatever shape it
* was authored in — the key half of what `validate-preset-comparands.ts` does
* for values, and the traversal `filter-token-unknown` already performs while
* reasoning only about the strings it finds.
*
* Holds no judgement: it does not know which object the filter is bound to and
* emits no findings. Resolution is {@link resolveFieldPath}'s job
* (`object-graph.ts`) and the verdict is the caller's.
*/
export function walkFilterFieldKeys(
node: unknown,
path: string,
visit: (key: FilterFieldKey) => void,
depth = 0,
): void {
if (depth > MAX_KEY_DEPTH) return;

if (Array.isArray(node)) {
// Triple: ['field', op, value] — the field position is a non-keyword
// string and the operator position is in the AST vocabulary (the
// `isFilterAST` test `validate-preset-comparands.ts` uses).
if (
typeof node[0] === 'string' && typeof node[1] === 'string'
&& !['and', 'or'].includes(node[0].toLowerCase())
&& VALID_AST_OPERATORS.has(node[1].toLowerCase())
) {
visit({ field: node[0], path: `${path}[0]` });
return;
}
// Group ['and'|'or', ...members] or a bare list — recurse the members.
node.forEach((member, i) => {
if (typeof member === 'string') return; // the leading keyword
walkFilterFieldKeys(member, `${path}[${i}]`, visit, depth + 1);
});
return;
}

if (!isPlainObject(node)) return;

// View filter rule: { field, operator[, value] }.
if (typeof node.field === 'string' && typeof node.operator === 'string') {
visit({ field: node.field, path: `${path}.field` });
return;
}

conditionFieldKeys(node, path, '', visit, depth);
}
31 changes: 31 additions & 0 deletions packages/lint/src/index.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -467,6 +467,37 @@ export {
} from './validate-chart-bindings.js';
export type { ChartBindingFinding, ChartBindingSeverity } from './validate-chart-bindings.js';

// [#14105] The layer BELOW the chart/widget binding rules above: a dataset's
// own `include[]`, `dimensions[].field`, `measures[].field` and filter KEYS,
// resolved against the object graph. Its base-object half is
// `validateObjectReferences`' `datasets[].object` site.
export {
validateDatasetReferences,
DATASET_INCLUDE_UNKNOWN,
DATASET_FIELD_UNKNOWN,
DATASET_FIELD_NOT_INCLUDED,
DATASET_FILTER_FIELD_UNKNOWN,
} from './validate-dataset-references.js';
export type { DatasetRefFinding, DatasetRefSeverity } from './validate-dataset-references.js';

// The two reusable seams the rule above is written on, exported because the
// queued siblings (#14148 widget filter keys + sortBy, #14107 list-view field
// positions) must reuse ONE mechanism rather than growing a second hop-walker
// and a second filter-key reader. Both hold mechanism only — no rule ids, no
// severities, no findings; the judgement stays with the rule that asks.
export {
indexObjectGraph,
resolveFieldPath,
isUnjudgeable,
nearestName,
suggestName,
listNames,
RELATIONSHIP_FIELD_TYPES,
} from './object-graph.js';
export type { ObjectGraph, GraphObject, GraphField, FieldPathVerdict } from './object-graph.js';
export { walkFilterFieldKeys } from './filter-walk.js';
export type { FilterFieldKey } from './filter-walk.js';

// #4762 — the two STATIC artifacts an object validation rule carries (a
// `format` rule's `regex`, a `json_schema` rule's `schema`) are fail-OPEN at
// runtime: one that does not compile is logged and skipped, so the rule is
Expand Down
Loading
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 95 additions & 0 deletions .changeset/dataset-reference-resolution.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
---
'@objectstack/lint': minor
---

Resolve an ADR-0021 dataset's own references — base object, `include[]`,
`dimensions[].field` / `measures[].field`, and filter KEYS — at
`validate`/`build` (#14105)

A dataset could name a **base object that does not exist**, join a
**relationship that does not exist**, and bind every dimension and measure to
**fields that do not exist**, and `objectstack validate` exited **0** with
`✓ Validation passed`. `objectstack build` also exited 0 and wrote the dangling
dataset into `dist/objectstack.json`.

The sting was that the author-time rule pass **already walked those exact
nodes**. Measured on published 17.2.0, each mutation applied on its own and
confirmed on disk before running:

| mutation | before | after |
|:----------------------------------------------------|:-------|:------|
| dimension `field` → a base field that does not exist | passed | `dataset-field-unknown` |
| dimension `field` → a joined field that does not exist | passed | `dataset-field-unknown` |
| measure `field` → a field that does not exist | passed | `dataset-field-unknown` |
| measure filter KEY → a field that does not exist | passed | `dataset-filter-field-unknown` |
| `include[]` → a relationship that does not exist | passed | `dataset-include-unknown` |
| `object` → an object that does not exist | passed | `object-reference-unknown` |

The two controls in that measurement — a duplicate measure name
(`DatasetSchema.superRefine`) and a bad date macro in a **measure filter**
(`filter-token-unknown`) — both failed the build, so datasets were
demonstrably in the validation path the whole time. `filter-token-unknown`
already stood at `datasets[1].measures[1].filter.last_update_at.$lt` and
reasoned about the **value**; nothing standing in that same position resolved
the **key**, or the sibling `field` one level up.

This matters more for a dataset than for most metadata because a dataset is the
semantic layer: dashboards and reports bind its dimensions and measures by name
(ADR-0021), and the consumer end of that binding is already guarded
(`widget-dataset-unknown` / `widget-dimension-unknown` / `widget-measure-unknown`,
#7529/#8902). So the surviving hole was the quiet one — every binding resolves,
the board renders, and the charts are empty or subtly wrong because the dataset
underneath addresses columns that do not exist.

**Five verdicts, all `error`.** Four are new rule ids on a new suite member,
`validateDatasetReferences`:

- `dataset-include-unknown` — an `include[]` entry that resolves to nothing, or
to a field that is not a relationship, so no join can be derived from it.
- `dataset-field-unknown` — a dimension or measure `field` path that resolves to
no column, on the base object or on any joined object along the path.
- `dataset-field-not-included` — the second real check: a dotted path that
RESOLVES, but whose relationship prefix was never declared in `include`.
ADR-0021 D-C joins only declared paths, so the column is out of the query's
reach however real it is.
- `dataset-filter-field-unknown` — a filter key on `Dataset.filter` or
`measures[].filter`, in any of the three authored filter shapes.

The fifth, the base object itself, lands on `validateObjectReferences` as a new
`datasets[].object` reference site rather than as a sixth id here. That rule's
charter IS object-name references that are plain `z.string()`, and putting it
there buys the curated cross-package severity ladder: the platform's own
`system.datasets.ts` declares five datasets over `sys_*` objects, three of which
live in packages a stack compiling plugin-auth alone cannot see. All five
resolve through `PLATFORM_PROVIDED_OBJECT_NAMES`; a local "not in this stack ⇒
error" check would have reported every one of them. When the base object does
not resolve, `validateDatasetReferences` skips the dataset entirely, so one typo
yields one finding rather than one per dimension, measure and filter key.

**Skips, so a finding is never a guess** (ADR-0072 D1): an object this stack
does not define, an object with no readable field map (ADR-0015 `external` and
introspected schemas), a registry-injected system column, and any hop *through*
one — an injected `owner_id` is a lookup at the registry whose target is
invisible here, so `owner_id.name` is unanswerable rather than a miss. The
shipped `showcase_task_metrics` dimension `{ field: 'created_at' }` is that skip's
live case, and every shipped dataset in the repo is silent under the new rule.

**Two reusable seams ship with it**, newly exported, because the same two
questions are asked at a dashboard widget's filter keys and `sortBy` and at a
list view's field positions, and three independent copies of a hop-walker drift:

- `object-graph.ts` — `indexObjectGraph` / `resolveFieldPath` / `isUnjudgeable`,
answering "what does this `relationship[.relationship].field` path resolve to?"
as a discriminated **verdict** union rather than a boolean, so a caller can
tell "this hop is not a relationship" from "this leaf does not exist" and write
the right prescription. Plus `nearestName` / `suggestName` / `listNames`.
- `walkFilterFieldKeys` (`filter-walk.ts`) — the FIELD-KEY half of a filter
subtree, beside the subtree-finding half that module already owns. It handles
all three authored shapes (Mongo condition object, `{ field, operator, value }`
rules, `[field, op, value]` triples), because a reader that handles only one
shape is the exact bug #3574 was filed against, and it composes a nested
condition object into one relationship path so `{ account: { region: … } }`
reports `account.region` rather than a bare `region` resolved against the
wrong object.

Both hold mechanism only — no rule ids, no severities, no findings.
132 changes: 132 additions & 0 deletions packages/lint/src/filter-walk.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -35,8 +35,21 @@
* they resolve an additional vocabulary (`AppContextSelector` ids such as
* `{active_package}`) that is meaningless in a filter, and restricting the walk
* is what holds false positives at zero.
*
* ## The two halves of a filter, and where each one is answered
*
* `walkAuthoredFilters` finds the SUBTREES. Inside one, a rule wants either the
* VALUES (`validate-filter-tokens.ts` classifies placeholders in them;
* `validate-preset-comparands.ts` judges ordering comparands) or the FIELD KEYS
* — the names the query is filtered BY. {@link walkFilterFieldKeys} is the
* second half (#14105), and it is here rather than in its first caller because
* the shape dispatch is identical to the one `validate-preset-comparands.ts`
* already performs on values: the platform authors filters three ways, and a
* reader that handles only one shape is the exact bug #3574 was filed against.
*/

import { VALID_AST_OPERATORS } from '@objectstack/spec/data';

/** Any plain metadata record. */
type AnyRec = Record<string, unknown>;

Expand DownExpand Up@@ -162,3 +175,122 @@ export function walkAuthoredFilters(
});
}
}

// ── The FIELD-KEY half of a filter subtree (#14105) ──────────────────────────

/** One field position inside an authored filter. */
export interface FilterFieldKey {
/**
* The field the condition filters BY, exactly as authored — a bare name
* (`status`), or a dotted relationship path (`account.region`, whether
* spelled that way or reached by descending a nested condition object).
*/
field: string;
/** Config path of the position, e.g. `datasets[1].measures[1].filter.last_update_at`. */
path: string;
}

/** Recursion guard — an authored filter is a bounded document, not a graph. */
const MAX_KEY_DEPTH = 32;

function isPlainObject(v: unknown): v is AnyRec {
return !!v && typeof v === 'object' && !Array.isArray(v) && !(v instanceof Date);
}

/**
* Emit the field positions of one Mongo-style condition NODE.
*
* `prefix` carries the relationship path accumulated by descending nested
* condition objects, so `{ account: { region: { $eq: 'emea' } } }` reports the
* single position `account.region` rather than a bare `region` that would
* resolve against the wrong object. A node whose value carries `$` operators is
* a leaf condition and ends the descent; a `$`-prefixed key that is not a
* recognised combinator is skipped and NOT descended, matching
* `validate-preset-comparands.ts` — an unrecognised operator's operand shape is
* not ours to guess.
*/
function conditionFieldKeys(
node: AnyRec,
path: string,
prefix: string,
visit: (key: FilterFieldKey) => void,
depth: number,
): void {
if (depth > MAX_KEY_DEPTH) return;
for (const [key, value] of Object.entries(node)) {
const here = `${path}.${key}`;
if (key === '$and' || key === '$or') {
if (Array.isArray(value)) {
value.forEach((arm, i) => {
if (isPlainObject(arm)) conditionFieldKeys(arm, `${here}[${i}]`, prefix, visit, depth + 1);
});
}
continue;
}
if (key === '$not') {
if (isPlainObject(value)) conditionFieldKeys(value, here, prefix, visit, depth + 1);
continue;
}
if (key.startsWith('$')) continue; // unrecognised combinator
const field = prefix ? `${prefix}.${key}` : key;
if (isPlainObject(value) && !Object.keys(value).some((k) => k.startsWith('$'))) {
// Nested relation / deep equality — the field position is one level
// deeper, so descend rather than reporting the intermediate hop twice.
// An EMPTY nested object addresses nothing further; report the hop itself.
if (Object.keys(value).length > 0) {
conditionFieldKeys(value, here, field, visit, depth + 1);
continue;
}
}
visit({ field, path: here });
}
}

/**
* Emit every FIELD KEY inside one authored filter subtree, whatever shape it
* was authored in — the key half of what `validate-preset-comparands.ts` does
* for values, and the traversal `filter-token-unknown` already performs while
* reasoning only about the strings it finds.
*
* Holds no judgement: it does not know which object the filter is bound to and
* emits no findings. Resolution is {@link resolveFieldPath}'s job
* (`object-graph.ts`) and the verdict is the caller's.
*/
export function walkFilterFieldKeys(
node: unknown,
path: string,
visit: (key: FilterFieldKey) => void,
depth = 0,
): void {
if (depth > MAX_KEY_DEPTH) return;

if (Array.isArray(node)) {
// Triple: ['field', op, value] — the field position is a non-keyword
// string and the operator position is in the AST vocabulary (the
// `isFilterAST` test `validate-preset-comparands.ts` uses).
if (
typeof node[0] === 'string' && typeof node[1] === 'string'
&& !['and', 'or'].includes(node[0].toLowerCase())
&& VALID_AST_OPERATORS.has(node[1].toLowerCase())
) {
visit({ field: node[0], path: `${path}[0]` });
return;
}
// Group ['and'|'or', ...members] or a bare list — recurse the members.
node.forEach((member, i) => {
if (typeof member === 'string') return; // the leading keyword
walkFilterFieldKeys(member, `${path}[${i}]`, visit, depth + 1);
});
return;
}

if (!isPlainObject(node)) return;

// View filter rule: { field, operator[, value] }.
if (typeof node.field === 'string' && typeof node.operator === 'string') {
visit({ field: node.field, path: `${path}.field` });
return;
}

conditionFieldKeys(node, path, '', visit, depth);
}
31 changes: 31 additions & 0 deletions packages/lint/src/index.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -467,6 +467,37 @@ export {
} from './validate-chart-bindings.js';
export type { ChartBindingFinding, ChartBindingSeverity } from './validate-chart-bindings.js';

// [#14105] The layer BELOW the chart/widget binding rules above: a dataset's
// own `include[]`, `dimensions[].field`, `measures[].field` and filter KEYS,
// resolved against the object graph. Its base-object half is
// `validateObjectReferences`' `datasets[].object` site.
export {
validateDatasetReferences,
DATASET_INCLUDE_UNKNOWN,
DATASET_FIELD_UNKNOWN,
DATASET_FIELD_NOT_INCLUDED,
DATASET_FILTER_FIELD_UNKNOWN,
} from './validate-dataset-references.js';
export type { DatasetRefFinding, DatasetRefSeverity } from './validate-dataset-references.js';

// The two reusable seams the rule above is written on, exported because the
// queued siblings (#14148 widget filter keys + sortBy, #14107 list-view field
// positions) must reuse ONE mechanism rather than growing a second hop-walker
// and a second filter-key reader. Both hold mechanism only — no rule ids, no
// severities, no findings; the judgement stays with the rule that asks.
export {
indexObjectGraph,
resolveFieldPath,
isUnjudgeable,
nearestName,
suggestName,
listNames,
RELATIONSHIP_FIELD_TYPES,
} from './object-graph.js';
export type { ObjectGraph, GraphObject, GraphField, FieldPathVerdict } from './object-graph.js';
export { walkFilterFieldKeys } from './filter-walk.js';
export type { FilterFieldKey } from './filter-walk.js';

// #4762 — the two STATIC artifacts an object validation rule carries (a
// `format` rule's `regex`, a `json_schema` rule's `schema`) are fail-OPEN at
// runtime: one that does not compile is logged and skipped, so the rule is
Expand Down
Loading
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 95 additions & 0 deletions .changeset/dataset-reference-resolution.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
---
'@objectstack/lint': minor
---

Resolve an ADR-0021 dataset's own references — base object, `include[]`,
`dimensions[].field` / `measures[].field`, and filter KEYS — at
`validate`/`build` (#14105)

A dataset could name a **base object that does not exist**, join a
**relationship that does not exist**, and bind every dimension and measure to
**fields that do not exist**, and `objectstack validate` exited **0** with
`✓ Validation passed`. `objectstack build` also exited 0 and wrote the dangling
dataset into `dist/objectstack.json`.

The sting was that the author-time rule pass **already walked those exact
nodes**. Measured on published 17.2.0, each mutation applied on its own and
confirmed on disk before running:

| mutation | before | after |
|:----------------------------------------------------|:-------|:------|
| dimension `field` → a base field that does not exist | passed | `dataset-field-unknown` |
| dimension `field` → a joined field that does not exist | passed | `dataset-field-unknown` |
| measure `field` → a field that does not exist | passed | `dataset-field-unknown` |
| measure filter KEY → a field that does not exist | passed | `dataset-filter-field-unknown` |
| `include[]` → a relationship that does not exist | passed | `dataset-include-unknown` |
| `object` → an object that does not exist | passed | `object-reference-unknown` |

The two controls in that measurement — a duplicate measure name
(`DatasetSchema.superRefine`) and a bad date macro in a **measure filter**
(`filter-token-unknown`) — both failed the build, so datasets were
demonstrably in the validation path the whole time. `filter-token-unknown`
already stood at `datasets[1].measures[1].filter.last_update_at.$lt` and
reasoned about the **value**; nothing standing in that same position resolved
the **key**, or the sibling `field` one level up.

This matters more for a dataset than for most metadata because a dataset is the
semantic layer: dashboards and reports bind its dimensions and measures by name
(ADR-0021), and the consumer end of that binding is already guarded
(`widget-dataset-unknown` / `widget-dimension-unknown` / `widget-measure-unknown`,
#7529/#8902). So the surviving hole was the quiet one — every binding resolves,
the board renders, and the charts are empty or subtly wrong because the dataset
underneath addresses columns that do not exist.

**Five verdicts, all `error`.** Four are new rule ids on a new suite member,
`validateDatasetReferences`:

- `dataset-include-unknown` — an `include[]` entry that resolves to nothing, or
to a field that is not a relationship, so no join can be derived from it.
- `dataset-field-unknown` — a dimension or measure `field` path that resolves to
no column, on the base object or on any joined object along the path.
- `dataset-field-not-included` — the second real check: a dotted path that
RESOLVES, but whose relationship prefix was never declared in `include`.
ADR-0021 D-C joins only declared paths, so the column is out of the query's
reach however real it is.
- `dataset-filter-field-unknown` — a filter key on `Dataset.filter` or
`measures[].filter`, in any of the three authored filter shapes.

The fifth, the base object itself, lands on `validateObjectReferences` as a new
`datasets[].object` reference site rather than as a sixth id here. That rule's
charter IS object-name references that are plain `z.string()`, and putting it
there buys the curated cross-package severity ladder: the platform's own
`system.datasets.ts` declares five datasets over `sys_*` objects, three of which
live in packages a stack compiling plugin-auth alone cannot see. All five
resolve through `PLATFORM_PROVIDED_OBJECT_NAMES`; a local "not in this stack ⇒
error" check would have reported every one of them. When the base object does
not resolve, `validateDatasetReferences` skips the dataset entirely, so one typo
yields one finding rather than one per dimension, measure and filter key.

**Skips, so a finding is never a guess** (ADR-0072 D1): an object this stack
does not define, an object with no readable field map (ADR-0015 `external` and
introspected schemas), a registry-injected system column, and any hop *through*
one — an injected `owner_id` is a lookup at the registry whose target is
invisible here, so `owner_id.name` is unanswerable rather than a miss. The
shipped `showcase_task_metrics` dimension `{ field: 'created_at' }` is that skip's
live case, and every shipped dataset in the repo is silent under the new rule.

**Two reusable seams ship with it**, newly exported, because the same two
questions are asked at a dashboard widget's filter keys and `sortBy` and at a
list view's field positions, and three independent copies of a hop-walker drift:

- `object-graph.ts` — `indexObjectGraph` / `resolveFieldPath` / `isUnjudgeable`,
answering "what does this `relationship[.relationship].field` path resolve to?"
as a discriminated **verdict** union rather than a boolean, so a caller can
tell "this hop is not a relationship" from "this leaf does not exist" and write
the right prescription. Plus `nearestName` / `suggestName` / `listNames`.
- `walkFilterFieldKeys` (`filter-walk.ts`) — the FIELD-KEY half of a filter
subtree, beside the subtree-finding half that module already owns. It handles
all three authored shapes (Mongo condition object, `{ field, operator, value }`
rules, `[field, op, value]` triples), because a reader that handles only one
shape is the exact bug #3574 was filed against, and it composes a nested
condition object into one relationship path so `{ account: { region: … } }`
reports `account.region` rather than a bare `region` resolved against the
wrong object.

Both hold mechanism only — no rule ids, no severities, no findings.
132 changes: 132 additions & 0 deletions packages/lint/src/filter-walk.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -35,8 +35,21 @@
* they resolve an additional vocabulary (`AppContextSelector` ids such as
* `{active_package}`) that is meaningless in a filter, and restricting the walk
* is what holds false positives at zero.
*
* ## The two halves of a filter, and where each one is answered
*
* `walkAuthoredFilters` finds the SUBTREES. Inside one, a rule wants either the
* VALUES (`validate-filter-tokens.ts` classifies placeholders in them;
* `validate-preset-comparands.ts` judges ordering comparands) or the FIELD KEYS
* — the names the query is filtered BY. {@link walkFilterFieldKeys} is the
* second half (#14105), and it is here rather than in its first caller because
* the shape dispatch is identical to the one `validate-preset-comparands.ts`
* already performs on values: the platform authors filters three ways, and a
* reader that handles only one shape is the exact bug #3574 was filed against.
*/

import { VALID_AST_OPERATORS } from '@objectstack/spec/data';

/** Any plain metadata record. */
type AnyRec = Record<string, unknown>;

Expand DownExpand Up@@ -162,3 +175,122 @@ export function walkAuthoredFilters(
});
}
}

// ── The FIELD-KEY half of a filter subtree (#14105) ──────────────────────────

/** One field position inside an authored filter. */
export interface FilterFieldKey {
/**
* The field the condition filters BY, exactly as authored — a bare name
* (`status`), or a dotted relationship path (`account.region`, whether
* spelled that way or reached by descending a nested condition object).
*/
field: string;
/** Config path of the position, e.g. `datasets[1].measures[1].filter.last_update_at`. */
path: string;
}

/** Recursion guard — an authored filter is a bounded document, not a graph. */
const MAX_KEY_DEPTH = 32;

function isPlainObject(v: unknown): v is AnyRec {
return !!v && typeof v === 'object' && !Array.isArray(v) && !(v instanceof Date);
}

/**
* Emit the field positions of one Mongo-style condition NODE.
*
* `prefix` carries the relationship path accumulated by descending nested
* condition objects, so `{ account: { region: { $eq: 'emea' } } }` reports the
* single position `account.region` rather than a bare `region` that would
* resolve against the wrong object. A node whose value carries `$` operators is
* a leaf condition and ends the descent; a `$`-prefixed key that is not a
* recognised combinator is skipped and NOT descended, matching
* `validate-preset-comparands.ts` — an unrecognised operator's operand shape is
* not ours to guess.
*/
function conditionFieldKeys(
node: AnyRec,
path: string,
prefix: string,
visit: (key: FilterFieldKey) => void,
depth: number,
): void {
if (depth > MAX_KEY_DEPTH) return;
for (const [key, value] of Object.entries(node)) {
const here = `${path}.${key}`;
if (key === '$and' || key === '$or') {
if (Array.isArray(value)) {
value.forEach((arm, i) => {
if (isPlainObject(arm)) conditionFieldKeys(arm, `${here}[${i}]`, prefix, visit, depth + 1);
});
}
continue;
}
if (key === '$not') {
if (isPlainObject(value)) conditionFieldKeys(value, here, prefix, visit, depth + 1);
continue;
}
if (key.startsWith('$')) continue; // unrecognised combinator
const field = prefix ? `${prefix}.${key}` : key;
if (isPlainObject(value) && !Object.keys(value).some((k) => k.startsWith('$'))) {
// Nested relation / deep equality — the field position is one level
// deeper, so descend rather than reporting the intermediate hop twice.
// An EMPTY nested object addresses nothing further; report the hop itself.
if (Object.keys(value).length > 0) {
conditionFieldKeys(value, here, field, visit, depth + 1);
continue;
}
}
visit({ field, path: here });
}
}

/**
* Emit every FIELD KEY inside one authored filter subtree, whatever shape it
* was authored in — the key half of what `validate-preset-comparands.ts` does
* for values, and the traversal `filter-token-unknown` already performs while
* reasoning only about the strings it finds.
*
* Holds no judgement: it does not know which object the filter is bound to and
* emits no findings. Resolution is {@link resolveFieldPath}'s job
* (`object-graph.ts`) and the verdict is the caller's.
*/
export function walkFilterFieldKeys(
node: unknown,
path: string,
visit: (key: FilterFieldKey) => void,
depth = 0,
): void {
if (depth > MAX_KEY_DEPTH) return;

if (Array.isArray(node)) {
// Triple: ['field', op, value] — the field position is a non-keyword
// string and the operator position is in the AST vocabulary (the
// `isFilterAST` test `validate-preset-comparands.ts` uses).
if (
typeof node[0] === 'string' && typeof node[1] === 'string'
&& !['and', 'or'].includes(node[0].toLowerCase())
&& VALID_AST_OPERATORS.has(node[1].toLowerCase())
) {
visit({ field: node[0], path: `${path}[0]` });
return;
}
// Group ['and'|'or', ...members] or a bare list — recurse the members.
node.forEach((member, i) => {
if (typeof member === 'string') return; // the leading keyword
walkFilterFieldKeys(member, `${path}[${i}]`, visit, depth + 1);
});
return;
}

if (!isPlainObject(node)) return;

// View filter rule: { field, operator[, value] }.
if (typeof node.field === 'string' && typeof node.operator === 'string') {
visit({ field: node.field, path: `${path}.field` });
return;
}

conditionFieldKeys(node, path, '', visit, depth);
}
31 changes: 31 additions & 0 deletions packages/lint/src/index.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -467,6 +467,37 @@ export {
} from './validate-chart-bindings.js';
export type { ChartBindingFinding, ChartBindingSeverity } from './validate-chart-bindings.js';

// [#14105] The layer BELOW the chart/widget binding rules above: a dataset's
// own `include[]`, `dimensions[].field`, `measures[].field` and filter KEYS,
// resolved against the object graph. Its base-object half is
// `validateObjectReferences`' `datasets[].object` site.
export {
validateDatasetReferences,
DATASET_INCLUDE_UNKNOWN,
DATASET_FIELD_UNKNOWN,
DATASET_FIELD_NOT_INCLUDED,
DATASET_FILTER_FIELD_UNKNOWN,
} from './validate-dataset-references.js';
export type { DatasetRefFinding, DatasetRefSeverity } from './validate-dataset-references.js';

// The two reusable seams the rule above is written on, exported because the
// queued siblings (#14148 widget filter keys + sortBy, #14107 list-view field
// positions) must reuse ONE mechanism rather than growing a second hop-walker
// and a second filter-key reader. Both hold mechanism only — no rule ids, no
// severities, no findings; the judgement stays with the rule that asks.
export {
indexObjectGraph,
resolveFieldPath,
isUnjudgeable,
nearestName,
suggestName,
listNames,
RELATIONSHIP_FIELD_TYPES,
} from './object-graph.js';
export type { ObjectGraph, GraphObject, GraphField, FieldPathVerdict } from './object-graph.js';
export { walkFilterFieldKeys } from './filter-walk.js';
export type { FilterFieldKey } from './filter-walk.js';

// #4762 — the two STATIC artifacts an object validation rule carries (a
// `format` rule's `regex`, a `json_schema` rule's `schema`) are fail-OPEN at
// runtime: one that does not compile is logged and skipped, so the rule is
Expand Down
Loading
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 95 additions & 0 deletions .changeset/dataset-reference-resolution.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
---
'@objectstack/lint': minor
---

Resolve an ADR-0021 dataset's own references — base object, `include[]`,
`dimensions[].field` / `measures[].field`, and filter KEYS — at
`validate`/`build` (#14105)

A dataset could name a **base object that does not exist**, join a
**relationship that does not exist**, and bind every dimension and measure to
**fields that do not exist**, and `objectstack validate` exited **0** with
`✓ Validation passed`. `objectstack build` also exited 0 and wrote the dangling
dataset into `dist/objectstack.json`.

The sting was that the author-time rule pass **already walked those exact
nodes**. Measured on published 17.2.0, each mutation applied on its own and
confirmed on disk before running:

| mutation | before | after |
|:----------------------------------------------------|:-------|:------|
| dimension `field` → a base field that does not exist | passed | `dataset-field-unknown` |
| dimension `field` → a joined field that does not exist | passed | `dataset-field-unknown` |
| measure `field` → a field that does not exist | passed | `dataset-field-unknown` |
| measure filter KEY → a field that does not exist | passed | `dataset-filter-field-unknown` |
| `include[]` → a relationship that does not exist | passed | `dataset-include-unknown` |
| `object` → an object that does not exist | passed | `object-reference-unknown` |

The two controls in that measurement — a duplicate measure name
(`DatasetSchema.superRefine`) and a bad date macro in a **measure filter**
(`filter-token-unknown`) — both failed the build, so datasets were
demonstrably in the validation path the whole time. `filter-token-unknown`
already stood at `datasets[1].measures[1].filter.last_update_at.$lt` and
reasoned about the **value**; nothing standing in that same position resolved
the **key**, or the sibling `field` one level up.

This matters more for a dataset than for most metadata because a dataset is the
semantic layer: dashboards and reports bind its dimensions and measures by name
(ADR-0021), and the consumer end of that binding is already guarded
(`widget-dataset-unknown` / `widget-dimension-unknown` / `widget-measure-unknown`,
#7529/#8902). So the surviving hole was the quiet one — every binding resolves,
the board renders, and the charts are empty or subtly wrong because the dataset
underneath addresses columns that do not exist.

**Five verdicts, all `error`.** Four are new rule ids on a new suite member,
`validateDatasetReferences`:

- `dataset-include-unknown` — an `include[]` entry that resolves to nothing, or
to a field that is not a relationship, so no join can be derived from it.
- `dataset-field-unknown` — a dimension or measure `field` path that resolves to
no column, on the base object or on any joined object along the path.
- `dataset-field-not-included` — the second real check: a dotted path that
RESOLVES, but whose relationship prefix was never declared in `include`.
ADR-0021 D-C joins only declared paths, so the column is out of the query's
reach however real it is.
- `dataset-filter-field-unknown` — a filter key on `Dataset.filter` or
`measures[].filter`, in any of the three authored filter shapes.

The fifth, the base object itself, lands on `validateObjectReferences` as a new
`datasets[].object` reference site rather than as a sixth id here. That rule's
charter IS object-name references that are plain `z.string()`, and putting it
there buys the curated cross-package severity ladder: the platform's own
`system.datasets.ts` declares five datasets over `sys_*` objects, three of which
live in packages a stack compiling plugin-auth alone cannot see. All five
resolve through `PLATFORM_PROVIDED_OBJECT_NAMES`; a local "not in this stack ⇒
error" check would have reported every one of them. When the base object does
not resolve, `validateDatasetReferences` skips the dataset entirely, so one typo
yields one finding rather than one per dimension, measure and filter key.

**Skips, so a finding is never a guess** (ADR-0072 D1): an object this stack
does not define, an object with no readable field map (ADR-0015 `external` and
introspected schemas), a registry-injected system column, and any hop *through*
one — an injected `owner_id` is a lookup at the registry whose target is
invisible here, so `owner_id.name` is unanswerable rather than a miss. The
shipped `showcase_task_metrics` dimension `{ field: 'created_at' }` is that skip's
live case, and every shipped dataset in the repo is silent under the new rule.

**Two reusable seams ship with it**, newly exported, because the same two
questions are asked at a dashboard widget's filter keys and `sortBy` and at a
list view's field positions, and three independent copies of a hop-walker drift:

- `object-graph.ts` — `indexObjectGraph` / `resolveFieldPath` / `isUnjudgeable`,
answering "what does this `relationship[.relationship].field` path resolve to?"
as a discriminated **verdict** union rather than a boolean, so a caller can
tell "this hop is not a relationship" from "this leaf does not exist" and write
the right prescription. Plus `nearestName` / `suggestName` / `listNames`.
- `walkFilterFieldKeys` (`filter-walk.ts`) — the FIELD-KEY half of a filter
subtree, beside the subtree-finding half that module already owns. It handles
all three authored shapes (Mongo condition object, `{ field, operator, value }`
rules, `[field, op, value]` triples), because a reader that handles only one
shape is the exact bug #3574 was filed against, and it composes a nested
condition object into one relationship path so `{ account: { region: … } }`
reports `account.region` rather than a bare `region` resolved against the
wrong object.

Both hold mechanism only — no rule ids, no severities, no findings.
132 changes: 132 additions & 0 deletions packages/lint/src/filter-walk.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -35,8 +35,21 @@
* they resolve an additional vocabulary (`AppContextSelector` ids such as
* `{active_package}`) that is meaningless in a filter, and restricting the walk
* is what holds false positives at zero.
*
* ## The two halves of a filter, and where each one is answered
*
* `walkAuthoredFilters` finds the SUBTREES. Inside one, a rule wants either the
* VALUES (`validate-filter-tokens.ts` classifies placeholders in them;
* `validate-preset-comparands.ts` judges ordering comparands) or the FIELD KEYS
* — the names the query is filtered BY. {@link walkFilterFieldKeys} is the
* second half (#14105), and it is here rather than in its first caller because
* the shape dispatch is identical to the one `validate-preset-comparands.ts`
* already performs on values: the platform authors filters three ways, and a
* reader that handles only one shape is the exact bug #3574 was filed against.
*/

import { VALID_AST_OPERATORS } from '@objectstack/spec/data';

/** Any plain metadata record. */
type AnyRec = Record<string, unknown>;

Expand DownExpand Up@@ -162,3 +175,122 @@ export function walkAuthoredFilters(
});
}
}

// ── The FIELD-KEY half of a filter subtree (#14105) ──────────────────────────

/** One field position inside an authored filter. */
export interface FilterFieldKey {
/**
* The field the condition filters BY, exactly as authored — a bare name
* (`status`), or a dotted relationship path (`account.region`, whether
* spelled that way or reached by descending a nested condition object).
*/
field: string;
/** Config path of the position, e.g. `datasets[1].measures[1].filter.last_update_at`. */
path: string;
}

/** Recursion guard — an authored filter is a bounded document, not a graph. */
const MAX_KEY_DEPTH = 32;

function isPlainObject(v: unknown): v is AnyRec {
return !!v && typeof v === 'object' && !Array.isArray(v) && !(v instanceof Date);
}

/**
* Emit the field positions of one Mongo-style condition NODE.
*
* `prefix` carries the relationship path accumulated by descending nested
* condition objects, so `{ account: { region: { $eq: 'emea' } } }` reports the
* single position `account.region` rather than a bare `region` that would
* resolve against the wrong object. A node whose value carries `$` operators is
* a leaf condition and ends the descent; a `$`-prefixed key that is not a
* recognised combinator is skipped and NOT descended, matching
* `validate-preset-comparands.ts` — an unrecognised operator's operand shape is
* not ours to guess.
*/
function conditionFieldKeys(
node: AnyRec,
path: string,
prefix: string,
visit: (key: FilterFieldKey) => void,
depth: number,
): void {
if (depth > MAX_KEY_DEPTH) return;
for (const [key, value] of Object.entries(node)) {
const here = `${path}.${key}`;
if (key === '$and' || key === '$or') {
if (Array.isArray(value)) {
value.forEach((arm, i) => {
if (isPlainObject(arm)) conditionFieldKeys(arm, `${here}[${i}]`, prefix, visit, depth + 1);
});
}
continue;
}
if (key === '$not') {
if (isPlainObject(value)) conditionFieldKeys(value, here, prefix, visit, depth + 1);
continue;
}
if (key.startsWith('$')) continue; // unrecognised combinator
const field = prefix ? `${prefix}.${key}` : key;
if (isPlainObject(value) && !Object.keys(value).some((k) => k.startsWith('$'))) {
// Nested relation / deep equality — the field position is one level
// deeper, so descend rather than reporting the intermediate hop twice.
// An EMPTY nested object addresses nothing further; report the hop itself.
if (Object.keys(value).length > 0) {
conditionFieldKeys(value, here, field, visit, depth + 1);
continue;
}
}
visit({ field, path: here });
}
}

/**
* Emit every FIELD KEY inside one authored filter subtree, whatever shape it
* was authored in — the key half of what `validate-preset-comparands.ts` does
* for values, and the traversal `filter-token-unknown` already performs while
* reasoning only about the strings it finds.
*
* Holds no judgement: it does not know which object the filter is bound to and
* emits no findings. Resolution is {@link resolveFieldPath}'s job
* (`object-graph.ts`) and the verdict is the caller's.
*/
export function walkFilterFieldKeys(
node: unknown,
path: string,
visit: (key: FilterFieldKey) => void,
depth = 0,
): void {
if (depth > MAX_KEY_DEPTH) return;

if (Array.isArray(node)) {
// Triple: ['field', op, value] — the field position is a non-keyword
// string and the operator position is in the AST vocabulary (the
// `isFilterAST` test `validate-preset-comparands.ts` uses).
if (
typeof node[0] === 'string' && typeof node[1] === 'string'
&& !['and', 'or'].includes(node[0].toLowerCase())
&& VALID_AST_OPERATORS.has(node[1].toLowerCase())
) {
visit({ field: node[0], path: `${path}[0]` });
return;
}
// Group ['and'|'or', ...members] or a bare list — recurse the members.
node.forEach((member, i) => {
if (typeof member === 'string') return; // the leading keyword
walkFilterFieldKeys(member, `${path}[${i}]`, visit, depth + 1);
});
return;
}

if (!isPlainObject(node)) return;

// View filter rule: { field, operator[, value] }.
if (typeof node.field === 'string' && typeof node.operator === 'string') {
visit({ field: node.field, path: `${path}.field` });
return;
}

conditionFieldKeys(node, path, '', visit, depth);
}
31 changes: 31 additions & 0 deletions packages/lint/src/index.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -467,6 +467,37 @@ export {
} from './validate-chart-bindings.js';
export type { ChartBindingFinding, ChartBindingSeverity } from './validate-chart-bindings.js';

// [#14105] The layer BELOW the chart/widget binding rules above: a dataset's
// own `include[]`, `dimensions[].field`, `measures[].field` and filter KEYS,
// resolved against the object graph. Its base-object half is
// `validateObjectReferences`' `datasets[].object` site.
export {
validateDatasetReferences,
DATASET_INCLUDE_UNKNOWN,
DATASET_FIELD_UNKNOWN,
DATASET_FIELD_NOT_INCLUDED,
DATASET_FILTER_FIELD_UNKNOWN,
} from './validate-dataset-references.js';
export type { DatasetRefFinding, DatasetRefSeverity } from './validate-dataset-references.js';

// The two reusable seams the rule above is written on, exported because the
// queued siblings (#14148 widget filter keys + sortBy, #14107 list-view field
// positions) must reuse ONE mechanism rather than growing a second hop-walker
// and a second filter-key reader. Both hold mechanism only — no rule ids, no
// severities, no findings; the judgement stays with the rule that asks.
export {
indexObjectGraph,
resolveFieldPath,
isUnjudgeable,
nearestName,
suggestName,
listNames,
RELATIONSHIP_FIELD_TYPES,
} from './object-graph.js';
export type { ObjectGraph, GraphObject, GraphField, FieldPathVerdict } from './object-graph.js';
export { walkFilterFieldKeys } from './filter-walk.js';
export type { FilterFieldKey } from './filter-walk.js';

// #4762 — the two STATIC artifacts an object validation rule carries (a
// `format` rule's `regex`, a `json_schema` rule's `schema`) are fail-OPEN at
// runtime: one that does not compile is logged and skipped, so the rule is
Expand Down
Loading
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 95 additions & 0 deletions .changeset/dataset-reference-resolution.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
---
'@objectstack/lint': minor
---

Resolve an ADR-0021 dataset's own references — base object, `include[]`,
`dimensions[].field` / `measures[].field`, and filter KEYS — at
`validate`/`build` (#14105)

A dataset could name a **base object that does not exist**, join a
**relationship that does not exist**, and bind every dimension and measure to
**fields that do not exist**, and `objectstack validate` exited **0** with
`✓ Validation passed`. `objectstack build` also exited 0 and wrote the dangling
dataset into `dist/objectstack.json`.

The sting was that the author-time rule pass **already walked those exact
nodes**. Measured on published 17.2.0, each mutation applied on its own and
confirmed on disk before running:

| mutation | before | after |
|:----------------------------------------------------|:-------|:------|
| dimension `field` → a base field that does not exist | passed | `dataset-field-unknown` |
| dimension `field` → a joined field that does not exist | passed | `dataset-field-unknown` |
| measure `field` → a field that does not exist | passed | `dataset-field-unknown` |
| measure filter KEY → a field that does not exist | passed | `dataset-filter-field-unknown` |
| `include[]` → a relationship that does not exist | passed | `dataset-include-unknown` |
| `object` → an object that does not exist | passed | `object-reference-unknown` |

The two controls in that measurement — a duplicate measure name
(`DatasetSchema.superRefine`) and a bad date macro in a **measure filter**
(`filter-token-unknown`) — both failed the build, so datasets were
demonstrably in the validation path the whole time. `filter-token-unknown`
already stood at `datasets[1].measures[1].filter.last_update_at.$lt` and
reasoned about the **value**; nothing standing in that same position resolved
the **key**, or the sibling `field` one level up.

This matters more for a dataset than for most metadata because a dataset is the
semantic layer: dashboards and reports bind its dimensions and measures by name
(ADR-0021), and the consumer end of that binding is already guarded
(`widget-dataset-unknown` / `widget-dimension-unknown` / `widget-measure-unknown`,
#7529/#8902). So the surviving hole was the quiet one — every binding resolves,
the board renders, and the charts are empty or subtly wrong because the dataset
underneath addresses columns that do not exist.

**Five verdicts, all `error`.** Four are new rule ids on a new suite member,
`validateDatasetReferences`:

- `dataset-include-unknown` — an `include[]` entry that resolves to nothing, or
to a field that is not a relationship, so no join can be derived from it.
- `dataset-field-unknown` — a dimension or measure `field` path that resolves to
no column, on the base object or on any joined object along the path.
- `dataset-field-not-included` — the second real check: a dotted path that
RESOLVES, but whose relationship prefix was never declared in `include`.
ADR-0021 D-C joins only declared paths, so the column is out of the query's
reach however real it is.
- `dataset-filter-field-unknown` — a filter key on `Dataset.filter` or
`measures[].filter`, in any of the three authored filter shapes.

The fifth, the base object itself, lands on `validateObjectReferences` as a new
`datasets[].object` reference site rather than as a sixth id here. That rule's
charter IS object-name references that are plain `z.string()`, and putting it
there buys the curated cross-package severity ladder: the platform's own
`system.datasets.ts` declares five datasets over `sys_*` objects, three of which
live in packages a stack compiling plugin-auth alone cannot see. All five
resolve through `PLATFORM_PROVIDED_OBJECT_NAMES`; a local "not in this stack ⇒
error" check would have reported every one of them. When the base object does
not resolve, `validateDatasetReferences` skips the dataset entirely, so one typo
yields one finding rather than one per dimension, measure and filter key.

**Skips, so a finding is never a guess** (ADR-0072 D1): an object this stack
does not define, an object with no readable field map (ADR-0015 `external` and
introspected schemas), a registry-injected system column, and any hop *through*
one — an injected `owner_id` is a lookup at the registry whose target is
invisible here, so `owner_id.name` is unanswerable rather than a miss. The
shipped `showcase_task_metrics` dimension `{ field: 'created_at' }` is that skip's
live case, and every shipped dataset in the repo is silent under the new rule.

**Two reusable seams ship with it**, newly exported, because the same two
questions are asked at a dashboard widget's filter keys and `sortBy` and at a
list view's field positions, and three independent copies of a hop-walker drift:

- `object-graph.ts` — `indexObjectGraph` / `resolveFieldPath` / `isUnjudgeable`,
answering "what does this `relationship[.relationship].field` path resolve to?"
as a discriminated **verdict** union rather than a boolean, so a caller can
tell "this hop is not a relationship" from "this leaf does not exist" and write
the right prescription. Plus `nearestName` / `suggestName` / `listNames`.
- `walkFilterFieldKeys` (`filter-walk.ts`) — the FIELD-KEY half of a filter
subtree, beside the subtree-finding half that module already owns. It handles
all three authored shapes (Mongo condition object, `{ field, operator, value }`
rules, `[field, op, value]` triples), because a reader that handles only one
shape is the exact bug #3574 was filed against, and it composes a nested
condition object into one relationship path so `{ account: { region: … } }`
reports `account.region` rather than a bare `region` resolved against the
wrong object.

Both hold mechanism only — no rule ids, no severities, no findings.
132 changes: 132 additions & 0 deletions packages/lint/src/filter-walk.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -35,8 +35,21 @@
* they resolve an additional vocabulary (`AppContextSelector` ids such as
* `{active_package}`) that is meaningless in a filter, and restricting the walk
* is what holds false positives at zero.
*
* ## The two halves of a filter, and where each one is answered
*
* `walkAuthoredFilters` finds the SUBTREES. Inside one, a rule wants either the
* VALUES (`validate-filter-tokens.ts` classifies placeholders in them;
* `validate-preset-comparands.ts` judges ordering comparands) or the FIELD KEYS
* — the names the query is filtered BY. {@link walkFilterFieldKeys} is the
* second half (#14105), and it is here rather than in its first caller because
* the shape dispatch is identical to the one `validate-preset-comparands.ts`
* already performs on values: the platform authors filters three ways, and a
* reader that handles only one shape is the exact bug #3574 was filed against.
*/

import { VALID_AST_OPERATORS } from '@objectstack/spec/data';

/** Any plain metadata record. */
type AnyRec = Record<string, unknown>;

Expand DownExpand Up@@ -162,3 +175,122 @@ export function walkAuthoredFilters(
});
}
}

// ── The FIELD-KEY half of a filter subtree (#14105) ──────────────────────────

/** One field position inside an authored filter. */
export interface FilterFieldKey {
/**
* The field the condition filters BY, exactly as authored — a bare name
* (`status`), or a dotted relationship path (`account.region`, whether
* spelled that way or reached by descending a nested condition object).
*/
field: string;
/** Config path of the position, e.g. `datasets[1].measures[1].filter.last_update_at`. */
path: string;
}

/** Recursion guard — an authored filter is a bounded document, not a graph. */
const MAX_KEY_DEPTH = 32;

function isPlainObject(v: unknown): v is AnyRec {
return !!v && typeof v === 'object' && !Array.isArray(v) && !(v instanceof Date);
}

/**
* Emit the field positions of one Mongo-style condition NODE.
*
* `prefix` carries the relationship path accumulated by descending nested
* condition objects, so `{ account: { region: { $eq: 'emea' } } }` reports the
* single position `account.region` rather than a bare `region` that would
* resolve against the wrong object. A node whose value carries `$` operators is
* a leaf condition and ends the descent; a `$`-prefixed key that is not a
* recognised combinator is skipped and NOT descended, matching
* `validate-preset-comparands.ts` — an unrecognised operator's operand shape is
* not ours to guess.
*/
function conditionFieldKeys(
node: AnyRec,
path: string,
prefix: string,
visit: (key: FilterFieldKey) => void,
depth: number,
): void {
if (depth > MAX_KEY_DEPTH) return;
for (const [key, value] of Object.entries(node)) {
const here = `${path}.${key}`;
if (key === '$and' || key === '$or') {
if (Array.isArray(value)) {
value.forEach((arm, i) => {
if (isPlainObject(arm)) conditionFieldKeys(arm, `${here}[${i}]`, prefix, visit, depth + 1);
});
}
continue;
}
if (key === '$not') {
if (isPlainObject(value)) conditionFieldKeys(value, here, prefix, visit, depth + 1);
continue;
}
if (key.startsWith('$')) continue; // unrecognised combinator
const field = prefix ? `${prefix}.${key}` : key;
if (isPlainObject(value) && !Object.keys(value).some((k) => k.startsWith('$'))) {
// Nested relation / deep equality — the field position is one level
// deeper, so descend rather than reporting the intermediate hop twice.
// An EMPTY nested object addresses nothing further; report the hop itself.
if (Object.keys(value).length > 0) {
conditionFieldKeys(value, here, field, visit, depth + 1);
continue;
}
}
visit({ field, path: here });
}
}

/**
* Emit every FIELD KEY inside one authored filter subtree, whatever shape it
* was authored in — the key half of what `validate-preset-comparands.ts` does
* for values, and the traversal `filter-token-unknown` already performs while
* reasoning only about the strings it finds.
*
* Holds no judgement: it does not know which object the filter is bound to and
* emits no findings. Resolution is {@link resolveFieldPath}'s job
* (`object-graph.ts`) and the verdict is the caller's.
*/
export function walkFilterFieldKeys(
node: unknown,
path: string,
visit: (key: FilterFieldKey) => void,
depth = 0,
): void {
if (depth > MAX_KEY_DEPTH) return;

if (Array.isArray(node)) {
// Triple: ['field', op, value] — the field position is a non-keyword
// string and the operator position is in the AST vocabulary (the
// `isFilterAST` test `validate-preset-comparands.ts` uses).
if (
typeof node[0] === 'string' && typeof node[1] === 'string'
&& !['and', 'or'].includes(node[0].toLowerCase())
&& VALID_AST_OPERATORS.has(node[1].toLowerCase())
) {
visit({ field: node[0], path: `${path}[0]` });
return;
}
// Group ['and'|'or', ...members] or a bare list — recurse the members.
node.forEach((member, i) => {
if (typeof member === 'string') return; // the leading keyword
walkFilterFieldKeys(member, `${path}[${i}]`, visit, depth + 1);
});
return;
}

if (!isPlainObject(node)) return;

// View filter rule: { field, operator[, value] }.
if (typeof node.field === 'string' && typeof node.operator === 'string') {
visit({ field: node.field, path: `${path}.field` });
return;
}

conditionFieldKeys(node, path, '', visit, depth);
}
31 changes: 31 additions & 0 deletions packages/lint/src/index.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -467,6 +467,37 @@ export {
} from './validate-chart-bindings.js';
export type { ChartBindingFinding, ChartBindingSeverity } from './validate-chart-bindings.js';

// [#14105] The layer BELOW the chart/widget binding rules above: a dataset's
// own `include[]`, `dimensions[].field`, `measures[].field` and filter KEYS,
// resolved against the object graph. Its base-object half is
// `validateObjectReferences`' `datasets[].object` site.
export {
validateDatasetReferences,
DATASET_INCLUDE_UNKNOWN,
DATASET_FIELD_UNKNOWN,
DATASET_FIELD_NOT_INCLUDED,
DATASET_FILTER_FIELD_UNKNOWN,
} from './validate-dataset-references.js';
export type { DatasetRefFinding, DatasetRefSeverity } from './validate-dataset-references.js';

// The two reusable seams the rule above is written on, exported because the
// queued siblings (#14148 widget filter keys + sortBy, #14107 list-view field
// positions) must reuse ONE mechanism rather than growing a second hop-walker
// and a second filter-key reader. Both hold mechanism only — no rule ids, no
// severities, no findings; the judgement stays with the rule that asks.
export {
indexObjectGraph,
resolveFieldPath,
isUnjudgeable,
nearestName,
suggestName,
listNames,
RELATIONSHIP_FIELD_TYPES,
} from './object-graph.js';
export type { ObjectGraph, GraphObject, GraphField, FieldPathVerdict } from './object-graph.js';
export { walkFilterFieldKeys } from './filter-walk.js';
export type { FilterFieldKey } from './filter-walk.js';

// #4762 — the two STATIC artifacts an object validation rule carries (a
// `format` rule's `regex`, a `json_schema` rule's `schema`) are fail-OPEN at
// runtime: one that does not compile is logged and skipped, so the rule is
Expand Down
Loading
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 95 additions & 0 deletions .changeset/dataset-reference-resolution.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
---
'@objectstack/lint': minor
---

Resolve an ADR-0021 dataset's own references — base object, `include[]`,
`dimensions[].field` / `measures[].field`, and filter KEYS — at
`validate`/`build` (#14105)

A dataset could name a **base object that does not exist**, join a
**relationship that does not exist**, and bind every dimension and measure to
**fields that do not exist**, and `objectstack validate` exited **0** with
`✓ Validation passed`. `objectstack build` also exited 0 and wrote the dangling
dataset into `dist/objectstack.json`.

The sting was that the author-time rule pass **already walked those exact
nodes**. Measured on published 17.2.0, each mutation applied on its own and
confirmed on disk before running:

| mutation | before | after |
|:----------------------------------------------------|:-------|:------|
| dimension `field` → a base field that does not exist | passed | `dataset-field-unknown` |
| dimension `field` → a joined field that does not exist | passed | `dataset-field-unknown` |
| measure `field` → a field that does not exist | passed | `dataset-field-unknown` |
| measure filter KEY → a field that does not exist | passed | `dataset-filter-field-unknown` |
| `include[]` → a relationship that does not exist | passed | `dataset-include-unknown` |
| `object` → an object that does not exist | passed | `object-reference-unknown` |

The two controls in that measurement — a duplicate measure name
(`DatasetSchema.superRefine`) and a bad date macro in a **measure filter**
(`filter-token-unknown`) — both failed the build, so datasets were
demonstrably in the validation path the whole time. `filter-token-unknown`
already stood at `datasets[1].measures[1].filter.last_update_at.$lt` and
reasoned about the **value**; nothing standing in that same position resolved
the **key**, or the sibling `field` one level up.

This matters more for a dataset than for most metadata because a dataset is the
semantic layer: dashboards and reports bind its dimensions and measures by name
(ADR-0021), and the consumer end of that binding is already guarded
(`widget-dataset-unknown` / `widget-dimension-unknown` / `widget-measure-unknown`,
#7529/#8902). So the surviving hole was the quiet one — every binding resolves,
the board renders, and the charts are empty or subtly wrong because the dataset
underneath addresses columns that do not exist.

**Five verdicts, all `error`.** Four are new rule ids on a new suite member,
`validateDatasetReferences`:

- `dataset-include-unknown` — an `include[]` entry that resolves to nothing, or
to a field that is not a relationship, so no join can be derived from it.
- `dataset-field-unknown` — a dimension or measure `field` path that resolves to
no column, on the base object or on any joined object along the path.
- `dataset-field-not-included` — the second real check: a dotted path that
RESOLVES, but whose relationship prefix was never declared in `include`.
ADR-0021 D-C joins only declared paths, so the column is out of the query's
reach however real it is.
- `dataset-filter-field-unknown` — a filter key on `Dataset.filter` or
`measures[].filter`, in any of the three authored filter shapes.

The fifth, the base object itself, lands on `validateObjectReferences` as a new
`datasets[].object` reference site rather than as a sixth id here. That rule's
charter IS object-name references that are plain `z.string()`, and putting it
there buys the curated cross-package severity ladder: the platform's own
`system.datasets.ts` declares five datasets over `sys_*` objects, three of which
live in packages a stack compiling plugin-auth alone cannot see. All five
resolve through `PLATFORM_PROVIDED_OBJECT_NAMES`; a local "not in this stack ⇒
error" check would have reported every one of them. When the base object does
not resolve, `validateDatasetReferences` skips the dataset entirely, so one typo
yields one finding rather than one per dimension, measure and filter key.

**Skips, so a finding is never a guess** (ADR-0072 D1): an object this stack
does not define, an object with no readable field map (ADR-0015 `external` and
introspected schemas), a registry-injected system column, and any hop *through*
one — an injected `owner_id` is a lookup at the registry whose target is
invisible here, so `owner_id.name` is unanswerable rather than a miss. The
shipped `showcase_task_metrics` dimension `{ field: 'created_at' }` is that skip's
live case, and every shipped dataset in the repo is silent under the new rule.

**Two reusable seams ship with it**, newly exported, because the same two
questions are asked at a dashboard widget's filter keys and `sortBy` and at a
list view's field positions, and three independent copies of a hop-walker drift:

- `object-graph.ts` — `indexObjectGraph` / `resolveFieldPath` / `isUnjudgeable`,
answering "what does this `relationship[.relationship].field` path resolve to?"
as a discriminated **verdict** union rather than a boolean, so a caller can
tell "this hop is not a relationship" from "this leaf does not exist" and write
the right prescription. Plus `nearestName` / `suggestName` / `listNames`.
- `walkFilterFieldKeys` (`filter-walk.ts`) — the FIELD-KEY half of a filter
subtree, beside the subtree-finding half that module already owns. It handles
all three authored shapes (Mongo condition object, `{ field, operator, value }`
rules, `[field, op, value]` triples), because a reader that handles only one
shape is the exact bug #3574 was filed against, and it composes a nested
condition object into one relationship path so `{ account: { region: … } }`
reports `account.region` rather than a bare `region` resolved against the
wrong object.

Both hold mechanism only — no rule ids, no severities, no findings.
132 changes: 132 additions & 0 deletions packages/lint/src/filter-walk.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -35,8 +35,21 @@
* they resolve an additional vocabulary (`AppContextSelector` ids such as
* `{active_package}`) that is meaningless in a filter, and restricting the walk
* is what holds false positives at zero.
*
* ## The two halves of a filter, and where each one is answered
*
* `walkAuthoredFilters` finds the SUBTREES. Inside one, a rule wants either the
* VALUES (`validate-filter-tokens.ts` classifies placeholders in them;
* `validate-preset-comparands.ts` judges ordering comparands) or the FIELD KEYS
* — the names the query is filtered BY. {@link walkFilterFieldKeys} is the
* second half (#14105), and it is here rather than in its first caller because
* the shape dispatch is identical to the one `validate-preset-comparands.ts`
* already performs on values: the platform authors filters three ways, and a
* reader that handles only one shape is the exact bug #3574 was filed against.
*/

import { VALID_AST_OPERATORS } from '@objectstack/spec/data';

/** Any plain metadata record. */
type AnyRec = Record<string, unknown>;

Expand DownExpand Up@@ -162,3 +175,122 @@ export function walkAuthoredFilters(
});
}
}

// ── The FIELD-KEY half of a filter subtree (#14105) ──────────────────────────

/** One field position inside an authored filter. */
export interface FilterFieldKey {
/**
* The field the condition filters BY, exactly as authored — a bare name
* (`status`), or a dotted relationship path (`account.region`, whether
* spelled that way or reached by descending a nested condition object).
*/
field: string;
/** Config path of the position, e.g. `datasets[1].measures[1].filter.last_update_at`. */
path: string;
}

/** Recursion guard — an authored filter is a bounded document, not a graph. */
const MAX_KEY_DEPTH = 32;

function isPlainObject(v: unknown): v is AnyRec {
return !!v && typeof v === 'object' && !Array.isArray(v) && !(v instanceof Date);
}

/**
* Emit the field positions of one Mongo-style condition NODE.
*
* `prefix` carries the relationship path accumulated by descending nested
* condition objects, so `{ account: { region: { $eq: 'emea' } } }` reports the
* single position `account.region` rather than a bare `region` that would
* resolve against the wrong object. A node whose value carries `$` operators is
* a leaf condition and ends the descent; a `$`-prefixed key that is not a
* recognised combinator is skipped and NOT descended, matching
* `validate-preset-comparands.ts` — an unrecognised operator's operand shape is
* not ours to guess.
*/
function conditionFieldKeys(
node: AnyRec,
path: string,
prefix: string,
visit: (key: FilterFieldKey) => void,
depth: number,
): void {
if (depth > MAX_KEY_DEPTH) return;
for (const [key, value] of Object.entries(node)) {
const here = `${path}.${key}`;
if (key === '$and' || key === '$or') {
if (Array.isArray(value)) {
value.forEach((arm, i) => {
if (isPlainObject(arm)) conditionFieldKeys(arm, `${here}[${i}]`, prefix, visit, depth + 1);
});
}
continue;
}
if (key === '$not') {
if (isPlainObject(value)) conditionFieldKeys(value, here, prefix, visit, depth + 1);
continue;
}
if (key.startsWith('$')) continue; // unrecognised combinator
const field = prefix ? `${prefix}.${key}` : key;
if (isPlainObject(value) && !Object.keys(value).some((k) => k.startsWith('$'))) {
// Nested relation / deep equality — the field position is one level
// deeper, so descend rather than reporting the intermediate hop twice.
// An EMPTY nested object addresses nothing further; report the hop itself.
if (Object.keys(value).length > 0) {
conditionFieldKeys(value, here, field, visit, depth + 1);
continue;
}
}
visit({ field, path: here });
}
}

/**
* Emit every FIELD KEY inside one authored filter subtree, whatever shape it
* was authored in — the key half of what `validate-preset-comparands.ts` does
* for values, and the traversal `filter-token-unknown` already performs while
* reasoning only about the strings it finds.
*
* Holds no judgement: it does not know which object the filter is bound to and
* emits no findings. Resolution is {@link resolveFieldPath}'s job
* (`object-graph.ts`) and the verdict is the caller's.
*/
export function walkFilterFieldKeys(
node: unknown,
path: string,
visit: (key: FilterFieldKey) => void,
depth = 0,
): void {
if (depth > MAX_KEY_DEPTH) return;

if (Array.isArray(node)) {
// Triple: ['field', op, value] — the field position is a non-keyword
// string and the operator position is in the AST vocabulary (the
// `isFilterAST` test `validate-preset-comparands.ts` uses).
if (
typeof node[0] === 'string' && typeof node[1] === 'string'
&& !['and', 'or'].includes(node[0].toLowerCase())
&& VALID_AST_OPERATORS.has(node[1].toLowerCase())
) {
visit({ field: node[0], path: `${path}[0]` });
return;
}
// Group ['and'|'or', ...members] or a bare list — recurse the members.
node.forEach((member, i) => {
if (typeof member === 'string') return; // the leading keyword
walkFilterFieldKeys(member, `${path}[${i}]`, visit, depth + 1);
});
return;
}

if (!isPlainObject(node)) return;

// View filter rule: { field, operator[, value] }.
if (typeof node.field === 'string' && typeof node.operator === 'string') {
visit({ field: node.field, path: `${path}.field` });
return;
}

conditionFieldKeys(node, path, '', visit, depth);
}
31 changes: 31 additions & 0 deletions packages/lint/src/index.ts
Original file line numberDiff line numberDiff line change
Expand Up@@ -467,6 +467,37 @@ export {
} from './validate-chart-bindings.js';
export type { ChartBindingFinding, ChartBindingSeverity } from './validate-chart-bindings.js';

// [#14105] The layer BELOW the chart/widget binding rules above: a dataset's
// own `include[]`, `dimensions[].field`, `measures[].field` and filter KEYS,
// resolved against the object graph. Its base-object half is
// `validateObjectReferences`' `datasets[].object` site.
export {
validateDatasetReferences,
DATASET_INCLUDE_UNKNOWN,
DATASET_FIELD_UNKNOWN,
DATASET_FIELD_NOT_INCLUDED,
DATASET_FILTER_FIELD_UNKNOWN,
} from './validate-dataset-references.js';
export type { DatasetRefFinding, DatasetRefSeverity } from './validate-dataset-references.js';

// The two reusable seams the rule above is written on, exported because the
// queued siblings (#14148 widget filter keys + sortBy, #14107 list-view field
// positions) must reuse ONE mechanism rather than growing a second hop-walker
// and a second filter-key reader. Both hold mechanism only — no rule ids, no
// severities, no findings; the judgement stays with the rule that asks.
export {
indexObjectGraph,
resolveFieldPath,
isUnjudgeable,
nearestName,
suggestName,
listNames,
RELATIONSHIP_FIELD_TYPES,
} from './object-graph.js';
export type { ObjectGraph, GraphObject, GraphField, FieldPathVerdict } from './object-graph.js';
export { walkFilterFieldKeys } from './filter-walk.js';
export type { FilterFieldKey } from './filter-walk.js';

// #4762 — the two STATIC artifacts an object validation rule carries (a
// `format` rule's `regex`, a `json_schema` rule's `schema`) are fail-OPEN at
// runtime: one that does not compile is logged and skipped, so the rule is
Expand Down
Loading
Loading