Skip to content

createDataSource declares the scannable shape it already returns - #985

Merged
bgmcmullen merged 1 commit into
masterfrom
fix/createdatasource-returns-scannable
Aug 21, 2026
Merged

createDataSource declares the scannable shape it already returns#985
bgmcmullen merged 1 commit into
masterfrom
fix/createdatasource-returns-scannable

Conversation

@bgmcmullen

Copy link
Copy Markdown
Contributor

What

Two return types in hypaware-plugin-kernel-types.d.ts move from
AsyncDataSource to the ScannableDataSource that #982 introduced:
DatasetRegistration.createDataSource and SinkQueryReader.createDataSource.

Type-only. No .js touched, no runtime behaviour changes.

Why

#982 bumped squirreling 0.15.3 to 0.16.1, where AsyncDataSource became a
union: a source implements either scan() (row) or prepareScan() + schema
(native batch). It added ScannableDataSource for the stronger shape, noting
that Hypaware's "storage, union, visibility, and legacy parquet adapters all
guarantee this stronger shape even when they also expose prepared native
batches", and migrated the internals accordingly: every dataset plugin
annotates ScannableDataSource, and dataSourceForTable declares
Promise<ScannableDataSource | null>.

The two plugin-facing interfaces stayed on the wide union. So the contract a
consumer reads promises less than every implementation delivers: receive a
source through DatasetRegistration, and scan() and columns are possibly
undefined.

Why CI here stayed green

Nothing in this repo consumes that contract the way a downstream repo does:

  • the one production caller (src/core/query/sql.js:440) hands the source
    straight to squirreling as a tables entry, and squirreling accepts the
    union by design
  • the one test that iterates rows itself narrowed at the call site rather
    than fixing the declaration, in Use native Icebird batches for faster queries #982 itself:
+ const scan = source.scan+ assert.ok(scan, 'cache-backed registration retains the row scan')- for await (const row of source.scan({}).rows()) {+ for await (const row of scan.call(source, {}).rows()) {

Everything else implementing createDataSource is a producer, already
annotated with the strong type.

Downstream

hypaware-server consumes these interfaces and iterates rows in five places
(plugins/github/src/dataset.js, plus three test files). Its typecheck went
red the first time CI resolved a client containing #982, against server code
that had not changed, because it consumes the client via file:../hypaware
and checks out this repo's default branch on every run. This change fixes it
with no server-side change at all.

Tradeoff worth a look

Narrowing a return type is free for callers but tightens the contract on
implementers: after this, anything implementing DatasetRegistration or
SinkQueryReader must return a source carrying scan(), so a plugin
returning batch-only (schema + prepareScan, no scan) becomes a type
error.

Every in-repo implementation already satisfies it, and prepareScan is added
alongside scan and never instead of it (ai-gateway/src/dataset.js:244
wraps and keeps both), so this looks like the intent of #982 rather than a
restriction. But it is an API decision rather than a typo fix, so please push
back if batch-only registrations are meant to be expressible.

If you would rather keep the wide interface, the alternative is that every
consumer narrows at its call sites the way the telemetry test does above.

Verification

Static: all in-repo implementations already return the strong shape, either
annotated ScannableDataSource or returning dataSourceForTable, which
declares it. This PR's CI run is the actual gate for typecheck and test.

🤖 Generated with Claude Code

#982 widened `AsyncDataSource` (squirreling 0.15.3 to 0.16.1) into a
row-or-batch union and added `ScannableDataSource` for the stronger shape
Hypaware's adapters actually guarantee. It then updated the internals:
every dataset plugin annotates `ScannableDataSource`, and
`dataSourceForTable` declares `Promise<ScannableDataSource | null>`.
The two plugin-facing interfaces were left on the wide union, so a
consumer that receives a source through `DatasetRegistration` or
`SinkQueryReader` is handed a value whose `scan()` and `columns` are
possibly undefined, even though every source Hypaware produces has them.
Nothing inside this repo noticed: the one production caller passes the
source straight to squirreling, which accepts the union by design, and
the one test that iterates rows itself narrowed at the call site instead
(`assert.ok(scan)` in claude-telemetry-events-dataset.test.js, added by
the same PR).
Downstream it is not avoidable. hypaware-server consumes these
interfaces and iterates rows in five places; its typecheck went red the
first time CI resolved a client with this surface, against server code
that had not changed.
Declaring what the implementations already promise. Type-only: no
runtime behaviour changes, and no source Hypaware builds is affected,
since `prepareScan` is always added alongside `scan`, never instead of
it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@bgmcmullen
bgmcmullen added this pull request to the merge queueAug 21, 2026
Merged via the queue into master with commit 41f2888Aug 21, 2026
8 checks passed
@bgmcmullen
bgmcmullen deleted the fix/createdatasource-returns-scannable branch August 21, 2026 19:54
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@bgmcmullen