Skip to content

sqlite: add virtual table support via createModule() - #65787

Open
TrevorBurnham wants to merge 1 commit into
nodejs:mainfrom
TrevorBurnham:sqlite-vtab-create-module
Open

sqlite: add virtual table support via createModule()#65787
TrevorBurnham wants to merge 1 commit into
nodejs:mainfrom
TrevorBurnham:sqlite-vtab-create-module

Conversation

@TrevorBurnham

@TrevorBurnhamTrevorBurnham commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Fixes#61539

This PR is a continuation of #61544 by @byteforge38, which became inactive.

It exposes SQLite's virtual table API through a new database.createModule(name, options) method, wrapping sqlite3_create_module_v2().

This PR enables read-only virtual tables backed by JavaScript data sources. A registered module can be used two ways:

  • Eponymous table: query the module name directly (SELECT * FROM module_name).
  • Named virtual table: CREATE VIRTUAL TABLE t USING module_name.

Hidden columns pass parameters via table-valued function syntax (SELECT * FROM module_name(param1, param2)).

options accepts columns, rows, directOnly, and useBigIntArguments.

Note: column affinity

A virtual table doesn't apply column affinity to the values xColumn returns, which makes declared types behave differently than they do on an ordinary table:

// real table: affinity converts the bound doubledb.exec('CREATE TABLE t(v INTEGER)');db.prepare('INSERT INTO t VALUES (?)').run(7);db.prepare('SELECT typeof(v) FROM t').get();// 'integer'// virtual table: no affinity applieddb.createModule('gs',{columns: [{name: 'v',type: 'INTEGER'}],*rows(){yield[7];}});db.prepare('SELECT typeof(v) FROM gs').get();// 'real'

This follows from the conversion rules discussed in #63826: A number is bound as REAL, a bigint as INTEGER. Value-based coercion would make it impossible to yield 666.0 into a REAL column, and bigint already gives callers explicit control.

Coercing to the declared type would be a different mechanism, driven by explicit intent rather than by guessing from the value, and it would leave REAL columns alone. But SQLite core doesn't do this for virtual tables (generate_series returns integers because it calls sqlite3_result_int64, not via affinity), and it adds per-cell cost.

I've documented the current behavior rather than changing it, since coercing to the declared type is behavior SQLite core doesn't have for virtual tables.

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/sqlite

@nodejs-github-botnodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. sqlite Issues and PRs related to the SQLite subsystem. labels Sep 4, 2026
@TrevorBurnham
TrevorBurnhamforce-pushed the sqlite-vtab-create-module branch 2 times, most recently from 7862ab0 to 69f939aCompareSeptember 4, 2026 19:28
Expose SQLite's virtual table API through a new
`database.createModule(name, options)` method, wrapping
`sqlite3_create_module_v2()`. This enables read-only virtual tables
backed by JavaScript data sources, usable either as an eponymous table
(`SELECT * FROM module_name`) or via `CREATE VIRTUAL TABLE t USING
module_name`. Hidden columns pass parameters using table-valued
function syntax (`SELECT * FROM module_name(param1, param2)`).
`options` accepts `columns`, `rows`, `directOnly`, and
`useBigIntArguments`. Column types are validated against INTEGER,
TEXT, REAL, BLOB, and ANY, and column names are quoted when building
the `sqlite3_declare_vtab()` schema.
Rebased from nodejs#61544, which was
opened by byteforge38 and became inactive. Changes on top of that
work:
- xColumn reports the value each hidden column was constrained to,
rather than NULL. SQLite treats xBestIndex's `omit` as a hint, so it
may recheck a constraint it already handed to xFilter; against NULL
that recheck rejected every row, and `gs(1, 3) WHERE start = 1`
returned no rows.
- xBestIndex lowers estimatedCost as it consumes constraints. With a
constant cost the planner was free to pick the unconstrained plan and
recheck afterwards, so a correlated parameter such as
`FROM t, gs(t.a, t.a + 1)` also returned no rows.
- Violations of the iteration protocol report a SQLite error instead of
calling PropagateJSError with no JavaScript exception pending. That
left `.all()` returning undefined and `exec()` reporting success.
- xBestIndex passes the constrained hidden-column indices to xFilter
through idxStr rather than an int bitmask, which previously aliased
for parameter indices at or above the width of an int.
- xFilter, xNext, and xColumn take a CallbackDepthGuard. Without it
close() from inside rows(), an iterator's next(), or a row getter
finalized the statement that SQLite was still stepping, crashing the
process.
- xClose calls the iterator's return() method so generator `finally`
blocks run when SQLite stops stepping early, as it does for LIMIT or
a `break` out of a for...of loop. It is skipped while tearing down
from ~StatementSync or ~DatabaseSync, which run from garbage
collection callbacks where JavaScript cannot be executed; an
abandoned generator does not run `finally` in JavaScript either. It
is also skipped when an error is already pending, so that error still
reaches the caller.
- VirtualTableModule holds a BaseObjectWeakPtr<DatabaseSync> to match
UserDefinedFunction instead of a raw pointer.
- createModule() rejects being called from an authorizer callback.
- Documents that values yielded by rows() follow the usual conversion
rules, so a number is stored as REAL and a BigInt as INTEGER even
when a column declares INTEGER, since virtual tables do not apply
column affinity to the values they return.
Refs: nodejs#61544
Refs: nodejs#63826Fixes: nodejs#61539
Co-authored-by: byteforge38 <stormcraft318@gmail.com>
Signed-off-by: Trevor Burnham <trevorburnham@gmail.com>
Assisted-by: Claude Opus 5
@TrevorBurnham
TrevorBurnhamforce-pushed the sqlite-vtab-create-module branch from 69f939a to 89b3a3aCompareSeptember 4, 2026 21:04
@TrevorBurnham
TrevorBurnham marked this pull request as ready for review September 4, 2026 23:17
@codecov

codecovBot commented Sep 5, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.25000% with 87 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.12%. Comparing base (68e0ce0) to head (89b3a3a).
⚠️ Report is 2 commits behind head on main.

Files with missing linesPatch %Lines
src/node_sqlite.cc80.96%44 Missing and 43 partials ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #65787 +/- ##
==========================================
- Coverage 90.13% 90.12% -0.02% 
==========================================
Files 769 769 Lines 261645 262109 +464 Branches 49671 49755 +84 ==========================================
+ Hits 235831 236217 +386 - Misses 16845 16856 +11 - Partials 8969 9036 +67 
Files with missing linesCoverage Δ
src/node_sqlite.h87.27% <100.00%> (+0.86%)⬆️
src/node_sqlite.cc82.04% <80.96%> (-0.20%)⬇️

... and 36 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

c++Issues and PRs that require attention from people who are familiar with C++.needs-ciPRs that need a full CI run.sqliteIssues and PRs related to the SQLite subsystem.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose SQLite virtual table API

2 participants

@TrevorBurnham@nodejs-github-bot