sqlite: support db.loadExtension

[himself65]

Closes#53898

This PR will implement loadExtension API to align with other SQLite3 JS library like better-sqlite3, node-sqlite3, jsr:@db/sqlite, bun:sqlite

Example Code:

import{load}from'sqlite-vec'import{DatabaseSync}from'node:sqlite'constdb=newDatabaseSync(':memory:',{allowExtension: true})load(db)// db.loadExtension() supportconst{ sqlite_version, vec_version }=db.prepare('select sqlite_version() as sqlite_version, vec_version() as vec_version;').get()console.log(`SQLite version: ${sqlite_version}, vector version ${vec_version}`)constitems=[[1,[0.1,0.1,0.1,0.1]],[2,[0.2,0.2,0.2,0.2]],[3,[0.3,0.3,0.3,0.3]],[4,[0.4,0.4,0.4,0.4]],[5,[0.5,0.5,0.5,0.5]]]constquery=[0.3,0.3,0.3,0.3]db.exec('CREATE VIRTUAL TABLE vec_items USING vec0(embedding float[4])')constinsertStmt=db.prepare('INSERT INTO vec_items(rowid, embedding) VALUES (?, ?)')for(const[id,vector]ofitems){constrowId=BigInt(id)constembedding=newUint8Array(Float32Array.from(vector).buffer)insertStmt.run(rowId,embedding)}constrows=db.prepare(` SELECT rowid, distance FROM vec_items WHERE embedding MATCH ? ORDER BY distance LIMIT 3`).all(newUint8Array(Float32Array.from(query).buffer))console.log(rows)

> sqlite-vec /Users/himself65/Code/node/out/Debug/node --experimental-sqlite ./index.mjs
(node:72169) ExperimentalWarning: SQLite is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
SQLite version: 3.46.0, vector version v0.1.1
[
{ rowid: 3, distance: 0 },
{ rowid: 4, distance: 0.19999998807907104 },
{ rowid: 2, distance: 0.20000001788139343 }
]

[himself65]

Found another small issue: nodejs/core-validate-commit#122

[avivkeller]

Found another small issue: nodejs/core-validate-commit#122

Duplicate of nodejs/core-validate-commit#121

[himself65]

I will re start this PR

[himself65]

I have no idea how to write a test for load extension now.

[himself65]

/cc @nodejs/sqlite (no such group?)

anyway I cc @nodejs/tsc since this is a breaking change

[codecov]

Codecov Report

Attention: Patch coverage is 69.44444% with 22 lines in your changes missing coverage. Please review.

Project coverage is 88.52%. Comparing base (3c2da4b) to head (08280ba).
Report is 233 commits behind head on main.

Files with missing lines	Patch %	Lines
src/node_sqlite.cc	69.44%	8 Missing and 14 partials ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #53900 +/- ##
==========================================
+ Coverage 87.99% 88.52% +0.53% 
==========================================
Files 656 657 +1 Lines 188999 189929 +930 Branches 35981 36472 +491 ==========================================
+ Hits 166301 168133 +1832 + Misses 15865 14992 -873 + Partials 6833 6804 -29 

Files with missing lines	Coverage Δ
src/node_errors.h	85.00% <ø> (ø)
src/node_sqlite.h	70.00% <ø> (ø)
src/node_sqlite.cc	80.25% <69.44%> (-0.93%)	⬇️

... and 108 files with indirect coverage changes

[benjamingr]

cc @cjihrig

[cjihrig]

since this is a breaking change

I took a quick look at the code. I'm not sure what makes this a breaking change though.

I do think the TSC needs to decide if they want to support loading arbitrary native code in this manner. It wouldn't be the first way that Node supports loading native code, so maybe it's fine.

I have no idea how to write a test for load extension now.

Is it possible to create a very trivial extension that can be accessed as a test fixture?

[tniessen]

I do think the TSC needs to decide if they want to support loading arbitrary native code in this manner.

It definitely must be disabled when the experimental permission model is enabled, just like native add-ons etc.

[himself65]

For the native test, I'm not sure if we should build an SQLite binding from scratch or if we should just copy some community dll/so/lib to fixtures.

[targos]

We also need a test that tries to load an unexpected file (that is not a sqlite extension). It should throw, not crash the process.

[jakecastelli]

I'm not sure if we should build an SQLite binding from scratch or if we should just copy some community dll/so/lib to fixtures.

IMO we should build from scratch to avoid any potential attack and make update tests fixture easier if we ever need to update them.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/63916/

[mcollina]

lgtm

[cjihrig]

FYI - I think test/parallel/test-permission-sqlite-load-extension.js is legitimately failing on Windows. I'm not sure if it's related to the use of common.escapePOSIXShell in that test or what.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/63919/

[himself65]

not sure how to fix it right now. setting up local env on my local windows pc

[himself65]

ok the error is legitimately falling, testing on my windows locally, I think because wrong usage of the cp api

[himself65]

OK, I found because \n on windows will truncate the script code, remove it

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/63985/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/63993/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/63994/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/63995/

[himself65]

will merge this today

[himself65]

Landed in 5c2f599

[aduh95]

It seems this doesn't work when node is built with --shared-sqlite