Uh oh!
There was an error while loading. Please reload this page.
feat(sqllite): npm dev launcher - #258
Conversation
`npm install extenddb` then one call: spawn the dev binary, wait for health,
hand back an endpoint, region and the seeded example credentials that any AWS
SDK accepts, plus a stop handle. A CLI (`npx extenddb start`) wraps the same
function for shell use.
Storage semantics are the deliberate part. File-based is the DEFAULT, at
./.extenddb/data.db relative to the caller's working directory: a database
that vanishes on exit surprises anyone who put data in it on purpose, so
persistence is what you get without asking, and the dotdir keeps one
project's tables out of another's. Memory mode is one flag away (--memory /
{ memory: true }) for tests and CI. Asking for both a persistence path and
memory mode is an error rather than a silent choice, because ignoring a path
the caller asked to persist to is data loss by configuration.
The launcher is a thin process supervisor, not a client: it configures the
server only through environment variables (EXTENDDB__SERVER__PORT,
EXTENDDB__STORAGE__SQLITE__PATH) from a scratch working directory so a stray
config file cannot leak in, resolves the binary from an explicit option,
EXTENDDB_BINARY, a platform package, or PATH, polls /health with a deadline,
and terminates the child with SIGTERM escalating to SIGKILL. Platform
binaries will ship as optionalDependencies (@extenddb/<platform>-<arch>), the
copy-only pattern that needs no compiler, no node-gyp and no postinstall
downloads; the release pipeline that publishes those packages is a separate
change.
packaging/README.md defines the launcher CONTRACT, not just this wrapper:
default path, memory flag, env-var configuration, health gating and stop
semantics are ecosystem-neutral on purpose, because pip and Maven launchers
follow and must not drift from npm on defaults.
Verified against a real dev-mode binary built from this commit's tree
(config-less serve + seeded credentials + CRUD + file persistence across a
restart were each proven live before the wrapper was written): the four test
scenarios pass, and the negative control discriminates: reverting the default
to memory fails the suite with "default storage must be the project dotdir
file, got :memory:".
Prior art: the earlier launcher exploration on an unrebaseable pre-merge
branch was ported fresh, with the storage default inverted per the product
decision and the platform-package scope aligned to @extenddb/*.…solve failure The release workflow mirrors release-image.yml's posture deliberately: dispatch-only from main (no tag trigger, so the reviewed definition runs), a gate that validates the npm-vX.Y.Z input as strict shape / existing tag / ancestor of main / version match against packaging/npm/package.json, five NATIVE runners (no cross-compilation) that each smoke test the binary and run the full launcher suite against it, and a single credentialed job in the reviewer-gated 'npm' environment that publishes with --provenance under the 'candidate' dist-tag only. Promotion to 'latest' is a manual dist-tag move documented in packaging/README.md. Platform packages ship the binary at the package root, matching the launcher's require.resolve contract, with os/cpu fields so npm selects exactly one via optionalDependencies. The launcher's optionalDependencies are rewritten at publish to pin the exact release version. Verified locally: gate script simulated with negative controls (four malformed tags rejected, version match proven); the publish job's staging logic run verbatim against the real dev binary, packed, and installed into a scratch project where the launcher resolved the platform package and served with EXTENDDB_BINARY unset and extenddb absent from PATH; negative control (platform package removed) fails, now with an actionable message naming the four resolution steps instead of a bare spawn ENOENT. Launcher suite: 4/4 after the error-path change.
Builds the slim dev-mode binary and runs the launcher suite, making the release workflow's 'same suite that gates packaging/ in CI' description true. PR coverage is Linux; all five platforms run in npm-publish.
…lose drift Two independent reviews (launcher internals: HOLD; supply chain: SHIP) produced two blockers and four should-fixes, all closed here. Blocker 1: the persistence test proved nothing. It asserted file size > 0 and a health check, which a server recreating its schema fresh every boot would also pass. The suite now signs real SigV4 DynamoDB calls in plain node (no SDK dependency): scenario 2 writes an item in lifetime one and reads it back in lifetime two; scenario 3 writes in memory mode and proves a restarted memory server does NOT know the table. Both negative controls discriminate: pointing run two at a different path fails the survival assertion, and sabotaging memory mode into a shared file fails scenario 3. Blocker 2: all six public tarballs omitted the Apache-2.0 LICENSE. The publish workflow now stages LICENSE and NOTICE from the repo root into every package (npm includes LICENSE automatically; NOTICE is added to each files whitelist). Verified by npm pack dry-run on both package shapes. Platform-list drift (supply chain #2): the publish loop now derives the platform list from the launcher's optionalDependencies, the single source of truth, and fails loudly on a missing template or missing build artifact before anything is published. Drift guard proven with a synthetic linux-ppc64 entry. Paths gap (supply chain #1): launcher.yml now also triggers on crates/**, Cargo.toml, Cargo.lock, so a server change that breaks the dev-mode contract is caught at PR time, not at release time. Smaller findings: --port and --db now reject missing or malformed values with usage instead of exporting EXTENDDB__SERVER__PORT=NaN or silently using the default; the resolution failure message now names the musl/Alpine case (a present glibc binary fails with the same ENOENT); README documents the hard-kill orphan, musl, and shared-file limits. Launcher suite: 4/4 against the real dev binary after all changes.
The unscoped `extenddb` name cannot be published. npm's package-name similarity filter rejects it with 403 Forbidden - PUT https://registry.npmjs.org/extenddb Package name too similar to existing package extend `extend` is the long-established jQuery.extend port, two edit operations away from `extenddb`. The check runs server-side on PUT, so there is no way to detect it before attempting a publish and no way to work around it from the package side; the only options are a support appeal for the name or a scoped name. The scoped name is available now and needs no appeal, so the launcher becomes `@extenddb/dev`. Verified: the five `@extenddb/<os>-<arch>` platform packages publish without complaint (the scope makes them unambiguous to the filter), so this affects the launcher only. Scope of the change is the registry name and the docs that quote it. The CLI command stays `extenddb` (the `bin` key is unchanged), so `npx extenddb start` and every log, process, and binary reference are untouched. optionalDependencies still pin the same five platform packages, and the publish workflow already passes `--access public`, which a scoped package requires.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
…oken
Replaces the stored NPM_TOKEN with npm trusted publishing, so there is no
long-lived publish credential in the repository at all. GitHub mints an
OIDC token for the run, npm exchanges it for a short-lived credential
scoped to that publish, and nothing survives the job.
Three changes make it work:
- Both `NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}` lines are gone. npm
picks up the OIDC path automatically when no token is present and
`id-token: write` is granted, and `--provenance` becomes implicit
(kept explicit for readability).
- npm is upgraded to 11 before publishing. The OIDC exchange needs
>= 11.5.1 and the Node 22 runner image bundles npm 10, which would
silently fall back to looking for a token and fail with ENEEDAUTH.
- The comments now record that `environment: npm` is load-bearing for
security rather than merely holding a secret: npm matches the
environment name as part of the trust relationship, so a run outside
that environment cannot mint a publish credential even from this
repository and workflow.
The registry side is configured per package (all six), matching exactly
ExtendDB / extenddb / npm-publish.yml / environment npm. Because every
field is matched literally, renaming this workflow file breaks publishing
until the six entries are updated; the header now says so.
npm requires a package to exist before a trusted publisher can be
configured, so all six were first published as 0.0.0 placeholders under
the `placeholder` dist-tag from a maintainer machine. That is also why the
unscoped launcher name surfaced as unpublishable and became @extenddb/dev.…d v tag The launcher had its own `npm-vX.Y.Z` tag namespace and treated packaging/npm/package.json as the source of truth, on the reasoning that it versions independently of the server. That reasoning does not survive contact with what the packages actually contain. The five platform packages ship the server binary. Any server release you want available through npm therefore republishes all five, and the launcher pins its optionalDependencies to its own version, so it republishes too. An independent launcher number buys exactly one thing, shipping a launcher-only fix without a server bump, and costs the ability to answer the question users actually ask: which server is inside @extenddb/dev@X.Y.Z. For a package whose entire payload is the server, that is the wrong trade. It is also the settled convention for this package shape: esbuild@0.28.2 pins @esbuild/* at 0.28.2, @biomejs/biome@2.5.8 pins @biomejs/cli-* at 2.5.8 (both checked against the registry, not recalled). So: - the dispatch input is now a `vX.Y.Z` tag, the same tag release-image.yml consumes, with the two publish workflows dispatched independently against it - the gate validates the tag against the workspace version in Cargo.toml, the single source of truth for a release across every distribution channel, and additionally refuses a drifted packaging/npm/package.json (the publish step rewrites that value anyway, so stale content would not break a release, it would quietly mislead whoever reads the manifest) - packaging/npm/package.json moves 0.1.0 -> 0.1.3 to match the workspace today, so the first real publish starts life aligned Accepted cost, recorded in packaging/README.md: a launcher-only fix rides the next server release rather than shipping under its own number. Also documents, at the step CodeQL points at, why the two actions/cache-poisoning/poisonable-step alerts on this workflow are not exploitable: `gate` has already proved the dispatched SHA is an ancestor of origin/main, so the code being built is reviewed code on main, and nothing in the workflow reads or writes the Actions cache at all (no actions/cache, no rust-cache, no `cache:` on setup-node), so there is no cache entry to poison. The comment states what must stay true if the job grows.
| - name: Install Rust toolchain | ||
| uses: dtolnay/rust-toolchain@1.97.0 | ||
| - name: Build the slim dev-mode binary |
There was a problem hiding this comment.
CodeQL alert 14 (actions/cache-poisoning/poisonable-step, high) lands on this build step.
Two properties make this unexploitable, neither visible to the query:
gatehas already proved this SHA is an ancestor oforigin/main(git merge-base --is-ancestor, line 128), so the code executed here is reviewed code already on main. Poisoning it needs push access to main, which permits strictly worse things directly.- Nothing in this workflow reads or writes the Actions cache: no
actions/cache, norust-cache, nocache:input onsetup-node. There is no cache entry to poison.
Both are recorded in the comment above the checkout, including what has to stay true if this job grows: adding a cache step here would make the finding real and would then need a trusted-ref guard.
Flagging rather than dismissing, since dismissing a security alert on my own PR is your call.
| set -euo pipefail | ||
| "target/release/${BIN_NAME}" --version | ||
| - name: Run the launcher test suite against the built binary |
There was a problem hiding this comment.
CodeQL alert 15 (actions/cache-poisoning/poisonable-step, high) lands on this test step, same cause as alert 14 on the build step above.
Two properties make this unexploitable, neither visible to the query:
gatehas already proved this SHA is an ancestor oforigin/main(git merge-base --is-ancestor, line 128), so the code executed here is reviewed code already on main. Poisoning it needs push access to main, which permits strictly worse things directly.- Nothing in this workflow reads or writes the Actions cache: no
actions/cache, norust-cache, nocache:input onsetup-node. There is no cache entry to poison.
Both are recorded in the comment above the checkout, including what has to stay true if this job grows: adding a cache step here would make the finding real and would then need a trusted-ref guard.
Flagging rather than dismissing, since dismissing a security alert on my own PR is your call.
LeeroyHannigan
commented
Aug 13, 2026
Three changes since this PR was last looked at, plus an explanation of the red CodeQL check. The launcher is now
Publishing authenticates with trusted publishing (OIDC), so no npm token exists in the repo. Both The launcher now versions with the server, off the same On the failing CodeQL check: two high Everything else is green, 17 checks including all three MongoDB jobs. |
npm-publish's gate refuses a manifest that has drifted from the workspace version: 'packaging/npm/package.json at <sha> says 0.1.3, expected 0.1.7'. The workspace is going to 0.1.7 (#296), so the manifest follows. optionalDependencies move 0.1.0 -> 0.1.7 in the same edit. The publish step already rewrites them to the release version, so this is not load-bearing, but a checked-in pin that no published platform package has ever carried misleads anyone reading the manifest. Verified with a real dev-mode binary: launcher suite 4/4 scenarios passed, and the CLI serves in memory mode (endpoint + seeded example credential + ':memory: (ephemeral)').
LeeroyHannigan
commented
Aug 19, 2026
Cleaned this up for review ( Manifest bumped to 0.1.7. Updated from main (was 52 commits behind, predating the whole vector search merge and the dev image's 18080 port). Clean merge, no conflicts. The two CodeQL alerts are dismissed as false positives. Verified rather than assumed: Verified with a real dev-mode binary, not just static review: launcher suite 4/4 scenarios passed, and the CLI serves correctly in memory mode (endpoint, seeded example credential, Worth noting the 18080 dev-image port change does not affect the launcher: it sets Sequencing after this merges: #296 lands the 0.1.7 workspace bump -> tag |
robinnsc
left a comment
There was a problem hiding this comment.
Read the launcher, the CLI, the test suite, and both workflows. This is a well designed distribution: the candidate first flow, keeping registry credentials out of every job that executes a freshly built binary, the dispatch only trigger with the ancestry gate, the drift guards derived from optionalDependencies, and persistence tests that actually discriminate (the negative controls in the description are the kind of thing most packaging PRs skip). Approving with three findings inline, one of them a real behavior bug, all small fixes.
Two things I noticed but consider fine as they are:
freePort()has the usual bind, close, reuse race: between the probe closing and the child binding, another process can take the port, and in the worst casewaitHealthygets a 200 from an unrelated local server. Inherent to the ephemeral port launcher pattern and vanishingly unlikely in practice; a code comment acknowledging it would do.- A publish failure mid loop leaves a partial candidate release, and npm version immutability means the same version cannot be rerun. The candidate dist tag means users never see it and recovery is a version bump, so this is acceptable, but a line in the README release section would save the first maintainer who hits it some head scratching.
| reject(new Error(`extenddb exited during startup (code ${code}): ${stderrTail}`)) | ||
| ); | ||
| }); | ||
| await Promise.race([waitHealthy(endpoint, options.startupTimeoutMs ?? 15000), spawned]); |
There was a problem hiding this comment.
The timeout path here leaks the child. If waitHealthy rejects while the server is alive but slow, start() throws and the caller has no handle to the process, so it keeps running and holding its port and the SQLite lock until the caller's process exits. For a test runner that catches the error and carries on, the next start() against the same dbPath then fails on the lock error, which the README describes as a user mistake rather than the leak it actually is.
Killing the child before rethrowing on any post spawn startup failure closes it:
try{awaitPromise.race([waitHealthy(endpoint,options.startupTimeoutMs??15000),spawned]);}catch(err){child.kill("SIGKILL");throwerr;}| One-time registry and repo setup, in order: create the npm org `extenddb`. | ||
| The launcher ships as the scoped `@extenddb/dev` rather than an unscoped | ||
| `extenddb`, because npm's package-name similarity filter refuses the latter | ||
| against the long-established `extend` package. Then create a granular | ||
| automation token scoped to the `@extenddb` scope with publish permission only, | ||
| and create the GitHub environment `npm` with deployment branch rule `main`, | ||
| required reviewers, and the token as `NPM_TOKEN`. |
There was a problem hiding this comment.
This paragraph disagrees with the workflow it documents. npm-publish.yml says the environment has no secrets because authentication is npm trusted publishing over OIDC, and the publish job indeed mints an id token and never reads a stored credential. Following this text instead creates a long lived automation token that nothing uses, which is exactly the credential the workflow design went out of its way not to have. I think this predates the switch to trusted publishing and wants rewriting to describe the trusted publisher setup (the workflow header already has the exact field values).
| ref: ${{ needs.gate.outputs.sha }} | ||
| - name: Install Rust toolchain | ||
| uses: dtolnay/rust-toolchain@1.97.0 |
There was a problem hiding this comment.
This is the one action in the release path that is not pinned to a SHA. 1.97.0 here is a mutable tag, and this step runs in the five jobs whose output is the binary actually published to npm, so a retagged action lands directly in release artifacts. That is the highest value target in this workflow, and it sits oddly next to checkout, setup-node, and both artifact actions all being SHA pinned. Worth pinning like the others (same applies to rust-cache@v2 over in launcher.yml, though that one runs in a credential free PR gate so the stakes are much lower).
Uh oh!
There was an error while loading. Please reload this page.
What
npm distribution for the ExtendDB dev server, as the first ecosystem of an
ecosystem-neutral launcher contract (pip etc.. follow the same document).
packaging/npm/: theextenddblauncher package.start()spawns the slimdev-mode binary, health-gates it, and returns an endpoint plus credentials
any AWS SDK accepts. Storage is file-based by default
(
./.extenddb/data.db, tables survive restarts);--memory/{ memory: true }opts into ephemeral; combining a persistence path withmemory mode is refused loudly. Binaries resolve from
@extenddb/<os>-<arch>platform packages viaoptionalDependencies, withEXTENDDB_BINARYand PATH as fallbacks and an actionable error naming allresolution steps (including the musl/Alpine trap) when none works.
packaging/npm/platform/: templates for the five platform packages(linux x64/arm64, darwin x64/arm64, win32 x64).
.github/workflows/npm-publish.yml: dispatch-only release workflowmirroring
release-image.yml(PR ci: publish the PostgreSQL container image for amd64 and arm64 on a version tag #251) clause for clause: gatednpm-vX.Y.Ztag validation (strict shape, existing tag, ancestor of main,version match at the tagged commit), five native runners that each smoke
test and run the launcher suite against their own binary, and a single
credentialed job in a reviewer-gated
npmenvironment publishing with--provenanceunder thecandidatedist-tag only. Promotion tolatestis a manual dist-tag move. The platform list is derived from
optionalDependencieswith loud drift guards. LICENSE and NOTICE arestaged into every tarball.
.github/workflows/launcher.yml: PR gate running the launcher suiteagainst a freshly built dev binary; triggers on
packaging/**andcrates/**so server changes that break the launcher contract fail at PRtime.
packaging/README.md: the launcher contract, release flow, one-timeregistry setup, and known limits (hard-kill orphan, musl, shared-file).
Why
Non-Rust consumers currently need Docker or a manual build to run ExtendDB
locally.
npm install extenddbgives Node projects a zero-configDynamoDB-compatible dev server for local dev and CI, with persistent tables
by default and an in-memory mode for ephemeral test runs. The launcher
contract is deliberately ecosystem-neutral so PyPI and Maven wrappers
implement the same document rather than imitating npm code.
Closes # (no linked issue; scoped from the embedded-distribution plan)
Testing done
EXTENDDB_BINARY=... node packaging/npm/test/launcher.test.js. Persistenceis proven with real data-plane calls (plain-node SigV4, no SDK dev
dependency): an item written by server lifetime one is read back by
lifetime two on the same file, and a restarted memory-mode server does not
know the previous server's table. Negative controls discriminate: pointing
run two at a different path fails the survival assertion; sabotaging memory
mode into a shared file fails the loss assertion.
verbatim,
npm packed the launcher and a platform package, installed bothtarballs into a scratch project, and the launcher resolved the platform
package and served with
EXTENDDB_BINARYunset andextenddbabsent fromPATH. Control: removing the platform package fails with the actionable
resolution error.
path traversal, embedded newline, wrong namespace): all rejected; version
match verified. Platform-list drift guard proven with a synthetic
linux-ppc64entry.and supply-chain / workflow security). All blocking and should-fix findings
are closed in this branch; action SHA pins verified against the GitHub API;
--tag candidatekeepinglatestunset on first publish verified againstnpm docs and [BUG] "npm publish" tags pre-versions as "latest" npm/cli#7553.
npm pack --dry-runon both package shapes confirms LICENSE + NOTICE shipin every tarball.
.rsfiles in the diff);cargo fmt --checkexit 0 on the branch; the Rust suites are unchanged from main.
Checklist
cargo test --workspace) — no Rust changes; suites identical to maincargo fmt --check)cargo clippy -- -W clippy::pedantic) — no Rust changesStoragetrait, auth model, on-diskformat, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).
ADR / RFC: none yet — this adds a new public surface (the npm launcher CLI and
its ecosystem-neutral contract, documented in
packaging/README.md). It doesnot change the server's wire protocol,
Storagetrait, auth model, on-diskformat, or the
extenddbbinary's CLI. If maintainers want the launchercontract captured as an ADR before merge, I'll extract
packaging/README.md'scontract section into one.
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.