Skip to content

feat(storage): enforce operational schema compatibility epochs - #2363

Closed
Astro-Han wants to merge 20 commits into
mainfrom
feat/2357-operational-schema-compatibility
Closed

feat(storage): enforce operational schema compatibility epochs#2363
Astro-Han wants to merge 20 commits into
mainfrom
feat/2357-operational-schema-compatibility

Conversation

@Astro-Han

@Astro-HanAstro-Han commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Refs #2357.

  • Establish the first released operational schema epoch as the compatibility baseline, so same-epoch older Maka versions preserve and continue using known data while ignoring newer scopes.
  • Reject higher reader epochs before migration, preserve the database on incompatibility, and publish all scope migrations, registry entries, and the minimum reader epoch in one transaction.
  • Require an authentic root migration owner at the database-open seam; ordinary path-based opens cannot perform a cross-epoch migration.
  • Keep a shared root-authority compatibility lock for the full lifetime of Headless storage; require an exclusive lock only for a breaking epoch migration.
  • Freeze an independent epoch-one reader/writer probe, exercise it bidirectionally through current production Session APIs, and enforce compatible/breaking declarations plus immutable historical probes in CI.
  • Preserve compatibility metadata in filtered Session Bundle exports.
  • Leave registered scopes from a newer same-epoch build untouched, including their DDL, while still migrating older scopes.
  • Fail schema governance closed when the comparison base is unavailable or invalid.

Verification

  • actionlint .github/workflows/ci.yml
  • node scripts/check-operational-schema-compatibility.mjs --base main
  • node --test scripts/check-operational-schema-compatibility.test.mjs
  • invalid all-zero governance base rejected
  • npm run format:check
  • npm run lint
  • npm run typecheck
  • npm run build
  • npm run test:dist --workspace @maka/storage: full suite passed
  • npm run test:dist --workspace @maka/headless: 1,508 passed, 0 failed
  • focused Runtime Host kernel, Desktop storage startup, Headless boundary, migration authority, and root-authority suites passed

Migration and review focus

  • This PR defines epoch 1; releases before this PR are intentionally outside the epoch contract.
  • Compatible schema additions stay within the current epoch. A breaking schema change must add a manifest declaration, advance the global reader epoch by exactly one, and add a frozen probe.
  • Cross-epoch migration is reachable only through an authenticated Interactive or Headless root owner. Direct store opens fail rather than bypass process exclusion.
  • A process with an older epoch blocks a breaking Headless or Runtime Host migration through the lifetime root lock. Same-epoch readers and writers remain concurrent.
  • Four independent adversarial reviews ran with opencode-go/deepseek-v4-flash:max, covering migration/data safety, cross-process lifecycle, governance/tests, and first-principles/Occam design.
  • The member review raised four blocking findings; all were accepted and fixed in separate reversible commits: migration authority at the DB seam, bidirectional epoch probing, rename-safe historical probe governance, and fail-closed CI history comparison.
  • The earlier proposal to remove the migration lock was rejected because the lock becomes the released epoch-1 process exclusion protocol for the first epoch-2 migration. The alleged execution-store close leak was disproved: the facade sessionStore.close() closes the complete persistence cluster.

@Astro-Han
Astro-Han marked this pull request as ready for review August 7, 2026 00:33

@zhiiwzhiiw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SQLite compatibility issues are noted inline.

Comment threadpackages/headless/src/headless-storage.ts Outdated
Comment threadpackages/storage/src/__tests__/operational-epoch-probe.test.ts Outdated
Comment threadscripts/check-operational-schema-compatibility.mjs
Comment thread.github/workflows/ci.yml Outdated
@Astro-Han
Astro-Han marked this pull request as draft August 7, 2026 03:28

@zhiiwzhiiw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining SQLite compatibility issues are noted inline.

Comment threadpackages/storage/src/operational-state-store.ts

const current = acquireOperationalStateDatabase(root);
try {
for (const [table, key, value] of [

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new bidirectional production-API coverage exercises only session_metadata. The remaining scopes in this loop are still verified as epoch-1 raw INSERT -> current raw COUNT; no current-writer output from runtime, core execution, workflow, usage, artifact, or automation is read or updated by the frozen epoch-1 consumer. A supposedly compatible change in any of those scopes could still break an older reader while this test passes. Please add bidirectional production-shaped coverage for every registered scope, or explicitly narrow the epoch contract.

return Number(match[1]);
}

function countBreakingChanges(source) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This count is not an append-only history check. A later PR can rewrite or remove an existing compatibility declaration while keeping the global epoch valid; for example, with two scopes requiring epoch 2, one historical breaking declaration can be reclassified while the other still keeps the manifest reader epoch at 2. Please structurally compare base and current manifests and require every historical scope/change list to be an exact prefix of the current one, rather than comparing regex counts.

@Astro-Han
Astro-Han requested a review from zhiiwAugust 7, 2026 05:12
@likun666661

Copy link
Copy Markdown
Member

Design-level blocker: before freezing epoch 1, we need to define the compatibility contract that this mechanism is intended to enforce.

This PR currently makes one global minimum_reader_epoch carry several materially different promises:

  1. an older binary can read a database written by a newer binary;
  2. an older binary can safely write/update that database;
  3. old and new binaries can use it concurrently;
  4. rollback to the older binary remains supported.

These implications do not follow from one another. In particular, reader compatibility does not establish writer compatibility: an old read-modify-write path can discard fields it does not understand, overwrite newer state, or violate a newer cross-table invariant. The lifetime compatibility lock prevents a cross-epoch migration while an old process is alive, but deliberately permits same-epoch mixed-version writers, so it does not address that semantic risk.

The frozen-probe/append-only design also appears to turn every historical same-epoch behavior into a permanent support obligation. Mature upgrade systems normally define a finite, asymmetric support matrix and a controlled upgrade sequence; they do not treat all historical readers and writers as universally compatible.

I think this PR should first state an explicit policy for each operational scope:

  • Is the data canonical, rebuildable, or ephemeral?
  • Which version pairs support old-reader/new-writer, old-writer/new-reader, and concurrent mixed-version access?
  • Is mixed-version access only a bounded rolling-upgrade state such as N/N-1?
  • After an incompatible contract phase, is rollback performed by an old binary opening the new database, or by restoring a snapshot?
  • How long must historical probes remain executable?

Then the mechanism can encode that policy. A likely shape is to separate minimum reader and minimum writer epochs, distinguish physical schema version from feature/behavior activation, and use expand -> activate -> contract. Old binaries may participate during the bounded expand window; after contract, rollback is snapshot restore rather than silently reopening the database with old code. The probe runner should execute every epoch that the declared support window includes, with production-shaped coverage for every supported matrix cell.

#2361 already addresses the urgent data-safety property by removing destructive reset behavior and failing closed on a newer schema. That gives us room to define the narrower product contract before this PR freezes epoch 1. As written, the implementation is rigorous, but it is rigorously enforcing a compatibility promise that has not yet been specified.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@Astro-Han@likun666661@zhiiw