feat!: make projections and dev runtime recovery-safe

[patrickleet]

Why

An older aggregate snapshot should not overwrite newer read-model state or resurrect a deleted row. Changing a service's event subscriptions should not reset its acknowledged work. A client-only development rebuild should not leave a healthy, retained API rejecting every command.

This PR brings those recovery guarantees into the framework, including an explicit way to rebuild existing snapshot projections without replaying domain side effects.

What application authors gain

• Order-safe snapshot projections: opt into aggregate-source fencing instead of writing application-specific stale-event checks. Rows and tombstones retain the authoritative source version in memory, SQLite and PostgreSQL.
• Safe, explicit rebuilds: reconstruct a projection from retained canonical occurrences while preserving inboxes, broker checkpoints and command ledgers. Incomplete/conflicting history and concurrent target changes fail closed.
• Durable subscription changes: NATS updates existing consumer filters without replacing the consumer, resetting acknowledgement progress or overwriting broker tuning. Historical catch-up remains explicit.
• Working commands after UI-only reloads: distributed dev admits retained process instances into the active generation. Preparing, replaced and retired instances remain fenced; schema compatibility alone does not authorize a process.
• One typed application inventory: Service::application(name, surface) derives modules from command namespaces and validates ownership against the full Surface. Larger applications can export their actual contract: complete manifests allow 4 MiB while individual opaque JSON values remain limited to 1 MiB.
• Preserved event identity across boundaries: event handlers can explicitly inherit causation into a downstream aggregate; NATS preserves declared payload content types and prevents metadata from shadowing reserved transport headers.

Typed application assembly

let surface = distributed::SurfaceSpec::from_surface("catalog", &full_surface)?;
let application = service.application("catalog", surface)?;
let manifest = application.manifest();

Use the runtime's complete Surface here; role-selected browser clients remain authorization views of it. Missing Service commands and unowned Surface commands are rejected instead of silently producing an incomplete artifact.

Event-driven policies can continue a causal command chain explicitly before recording downstream events:

ctx.inherit_causation(&mut downstream)?;
downstream.record(observation)?;

Missing incoming causation is an error; this helper does not invent command identity for external events.

Authoring example

distributed::projection! {
    pub const TODOS: ProjectionDescriptor<EventualOnly> = {
        name: "project_todos",
        version: 2,
        epoch: "todos-source-snapshots-v1",
        model: Todos,
        source: aggregate_snapshot,
        on {
            events: [TodoCreatedDomainEvent, TodoCompletedDomainEvent],
            mutation: SaveTodo,
            input: { todo: body },
        },
    };
}

The fence compares (aggregate_sequence, publication_ordinal) within the owning aggregate stream. A stale event confirms the current row revision without inventing a row update. A conflicting equal version or another aggregate attempting to take over the same key is rejected.

This mode is for complete replacement snapshots with stable keys—not counters, partial patches, joins, or arbitrary delta reordering. Browser optimism still uses the shared mutation program and committed record revisions for confirmation.

Existing read models

use distributed::projection::rebuild::SnapshotProjectionRebuild;

// Explicit offline maintenance: stop producers, drain outboxes,
// stop consumers, and back up the read model first.
let rebuild = SnapshotProjectionRebuild::begin(&repository, &projector).await?;
let events = nats_bus.retained_domain_events().await?;
let plan = rebuild.from_complete_history(&events)?;
plan.apply(&repository).await?;

The rebuild API is bus-neutral; the NATS helper reads a stable, gap-free retained stream without consuming or acknowledging it. The caller must still establish historical coverage: a stream starting at sequence one does not prove that all aggregate history was published there.

Rebuilds are bounded offline maintenance for one active local unit-partition binding: up to 10,000 records and 100,000 occurrences / 64 MiB of canonical history. They do not migrate schemas or turn independent projections into one transaction.

Breaking changes and upgrade

• Apply framework migration 0005_projection_source_snapshots before using SQL-backed source fences. ProjectionRecordMetadata gains source_snapshot.
• Existing unfenced rows need an explicit valid-history rebuild before opting into snapshot semantics; changing the projection epoch is not a migration or fallback.
• Upgrade the CLI and runtime together, then restart distributed dev to establish matching process-instance membership.

The original four functional commits are preserved separately. This PR also consolidates all five commits from #217, cherry-picked with source provenance, including its resolved Content-Type review fix. A formatting-only commit reproduces cargo fmt --all with the repository configuration. Follow-up test/CI commits cover migration five and the full eventual-command lifecycle.

Validation

• Fresh Rust library suite (graphql,sqlite,postgres,nats): 988 passed, 0 failed, 1 explicitly ignored live-NATS test. Environment-gated live cases in that first run are not credited as live evidence.
• Fresh JavaScript suite: 344 passed, 0 failed.
• cargo fmt --all --check and git diff origin/main --check: pass.
• Fresh CLI unit suite: 247 passed; CLI lifecycle integration: 11 passed.
• Repository integration: 13 SQLite and 10 live PostgreSQL tests passed, including upgrade assertions for migration five.
• Source snapshot/rebuild matrix: 8 passed, including live PostgreSQL reordering, restart, rebuild and rollback. The PostgreSQL CI feature combination also passed locally.
• Live NATS archive/filter tests: 2 passed, including the previously ignored broker test. CI now explicitly runs these and the live PostgreSQL fencing/rebuild cases instead of silently skipping environment-gated checks.
• E2E behavioral suite: 14 passed; projection-proof helper regressions: 5 passed.
• Live browser CI on d8d6d70a: 20 browser tests passed, followed by all five lifecycle transitions (two UI-only, application, framework, incompatible contract). Every transition proves a valid command receipt, the exact authoritative GraphQL row, and fresh-page visibility. Rebuilt-process transitions needed two authoritative reads, confirming that immediate SSR had raced eventual projection. The test retains its final browser assertion and timeout; it does not use optimistic state as persistence proof.
• The first local live-adapter attempt failed on host connectivity; the successful live runs above supersede it. Initial CI's stale migration assertions and immediate-SSR test race are corrected.
• After incorporating feat: support production-sized typed applications #217: Rust library suite (graphql,nats) 913 passed, 0 failed, 1 ignored; added missing/unowned Surface rejection assertions also pass. The live NATS custom-content-type round-trip passes.
• The existing browser suite and original lifecycle restoration assertions remain unchanged. The newly added eventual-command probe checks projection convergence before fresh-page rendering; it does not claim read-your-writes across an immediate refresh before projection.
• Consolidated CI on 37c7143e is running. This PR remains draft pending verification; the preceding run was superseded by this push.

No application-specific project details or fixtures are required to use these APIs.

[coderabbitai]

Important

Review skipped

Too many files!

This PR contains 124 files, which is 24 over the limit of 100.

To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch.

Upgrade to a paid plan to raise the limit.

This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: 16f1ee97-84ca-475f-b3f3-f4bd7751ba98

📥 Commits

Reviewing files that changed from the base of the PR and between 8a2e208 and 37c7143.

📒 Files selected for processing (124)

• .github/workflows/integration-e2e-ui.yaml
• .github/workflows/integration-nats.yaml
• .github/workflows/integration-postgres.yaml
• README.md
• distributed_cli/README.md
• distributed_cli/src/cli.rs
• distributed_cli/src/client_compiler/mod.rs
• distributed_cli/src/client_compiler/projection_delta/wire.rs
• distributed_cli/src/client_compiler/render/commands.rs
• distributed_cli/src/client_compiler/tests.rs
• distributed_cli/src/contracts/classification.rs
• distributed_cli/src/contracts/closeout.rs
• distributed_cli/src/contracts/mod.rs
• distributed_cli/src/contracts/program.rs
• distributed_cli/src/contracts/snapshots.rs
• distributed_cli/src/contracts/tests.rs
• distributed_cli/src/contracts/transaction.rs
• distributed_cli/src/generate/service_crate.rs
• distributed_cli/src/js_framework.rs
• distributed_cli/src/lib.rs
• distributed_cli/src/lifecycle/build.rs
• distributed_cli/src/lifecycle/dev.rs
• distributed_cli/src/wasm_pures.rs
• distributed_cli/tests/cli_client.rs
• distributed_cli/tests/cli_lifecycle.rs
• distributed_cli/tests/cli_scaffold.rs
• distributed_macros/src/digest.rs
• distributed_macros/src/domain_event.rs
• distributed_macros/src/lib.rs
• distributed_macros/src/portable_command.rs
• distributed_macros/tests/application.rs
• examples/graphiql.rs
• js/src/replica/command-runtime/create.ts
• js/tests/replica-command-runtime.test.mjs
• migrations/inventory.json
• migrations/postgres/0005_projection_source_snapshots.sql
• migrations/sqlite/0005_projection_source_snapshots.sql
• src/application/capability.rs
• src/application/command.rs
• src/application/manifest.rs
• src/application/module.rs
• src/application/plan.rs
• src/application/registration.rs
• src/application/runtime_host.rs
• src/bus/message.rs
• src/bus/nats.rs
• src/bus/nats_bus.rs
• src/bus/sql_bus_common.rs
• src/command_ledger/tests.rs
• src/graphql/client_manifest/export.rs
• src/graphql/client_manifest/identity.rs
• src/graphql/client_manifest/mod.rs
• src/graphql/client_manifest/tests.rs
• src/graphql/command_contract/tests.rs
• src/graphql/engine/request.rs
• src/graphql/http.rs
• src/graphql/projection_delta/types.rs
• src/graphql/protocol/mod.rs
• src/graphql/protocol/tests.rs
• src/graphql/protocol/types.rs
• src/graphql/surface/tests.rs
• src/graphql/surface/types.rs
• src/in_memory_repo/projection_protocol/direct_projection.rs
• src/in_memory_repo/projection_protocol/mod.rs
• src/in_memory_repo/projection_protocol/rebuild.rs
• src/in_memory_repo/projection_protocol/state_impl.rs
• src/in_memory_repo/projection_protocol/store_impl.rs
• src/lib.rs
• src/microsvc/context.rs
• src/microsvc/lifecycle.rs
• src/microsvc/mod.rs
• src/microsvc/service/defaults.rs
• src/microsvc/service/mod.rs
• src/microsvc/service/runtime.rs
• src/microsvc/service/tests.rs
• src/projection/executor.rs
• src/projection/mod.rs
• src/projection/placement.rs
• src/projection/plan.rs
• src/projection/program.rs
• src/projection/rebuild.rs
• src/projection/source_snapshot_tests.rs
• src/projection_protocol.rs
• src/projection_protocol/source_snapshot.rs
• src/projection_protocol/store/commit.rs
• src/projection_protocol/store/identity.rs
• src/projection_protocol/store/query.rs
• src/projection_protocol/store/replay.rs
• src/projection_protocol/store/tests.rs
• src/projection_protocol/store/trait.rs
• src/projection_protocol/workspace.rs
• src/queued_repo/repository.rs
• src/sqlx_repo/projection_protocol/mod.rs
• src/sqlx_repo/projection_protocol/reads.rs
• src/sqlx_repo/projection_protocol/rebuild.rs
• src/sqlx_repo/projection_protocol/store_impl.rs
• src/sqlx_repo/projection_protocol/writes.rs
• src/sqlx_repo/repo/backend.rs
• src/table/mod.rs
• src/table/mutation.rs
• tests/application_composition.rs
• tests/application_plans.rs
• tests/causal_public_invoke/main.rs
• tests/distributed_read_model/checkout_saga_service/service.rs
• tests/e2e-ui/README.md
• tests/e2e-ui/package.json
• tests/e2e-ui/scripts/lifecycle-command-proof.mjs
• tests/e2e-ui/scripts/lifecycle-command-proof.test.mjs
• tests/e2e-ui/scripts/lifecycle-reload.mjs
• tests/fixtures/source_snapshot_delete.graphql
• tests/fixtures/source_snapshot_save.graphql
• tests/graphql_harden/authz.rs
• tests/graphql_harden/dos.rs
• tests/graphql_harden/residual.rs
• tests/graphql_query_protocol/main.rs
• tests/graphql_query_protocol_postgres/main.rs
• tests/graphql_sqlite/main.rs
• tests/graphql_subscriptions_sqlite/main.rs
• tests/metrics_exposition/main.rs
• tests/microsvc/transport_http.rs
• tests/nats_transport/main.rs
• tests/postgres_repository/main.rs
• tests/sqlite_repository/main.rs
• tests/typed_commands/main.rs

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}