Skip to content

fix electric collection progressive mode - #852

Merged
samwillis merged 7 commits into
mainfrom
samwillis/fix-electric-progressive
Nov 21, 2025
Merged

fix electric collection progressive mode#852
samwillis merged 7 commits into
mainfrom
samwillis/fix-electric-progressive

Conversation

@samwillis

@samwillissamwillis commented Nov 19, 2025

Copy link
Copy Markdown
Collaborator

Depends on electric-sql/electric#3463, must have deps bumped to new electric client version when that is released.done

Fix progressive mode to use fetchSnapshot and atomic swap

Progressive mode was broken because requestSnapshot() injected snapshots into the stream in causally correct position, which didn't work properly with the full mode stream. This release fixes progressive mode by:

Core Changes:

  • Use fetchSnapshot() during initial sync to fetch and apply snapshots immediately in sync transactions
  • Buffer all stream messages during initial sync (renamed flag to isBufferingInitialSync)
  • Perform atomic swap on first up-to-date: truncate snapshot data → apply buffered messages → mark ready
  • Track txids/snapshots only after atomic swap (enables correct optimistic transaction confirmation)

Test Infrastructure:

  • Added ELECTRIC_TEST_HOOKS symbol for test control (hidden from public API)
  • Added progressiveTestControl.releaseInitialSync() to E2E test config for explicit transition control
  • Created comprehensive progressive mode E2E test suite (8 tests):
    • Explicit snapshot phase and atomic swap validation
    • Txid tracking behavior (Electric-only)
    • Multiple concurrent snapshots with deduplication
    • Incremental updates after swap
    • Predicate handling and resilience tests

Bug Fixes:

  • Fixed type errors in test files
  • All 166 unit tests + 95 E2E tests passing

@changeset-bot

changeset-botBot commented Nov 19, 2025

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 2846a6e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 11 packages
NameType
@tanstack/electric-db-collectionPatch
@tanstack/db-collection-e2ePatch
@tanstack/dbPatch
@tanstack/angular-dbPatch
@tanstack/powersync-db-collectionPatch
@tanstack/react-dbPatch
@tanstack/rxdb-db-collectionPatch
@tanstack/solid-dbPatch
@tanstack/svelte-dbPatch
@tanstack/trailbase-db-collectionPatch
@tanstack/vue-dbPatch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@samwillis
samwillis marked this pull request as draft November 19, 2025 12:18
@samwillis
samwillisforce-pushed the samwillis/fix-electric-progressive branch from 85a5bd7 to b702326CompareNovember 19, 2025 12:33
@pkg-pr-new

pkg-pr-newBot commented Nov 19, 2025

Copy link
Copy Markdown
More templates

@tanstack/angular-db

npm i https://pkg.pr.new/@tanstack/angular-db@852

@tanstack/db

npm i https://pkg.pr.new/@tanstack/db@852

@tanstack/db-ivm

npm i https://pkg.pr.new/@tanstack/db-ivm@852

@tanstack/electric-db-collection

npm i https://pkg.pr.new/@tanstack/electric-db-collection@852

@tanstack/offline-transactions

npm i https://pkg.pr.new/@tanstack/offline-transactions@852

@tanstack/powersync-db-collection

npm i https://pkg.pr.new/@tanstack/powersync-db-collection@852

@tanstack/query-db-collection

npm i https://pkg.pr.new/@tanstack/query-db-collection@852

@tanstack/react-db

npm i https://pkg.pr.new/@tanstack/react-db@852

@tanstack/rxdb-db-collection

npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@852

@tanstack/solid-db

npm i https://pkg.pr.new/@tanstack/solid-db@852

@tanstack/svelte-db

npm i https://pkg.pr.new/@tanstack/svelte-db@852

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@852

@tanstack/vue-db

npm i https://pkg.pr.new/@tanstack/vue-db@852

commit: 2846a6e

@github-actions

github-actionsBot commented Nov 19, 2025

Copy link
Copy Markdown
Contributor

Size Change: 0 B

Total Size: 86.2 kB

ℹ️ View Unchanged
FilenameSize
./packages/db/dist/esm/collection/change-events.js1.38 kB
./packages/db/dist/esm/collection/changes.js977 B
./packages/db/dist/esm/collection/events.js388 B
./packages/db/dist/esm/collection/index.js3.24 kB
./packages/db/dist/esm/collection/indexes.js1.1 kB
./packages/db/dist/esm/collection/lifecycle.js1.67 kB
./packages/db/dist/esm/collection/mutations.js2.26 kB
./packages/db/dist/esm/collection/state.js3.43 kB
./packages/db/dist/esm/collection/subscription.js2.42 kB
./packages/db/dist/esm/collection/sync.js2.29 kB
./packages/db/dist/esm/deferred.js207 B
./packages/db/dist/esm/errors.js4.19 kB
./packages/db/dist/esm/event-emitter.js748 B
./packages/db/dist/esm/index.js2.64 kB
./packages/db/dist/esm/indexes/auto-index.js742 B
./packages/db/dist/esm/indexes/base-index.js766 B
./packages/db/dist/esm/indexes/btree-index.js1.87 kB
./packages/db/dist/esm/indexes/lazy-index.js1.1 kB
./packages/db/dist/esm/indexes/reverse-index.js513 B
./packages/db/dist/esm/local-only.js837 B
./packages/db/dist/esm/local-storage.js2.1 kB
./packages/db/dist/esm/optimistic-action.js359 B
./packages/db/dist/esm/paced-mutations.js496 B
./packages/db/dist/esm/proxy.js3.22 kB
./packages/db/dist/esm/query/builder/functions.js733 B
./packages/db/dist/esm/query/builder/index.js3.96 kB
./packages/db/dist/esm/query/builder/ref-proxy.js917 B
./packages/db/dist/esm/query/compiler/evaluators.js1.35 kB
./packages/db/dist/esm/query/compiler/expressions.js430 B
./packages/db/dist/esm/query/compiler/group-by.js1.8 kB
./packages/db/dist/esm/query/compiler/index.js1.96 kB
./packages/db/dist/esm/query/compiler/joins.js2 kB
./packages/db/dist/esm/query/compiler/order-by.js1.25 kB
./packages/db/dist/esm/query/compiler/select.js1.07 kB
./packages/db/dist/esm/query/expression-helpers.js1.43 kB
./packages/db/dist/esm/query/ir.js673 B
./packages/db/dist/esm/query/live-query-collection.js360 B
./packages/db/dist/esm/query/live/collection-config-builder.js5.26 kB
./packages/db/dist/esm/query/live/collection-registry.js264 B
./packages/db/dist/esm/query/live/collection-subscriber.js1.74 kB
./packages/db/dist/esm/query/live/internal.js130 B
./packages/db/dist/esm/query/optimizer.js2.56 kB
./packages/db/dist/esm/query/predicate-utils.js2.88 kB
./packages/db/dist/esm/query/subset-dedupe.js921 B
./packages/db/dist/esm/scheduler.js1.21 kB
./packages/db/dist/esm/SortedMap.js1.18 kB
./packages/db/dist/esm/strategies/debounceStrategy.js247 B
./packages/db/dist/esm/strategies/queueStrategy.js428 B
./packages/db/dist/esm/strategies/throttleStrategy.js246 B
./packages/db/dist/esm/transactions.js2.9 kB
./packages/db/dist/esm/utils.js881 B
./packages/db/dist/esm/utils/browser-polyfills.js304 B
./packages/db/dist/esm/utils/btree.js5.61 kB
./packages/db/dist/esm/utils/comparison.js852 B
./packages/db/dist/esm/utils/index-optimization.js1.51 kB
./packages/db/dist/esm/utils/type-guards.js157 B

compressed-size-action::db-package-size

@github-actions

github-actionsBot commented Nov 19, 2025

Copy link
Copy Markdown
Contributor

Size Change: 0 B

Total Size: 3.34 kB

ℹ️ View Unchanged
FilenameSize
./packages/react-db/dist/esm/index.js225 B
./packages/react-db/dist/esm/useLiveInfiniteQuery.js1.17 kB
./packages/react-db/dist/esm/useLiveQuery.js1.11 kB
./packages/react-db/dist/esm/useLiveSuspenseQuery.js431 B
./packages/react-db/dist/esm/usePacedMutations.js401 B

compressed-size-action::react-db-package-size

@samwillis
samwillisforce-pushed the samwillis/fix-electric-progressive branch from 3941a16 to 132b57fCompareNovember 19, 2025 14:17
@samwillis
samwillis marked this pull request as ready for review November 19, 2025 14:17

@KyleAMathewsKyleAMathews left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! All that work you did on truncate is coming in handy! Very nice semantics to be able to easily do this clean swap.

@kevin-dp

Copy link
Copy Markdown
Contributor

Mostly note to self during review:

Perform atomic swap on first up-to-date: truncate snapshot data → apply buffered messages → mark ready

--> Need to check that we also cancel in-flight fetchSnapshot requests and/or ignore fetchSnapshot requests that resolve after the swap.

kevin-dp
kevin-dp previously requested changes Nov 20, 2025

@kevin-dpkevin-dp left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great work, i think we're almost there but i believe there is a bug with fetchSnapshot requests that resolve after the up-to-date message. For the rest i only have minor comments/nits.

Comment threadpackages/electric-db-collection/src/electric.ts Outdated
Comment threadpackages/electric-db-collection/src/electric.ts Outdated
Comment threadpackages/electric-db-collection/src/electric.ts Outdated
Comment threadpackages/electric-db-collection/src/electric.ts Outdated
@samwillis
samwillisforce-pushed the samwillis/fix-electric-progressive branch from a24fe7f to 2846a6eCompareNovember 21, 2025 11:32
@samwillis

Copy link
Copy Markdown
CollaboratorAuthor

@kevin-dp I've addressed your review. We should still come back to this file and tidy it up more.

@samwillis
samwillisforce-pushed the samwillis/fix-electric-progressive branch from 8fc3efd to 2846a6eCompareNovember 21, 2025 19:14
@samwillis
samwillis dismissed kevin-dp’s stale reviewNovember 21, 2025 19:35

Discussed with Kyle, merging as is before the weekend. We can make further changes later.

@samwillis
samwillis merged commit acb3e4f into mainNov 21, 2025
13 checks passed
@samwillis
samwillis deleted the samwillis/fix-electric-progressive branch November 21, 2025 19:35
@github-actionsgithub-actionsBot mentioned this pull request Nov 21, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@samwillis@kevin-dp@KyleAMathews