Add loadSubset State Tracking and On-Demand Sync Mode

[samwillis]

Overview

This PR adds comprehensive loading status tracking to collections and live queries, enabling UIs to display loading indicators when more data is being fetched. It also introduces a new syncMode configuration option to control when collections load data - either eagerly on initial sync or on-demand as queries request it.

Problem

1. No Loading Indicators: When live queries pushed predicates down to source collections via syncMore, there was no indication that data was being loaded. This made it impossible to show proper loading states in the UI.

2. Always-Eager Loading: Collections always loaded all data immediately during initial sync, even when using predicate pushdown. There was no way to configure collections to only load data as it was requested by queries.

Solution

This PR implements a multi-layered loading state tracking system with configurable sync modes:

Loading State Tracking

1. Collection Subscriptions: Track their own loading state when requesting snapshots, with a public Subscription interface for external consumers
2. Collections: Track pending load promises and expose an isLoadingSubset property
3. Live Queries: Automatically reflect the loading state of their source collection subscriptions

Key Isolation Property: Each live query maintains its own loading state based on its own subscriptions. When a live query triggers loading via predicate pushdown, only that live query's isLoadingSubset becomes true. Other live queries that share the same source collection but don't need that specific snapshot remain unaffected.

Sync Modes

Collections can now be configured with two sync modes:

• eager (default): Loads all data immediately during initial sync. loadSubset calls are bypassed.
• on-demand: Only loads data as it's requested via loadSubset calls. Requires a loadSubset handler.

Changes

1. Renamed syncMore/loadMore to loadSubset

Files:packages/db/src/collection/sync.ts, packages/db/src/types.ts, packages/db/src/collection/subscription.ts

• Renamed syncMore → loadSubset (collection method)
• Renamed onLoadMore → loadSubset (sync config property)
• Renamed OnLoadMoreOptions → LoadSubsetOptions (type)
• Renamed syncOnLoadMoreFn → syncLoadSubsetFn (internal)
• Renamed pendingLoadMorePromises → pendingLoadSubsetPromises (internal)

Rationale: "loadSubset" better reflects that this operation loads a filtered/limited subset of data, not just "more" data.

2. Added syncMode Configuration

Files:packages/db/src/types.ts, packages/db/src/collection/sync.ts

New syncMode property on collection config:

typeSyncMode='eager'|'on-demand'interfaceCollectionConfig{syncMode?: SyncMode// defaults to 'eager'// ... other properties}

Behavior:

• eager (default): loadSubset is bypassed and returns undefined. All data is loaded during initial sync.
• on-demand: loadSubset calls proceed normally. Data is only loaded as requested.

Validation:

• If syncMode: 'on-demand' is set but no loadSubset handler is provided, throws CollectionConfigurationError with a helpful message
• Validation occurs when the sync function is first called

3. Standardized loadSubset Return Type

Files:packages/db/src/collection/sync.ts, packages/db/src/types.ts

• Changed to consistently return Promise<void> | undefined
• Returns undefined when syncMode is 'eager' or no sync implementation is configured
• Wraps synchronous loadSubset results in Promise.resolve()
• Updated SyncConfigRes['loadSubset'] type signature

4. CollectionSubscription Status Tracking & Events

File:packages/db/src/collection/subscription.ts

Added comprehensive status tracking:

• Status Property: status: 'ready' | 'loadingSubset' (readonly getter with private mutable field)
• Concurrent Tracking: pendingLoadSubsetPromises: Set<Promise<void>>
• Events:
  • status:change - Emitted when status transitions
  • status:ready - Emitted when entering ready state
  • status:loadingSubset - Emitted when entering loading state
  • unsubscribed - New event emitted when subscription is destroyed
• Error Handling: Status returns to ready even on promise rejection
• Cleanup: unsubscribe() emits unsubscribed event before clearing listeners

5. Generic isLoadingSubset for All Collections

Files:packages/db/src/collection/sync.ts, packages/db/src/collection/index.ts, packages/db/src/collection/events.ts

All collections now track their loading state:

• Property: isLoadingSubset (boolean getter, not a method)
• Tracking: pendingLoadSubsetPromises: Set<Promise<void>> in CollectionSyncManager
• Events: loadingSubset:change event when state transitions
• API: trackLoadPromise(promise: Promise<void>) method for internal coordination
• Access: Made _sync public on Collection for internal use

6. Live Query Integration

Files:packages/db/src/query/live/collection-subscriber.ts, packages/db/src/query/live/collection-config-builder.ts

Live queries reflect loading state from their subscriptions:

• CollectionSubscriber subscribes to subscription status:change events
• Creates deferred promises when subscriptions enter loadingSubset state
• Passes promises to result collection via liveQueryCollection.trackLoadPromise()
• Result collection's isLoadingSubset reflects subscription loading states
• Proper cleanup on unsubscribe (resolves pending promises)

Loading State Isolation: Each live query's subscriptions are independent. Query A triggering loadSubset doesn't affect Query B's isLoadingSubset status.

7. Subscription Interface for External Consumers

File:packages/db/src/types.ts

New public Subscription interface:

exportinterfaceSubscriptionextendsEventEmitter<SubscriptionEvents>{readonlystatus: SubscriptionStatus}exporttypeLoadSubsetOptions={where?: BasicExpression<boolean>orderBy?: OrderBylimit?: numbersubscription?: Subscription// Optional, for sync implementation use}

Purpose: Allows sync implementations to:

• Track which subscription triggered a loadSubset call
• Subscribe to subscription lifecycle events (e.g., unsubscribed)
• Implement advanced caching/ref-counting based on subscription lifecycle

8. Reusable Event Emitter

File:packages/db/src/event-emitter.ts (new)

Extracted event emitter logic into a reusable base class:

• Type-Safe: Generic EventEmitter<TEvents> with full type safety
• Methods: on, once, off, waitFor
• Protected: emitInner (for subclass use), clearListeners
• Error Handling: Re-throws listener errors via queueMicrotask
• Usage:
  • CollectionEventsManager extends it and wraps emitInner with public emit
  • CollectionSubscription extends it and uses emitInner internally

9. Fixed Local-Only Collection Types

File:packages/db/src/local-only.ts

Fixed mutation function typing issues:

• Wrapper functions now accept broader UtilsRecord type parameters
• Properly handles contravariance in function parameter types
• Removed unnecessary type casts in return statement

10. Comprehensive Test Coverage

Files:packages/db/tests/collection-subscription.test.ts, packages/db/tests/collection.test.ts, packages/db/tests/query/live-query-collection.test.ts

Added extensive test suites:

• CollectionSubscription: Status tracking, event emission, concurrent promises, error handling, cleanup
• Collection.isLoadingSubset: Property tracking, event emission, concurrent loads, error handling
• Live Query Integration: Result collection reflects subscription states, isolation between queries
• All tests using loadSubset: Updated to use syncMode: 'on-demand'

11. Enhanced setWindow with Loading Awareness

File:packages/db/src/query/live/collection-config-builder.ts

The setWindow utility function now returns a value that indicates whether subset loading was triggered, allowing callers to wait for data loading to complete:

Return Type: true | Promise<void>

• Returns true when isLoadingSubset is false after calling the window function - no loading was triggered
• Returns Promise<void> when isLoadingSubset is true - loading was triggered and the promise resolves when loading completes

Implementation Details:

• After calling windowFn() and maybeRunGraphFn(), checks this.liveQueryCollection?.isLoadingSubset
• If loading is active, subscribes to the loadingSubset:change event
• Returns a promise that resolves when isLoadingSubset becomes false
• Automatically unsubscribes from the event once loading completes

Usage Pattern:

constresult=liveQuery.utils.setWindow({offset: 10,limit: 20})// Type guard patternif(result!==true){// Loading was triggered, wait for it to completeawaitresultconsole.log('Data loaded and window updated')}else{// No loading needed, window updated immediatelyconsole.log('Window updated synchronously')}// Or simply always await (works with both return types)constresult=liveQuery.utils.setWindow({offset: 10,limit: 20})if(result!==true){awaitresult}

Test Coverage:

• Test that setWindow returns true when no loading is triggered
• Integration test that validates the full async flow with fake timers:
  • Verifies Promise<void> is returned when loading is triggered
  • Confirms isLoadingSubset becomes true during loading
  • Validates promise doesn't resolve until loading completes
  • Checks isLoadingSubset returns to false after resolution
  • Verifies correct data is loaded in the new window

API

Sync Mode Configuration

// Eager mode (default) - loads all data immediatelyconsteagerCollection=createCollection({getKey: (item)=>item.id,syncMode: 'eager',// optional, this is the defaultsync: {sync: ({ begin, write, commit, markReady })=>{// Load all data herebegin()allData.forEach(item=>write({type: 'insert',value: item}))commit()markReady()}}})// On-demand mode - only loads data as requestedconstonDemandCollection=createCollection({getKey: (item)=>item.id,syncMode: 'on-demand',sync: {sync: ({ markReady })=>{markReady()// Don't load data initiallyreturn{// Required for on-demand modeloadSubset: async(options)=>{const{ where, limit, orderBy, subscription }=options// Load only the requested subsetconstdata=awaitfetchDataSubset(where,limit,orderBy)// ... apply data to collection}}}}})

Collection Subscription

constsubscription=collection.subscribeChanges(callback,options)// Status propertyconsole.log(subscription.status)// 'ready' | 'loadingSubset'// Event listenerssubscription.on('status:change',(event)=>{console.log(`Status: ${event.previousStatus} → ${event.status}`)})subscription.on('status:loadingSubset',(event)=>{console.log('Loading more data...')})subscription.on('status:ready',(event)=>{console.log('Data loaded')})subscription.on('unsubscribed',(event)=>{console.log('Subscription destroyed')})// Cleanupsubscription.unsubscribe()

Collection

// All collections have isLoadingSubset propertyconsole.log(collection.isLoadingSubset)// boolean// Listen for loading state changescollection.on('loadingSubset:change',(event)=>{console.log(`Loading: ${event.isLoadingSubset}`)})

Live Query

constliveQuery=createLiveQueryCollection({query: (q)=>q.from({users: userCollection}).where(({ users })=>users.active)})// Result collection automatically tracks loading from subscriptionsconsole.log(liveQuery.isLoadingSubset)// booleanliveQuery.on('loadingSubset:change',(event)=>{if(event.isLoadingSubset){showLoadingSpinner()}else{hideLoadingSpinner()}})

Breaking Changes

Renamed Methods and Types

• syncMore → loadSubset (collection method)
• onLoadMore → loadSubset (sync config property)
• OnLoadMoreOptions → LoadSubsetOptions (type)

Migration:

// Beforeconstresult=awaitcollection.syncMore({where: expr})
sync: {sync: ()=>({onLoadMore: (options)=>{/* ... */}})}// Afterconstresult=awaitcollection._sync.loadSubset({where: expr})
sync: {sync: ()=>({loadSubset: (options)=>{/* ... */}})}

Note: loadSubset is now called via collection._sync.loadSubset() as it's an internal coordination API, not for general public use.

Migration Guide

For Collection Users

No changes required if you're just using collections - isLoadingSubset is automatically available.

For Sync Implementers

1. Rename onLoadMore to loadSubset:

// Beforereturn{onLoadMore: (options)=>{/* ... */}}// After return{loadSubset: (options)=>{/* ... */}}

2. (Optional) Use the new subscription parameter for advanced use cases:

return{loadSubset: (options)=>{// Track which subscription triggered thisconstsub=options.subscription// Can subscribe to unsubscribe event for cleanupsub?.on('unsubscribed',()=>{// Clean up resources for this subscription})}}

3. (Optional) Configure syncMode for on-demand loading:

createCollection({syncMode: 'on-demand',// Only load data as requestedsync: {sync: ({ markReady })=>{markReady()return{loadSubset: async(options)=>{/* load subset */}}}}})

[changeset-bot]

🦋 Changeset detected

Latest commit: f31a67e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@tanstack/db	Patch
@tanstack/react-db	Patch
@tanstack/angular-db	Patch
@tanstack/electric-db-collection	Patch
@tanstack/query-db-collection	Patch
@tanstack/rxdb-db-collection	Patch
@tanstack/solid-db	Patch
@tanstack/svelte-db	Patch
@tanstack/trailbase-db-collection	Patch
@tanstack/vue-db	Patch
todos	Patch
@tanstack/db-example-react-todo	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[pkg-pr-new]

More templates

• todos
• tanstack-start-db-projects-example
• @tanstack/db-example-react-todo
• @tanstack/db-example-solid-todo

@tanstack/angular-db

npm i https://pkg.pr.new/@tanstack/angular-db@669

@tanstack/db

npm i https://pkg.pr.new/@tanstack/db@669

@tanstack/db-ivm

npm i https://pkg.pr.new/@tanstack/db-ivm@669

@tanstack/electric-db-collection

npm i https://pkg.pr.new/@tanstack/electric-db-collection@669

@tanstack/query-db-collection

npm i https://pkg.pr.new/@tanstack/query-db-collection@669

@tanstack/react-db

npm i https://pkg.pr.new/@tanstack/react-db@669

@tanstack/rxdb-db-collection

npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@669

@tanstack/solid-db

npm i https://pkg.pr.new/@tanstack/solid-db@669

@tanstack/svelte-db

npm i https://pkg.pr.new/@tanstack/svelte-db@669

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@669

@tanstack/vue-db

npm i https://pkg.pr.new/@tanstack/vue-db@669

commit: 1c54b1b

[github-actions]

Size Change: 0 B

Total Size: 83.6 kB

ℹ️ View Unchanged

Filename	Size
./packages/db/dist/esm/collection/change-events.js	963 B
./packages/db/dist/esm/collection/changes.js	1.01 kB
./packages/db/dist/esm/collection/events.js	413 B
./packages/db/dist/esm/collection/index.js	3.23 kB
./packages/db/dist/esm/collection/indexes.js	1.16 kB
./packages/db/dist/esm/collection/lifecycle.js	1.8 kB
./packages/db/dist/esm/collection/mutations.js	2.52 kB
./packages/db/dist/esm/collection/state.js	3.79 kB
./packages/db/dist/esm/collection/subscription.js	2.2 kB
./packages/db/dist/esm/collection/sync.js	2.2 kB
./packages/db/dist/esm/deferred.js	230 B
./packages/db/dist/esm/errors.js	3.57 kB
./packages/db/dist/esm/event-emitter.js	798 B
./packages/db/dist/esm/index.js	1.65 kB
./packages/db/dist/esm/indexes/auto-index.js	794 B
./packages/db/dist/esm/indexes/base-index.js	835 B
./packages/db/dist/esm/indexes/btree-index.js	2 kB
./packages/db/dist/esm/indexes/lazy-index.js	1.21 kB
./packages/db/dist/esm/indexes/reverse-index.js	577 B
./packages/db/dist/esm/local-only.js	967 B
./packages/db/dist/esm/local-storage.js	2.33 kB
./packages/db/dist/esm/optimistic-action.js	294 B
./packages/db/dist/esm/proxy.js	3.86 kB
./packages/db/dist/esm/query/builder/functions.js	615 B
./packages/db/dist/esm/query/builder/index.js	4.04 kB
./packages/db/dist/esm/query/builder/ref-proxy.js	938 B
./packages/db/dist/esm/query/compiler/evaluators.js	1.55 kB
./packages/db/dist/esm/query/compiler/expressions.js	760 B
./packages/db/dist/esm/query/compiler/group-by.js	2.04 kB
./packages/db/dist/esm/query/compiler/index.js	2.21 kB
./packages/db/dist/esm/query/compiler/joins.js	2.65 kB
./packages/db/dist/esm/query/compiler/order-by.js	1.43 kB
./packages/db/dist/esm/query/compiler/select.js	1.28 kB
./packages/db/dist/esm/query/ir.js	785 B
./packages/db/dist/esm/query/live-query-collection.js	404 B
./packages/db/dist/esm/query/live/collection-config-builder.js	5.49 kB
./packages/db/dist/esm/query/live/collection-registry.js	233 B
./packages/db/dist/esm/query/live/collection-subscriber.js	2.11 kB
./packages/db/dist/esm/query/optimizer.js	3.26 kB
./packages/db/dist/esm/scheduler.js	1.29 kB
./packages/db/dist/esm/SortedMap.js	1.24 kB
./packages/db/dist/esm/transactions.js	3.05 kB
./packages/db/dist/esm/utils.js	1.01 kB
./packages/db/dist/esm/utils/browser-polyfills.js	365 B
./packages/db/dist/esm/utils/btree.js	6.01 kB
./packages/db/dist/esm/utils/comparison.js	754 B
./packages/db/dist/esm/utils/index-optimization.js	1.73 kB

_{compressed-size-action::db-package-size}

[github-actions]

Size Change: 0 B

Total Size: 2.36 kB

ℹ️ View Unchanged

Filename	Size
./packages/react-db/dist/esm/index.js	168 B
./packages/react-db/dist/esm/useLiveInfiniteQuery.js	885 B
./packages/react-db/dist/esm/useLiveQuery.js	1.31 kB

_{compressed-size-action::react-db-package-size}

[KyleAMathews]

🚀

[KyleAMathews]

I thought the plan here was isLoadingMore is true only if a live query calls updatePredicate? Otherwise a live query would be see there isLoadingMore set to true when e.g. a joined collection needs to grab an object, etc. So any UI set to this would be flickering on and off seemingly randomly w/o any way to control it by the dev.

[samwillis]

This bit of the code all dates to when I was working over the weekend. Will update...

I do wander if we need to expose that it is grabbing data from the server though, joins lazy appearing without the dev being able to put a placeholder/spinner does feel messy.

Maybe we need to separate flags?

Also not that this version from the weekend made the isLoadingMore prop a standard on all collections. For base collections it's true when their own loadSubset is pending, for live query collections it's when a subscription trigger a loadSubset that returns a promise.

[samwillis]

Maybe we need an isLoadingMore and a isLoadingSubset, the latter whenever any subset triggered by the collection is loading, and isLoadingMore for when it's triggered by an offset/limit change (which do not isn't complete yet, and not in this PR)

[samwillis]

To fully implement this as only triggered when the offset/limit is changed we will have to rebase this PR on #663 or wait till thats merged. I suggest we leave the implementation as is now (setting isLoading on any subscription triggering a loadSubset) and fix in a future PR.

db/packages/db/src/query/live/collection-subscriber.ts

Lines 77 to 84 in 9ad1169

	conststatusUnsubscribe=subscription.on(`status:change`,(event)=>{
	// TODO: For now we are setting this loading state whenever the subscription
	// status changes to 'loadingMore'. But we have discussed it only happening
	// when the the live query has it's offset/limit changed, and that triggers the
	// subscription to request a snapshot. This will require more work to implement,
	// and builds on https://github.com/TanStack/db/pull/663 which this PR
	// does not yet depend on.
	if(event.status===`loadingMore`){

The question is if collection. isLoadingMore should be collection.isLoadingSubset and that isLoadingMore is a live query only thing?

[kevin-dp]

Nice work, left a few minor comments.

[kevin-dp]

I would rather keep this method such that we can keep the _sync property private

[samwillis]

I disagree, syncMore is an internal method that shouldering be presented on the front he public api. The managers having the underscore marks them as "internal implementation but exposed for debugging and internal communication". I would prefer to not present syncMore on the prompts when people are interacting with a collection.

[kevin-dp]

Should we also check that the promise is not yet resolved? Because if it is already resolved then there is no need to do the below as that may lead to some flickering in the UI if one would e.g. show a spinner when more is loading.

[samwillis]

Unfortunately there is no api to do this, thats why loadSubset can return true to indicate that it has the data synchronously.

[github-actions]

🎉 This PR has been released!

• @tanstack/db@0.4.10
• @tanstack/react-db@0.1.32

Thank you for your contribution!