[Bug]: Projection bootstrap silently caps catch-up at 1,000 un-applied events per projector (readFromSequence default limit); the rest are never projected

[krutftw]

What happened

While validating a bulk import of project.created events against state.sqlite, I checked how the server catches up projections on startup and found that the catch-up is silently capped at 1,000 events per projector: anything beyond that is never projected, and the gap becomes permanent as soon as the engine appends its next event.

Diagnosis

apps/server/src/orchestration/Layers/ProjectionPipeline.ts, bootstrapProjector:

Stream.runForEach(
 eventStore.readFromSequence(
 Option.isSome(stateRow) ? stateRow.value.lastAppliedSequence : 0,
 ),
 (event) => runProjectorForEvent(projector, event),
)

readFromSequence(sequenceExclusive, limit = DEFAULT_READ_FROM_SEQUENCE_LIMIT) defaults to 1 000 (apps/server/src/persistence/Services/OrchestrationEventStore.ts, DEFAULT_READ_FROM_SEQUENCE_LIMIT = 1_000; the layer pages in READ_PAGE_SIZE = 500 chunks up to that limit). The store already exposes an unbounded read (readAll = readFromSequence(0, Number.MAX_SAFE_INTEGER)), but bootstrap does not pass a limit, so:

1. If max(sequence) - last_applied_sequence > 1000 at startup, only the first 1 000 events are projected and projection_state is advanced to last_applied + 1000.
2. The next live command appends event N+1 and runProjectorForEvent upserts last_applied_sequence = N+1, so events last_applied+1001 … N are never replayed by any future bootstrap — e.g. projects missing from projection_projects (and therefore from requireActiveProjectWorkspaceRootAbsent), threads/messages missing from their projections.

How such a backlog can arise today: events appended while the server is not running — the offline mode of the t3 project CLI (which also falls back to offline writes while a live server is slow, see #7504), any external writer, or a crash/kill between append and projection repeated over time. It is also the kind of invariant that bites later (imports, replication, tests). The fix is one line: pass Number.MAX_SAFE_INTEGER (the stream still pages in 500-row chunks, so memory is bounded) — or use readAll-style semantics for bootstrap.

Steps to reproduce

1. Stop the server. Append 1 001+ events to orchestration_events (e.g. project.created rows with the engine's row shape; or run the offline project CLI 1 001 times) without touching projection_state.
2. Start the server → projection_state.last_applied_sequence = previous + 1000; the last event(s) are not in projection_projects.
3. Create anything in the UI (one new event) → last_applied_sequence jumps to max; the skipped events are never projected.

Version

0.0.34-nightly.20260819.1132 (commit 36f4314); same code on main today.

Environment

Windows 11 Pro 10.0.26100, desktop app. Verified by reading the shipped bundle and the source on main.

Evidence

# bundle (0.0.34-nightly.20260819.1132)
const DEFAULT_READ_FROM_SEQUENCE_LIMIT = 1e3;
const READ_PAGE_SIZE = 500;
const readFromSequence = (sequenceExclusive, limit = DEFAULT_READ_FROM_SEQUENCE_LIMIT) => { ... }
readAll: () => readFromSequence(0, Number.MAX_SAFE_INTEGER)
const bootstrapProjector = (projector) => projectionStateRepository.getByProjector({ projector: projector.name })
 .pipe(flatMap$1((stateRow) => runForEach(eventStore.readFromSequence(isSome(stateRow) ? stateRow.value.lastAppliedSequence : 0), (event) => runProjectorForEvent(projector, event))));

Related issues

#7504 (project CLI offline fallback) is one way to produce un-applied events; no existing issue covers the bootstrap cap.

Fix applied or workaround

None needed locally (my import was 65 events). Workaround for anyone importing more: keep each batch under 1 000 and start+quit the app between batches. PR with the one-line fix attached.

Filed by

Claude (Fable 5) via Claude Code on the reporter's machine, at the reporter's request.