Reduce sync work: faster snapshot projection + benchmarks by hahn-kev-bot · Pull Request #86 · sillsdev/harmony

hahn-kev-bot · 2026-07-21T09:56:36Z

Overview

Reduces the work done during sync (AddRangeFromSync → SnapshotWorker.UpdateSnapshots → CrdtRepository.AddSnapshots) and adds a benchmark suite to measure it. On the CreateWords workload at 10k changes this branch takes sync from ~1.96 s / 976 MB allocated down to ~0.99 s / 538 MB — roughly 2× faster and ~45% less memory.

What changed

Snapshot pre-load (SnapshotWorker / DataModel)

UpdateSnapshots now bulk-loads the relevant current snapshots (with their Commit) into a cache keyed by entity id, and SnapshotWorker reads full snapshots straight from that cache instead of issuing a FindSnapshot DB round-trip per cache hit.

Fast raw-SQL projection (FastProjection, new)

Snapshot rows are still inserted through EF unchanged, but the projected tables are now populated with hand-written raw SQL INSERT ... ON CONFLICT(pk) DO UPDATE (one upsert per entity row) instead of going through EF's change tracker (FindAsync / SetValues / graph tracking).
Everything the SQL needs — table/column names (correctly delimited), primary key, the SnapshotId shadow FK, value converters — is derived from the EF model, so there is no per-entity code.
It dedups to the latest snapshot per entity, runs deletes before upserts (children-first) then upserts (parents-first) for FK/unique-constraint safety, and reuses the caller's transaction.
FastProjection is an injectable singleton; its per-type SQL metadata cache lives on an internal ConcurrentDictionary on CrdtConfig, shared across repositories/contexts. This replaces the previous EF change-tracker projection path in CrdtRepository.AddSnapshots, which is removed.

Benchmarks (new SIL.Harmony.Benchmarks project)

BenchmarkDotNet suite with a DataModelSyncBenchmarks (7 sync workloads) and an AddSnapshotsBenchmarks that isolates the persist step, [MemoryDiagnoser] enabled. Run with dotnet run -c Release --project src/SIL.Harmony.Benchmarks.

Testing

Full test suite passes: 241 passing. The only failures are the 6 DataModelPerformanceBenchmarks timing-threshold tests, which also fail on main (environmental, not caused by this change).

Notes for reviewers

The projection is SQLite-specific in a couple of spots that matter for FK correctness (GUIDs stored as uppercase TEXT; ON CONFLICT after a SELECT needs the SQLite WHERE true disambiguator in the code history — the current per-query path uses VALUES). If other providers are ever targeted, FastProjection would need revisiting.
Known limitation: an intra-batch self-reference (e.g. Word.AntonymId pointing at another new Word in the same batch) is not ordered; it's a nullable SET NULL FK and not exercised by current workloads.

Previously AddSnapshots projected each snapshot by calling FindAsync per entity, which issued one database query per snapshot (and, on an initial sync of new data, every query returned null after a round-trip). Pre-load the projected rows that already exist for the batch with a single tracked query per object type. ProjectSnapshot then resolves existing entities from the change tracker and skips the lookup entirely for entities that have no projected row yet, collapsing N queries down to roughly one per distinct object type. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019zEmz7jPRPF6Lv8h6YAWBW

Adds a BenchmarkDotNet suite that measures CrdtRepository.AddSnapshots on its own, across 7 workloads mirroring DataModelSyncBenchmarks. Expensive DB seeding runs once in a template DB; each iteration forks the DB and recomputes the snapshot batch so no EF-tracked state leaks across iterations. - SnapshotWorker.ComputeSnapshotsToPersist: returns the exact snapshot list UpdateSnapshots would persist, without writing it. - DataModelTestBase: internal CreateRepository() and CrdtConfig accessors. - BenchmarkWorkloadBuilders: shared commit builders extracted from DataModelSyncBenchmarks (+ BuildUpdateExisting). - Program.cs: run both suites via BenchmarkSwitcher (handles --filter/args). - Remove leftover Console.WriteLine debug lines from AddSnapshots. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Two experimental fast AddSnapshots implementations that keep the EF snapshot insert unchanged but populate projected tables with raw INSERT ... ON CONFLICT upserts instead of going through EF's change tracker: - FAST: one upsert command per entity row - FAST_JSON: one command per entity type, rows passed as a single JSON array expanded with SQLite json_each/json_extract FastProjection derives table/column names, primary key, the SnapshotId shadow FK, and value converters from the EF model (no per-entity code). It dedups to the latest snapshot per entity, runs deletes before upserts (children-first) then upserts (parents-first) for FK/unique-constraint safety, and reuses the caller's transaction. CrdtRepository.AddSnapshots now selects via #if FAST_JSON / #elif FAST / #else. Program.cs adds a third FAST_JSON benchmark job and DataModelSyncBenchmarks enables [MemoryDiagnoser]. Benchmarks (CreateWords, 1000): both fast paths ~35% faster and ~32% fewer allocations than baseline; per-query vs JSON-batch shows no measurable difference against in-memory SQLite. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The JSON-batch projection benchmarked identically to the per-query path against in-memory SQLite (same time and allocations), so remove it and keep only the per-query raw-SQL upsert path. FastProjection loses the useJsonBatch parameter and all json_each/json_extract code; CrdtRepository.AddSnapshots collapses to #if FAST / #else; the benchmark drops the FAST_JSON job. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Remove the EF change-tracker slow path and the #if FAST conditional so AddSnapshots always uses FastProjection. Deletes the now-dead slow-path helpers (ProjectSnapshot, GetEntityEntry, LoadExistingEntityIds, LoadExistingEntities). The benchmark collapses to a single job since FAST vs DEFAULT are now identical. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

FastProjection is now an injected singleton (registered in AddCrdtDataCore and resolved into CrdtRepository via ActivatorUtilities) instead of a static class. Its per-type projected-table SQL metadata cache moves from a static field onto an internal ConcurrentDictionary on CrdtConfig, so it's shared across repositories/contexts and tied to config lifetime. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

coderabbitai · 2026-07-21T09:56:44Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
auto_review:
drafts: true

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch reduce-sync-work

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

# Conflicts: # src/SIL.Harmony/Config/HarmonyConfig.cs # src/SIL.Harmony/SnapshotWorker.cs

Notify DI interceptors and HarmonyConfig.OnProjectedEntitiesChanged after projected SQL with the latest upsert or delete per entity.

Keep the slnx migration from main and include SIL.Harmony.Benchmarks in the solution.

Main now uses Microsoft.Testing.Platform, so solution-wide dotnet test was launching the Benchmarks exe and failing on unknown MTP flags.

hahn-kev-bot · 2026-07-21T09:56:36Z

Overview

Reduces the work done during sync (AddRangeFromSync → SnapshotWorker.UpdateSnapshots → CrdtRepository.AddSnapshots) and adds a benchmark suite to measure it. On the CreateWords workload at 10k changes this branch takes sync from ~1.96 s / 976 MB allocated down to ~0.99 s / 538 MB — roughly 2× faster and ~45% less memory.

What changed

Snapshot pre-load (SnapshotWorker / DataModel)

UpdateSnapshots now bulk-loads the relevant current snapshots (with their Commit) into a cache keyed by entity id, and SnapshotWorker reads full snapshots straight from that cache instead of issuing a FindSnapshot DB round-trip per cache hit.

Fast raw-SQL projection (FastProjection, new)

Snapshot rows are still inserted through EF unchanged, but the projected tables are now populated with hand-written raw SQL INSERT ... ON CONFLICT(pk) DO UPDATE (one upsert per entity row) instead of going through EF's change tracker (FindAsync / SetValues / graph tracking).
Everything the SQL needs — table/column names (correctly delimited), primary key, the SnapshotId shadow FK, value converters — is derived from the EF model, so there is no per-entity code.
It dedups to the latest snapshot per entity, runs deletes before upserts (children-first) then upserts (parents-first) for FK/unique-constraint safety, and reuses the caller's transaction.
FastProjection is an injectable singleton; its per-type SQL metadata cache lives on an internal ConcurrentDictionary on CrdtConfig, shared across repositories/contexts. This replaces the previous EF change-tracker projection path in CrdtRepository.AddSnapshots, which is removed.

Benchmarks (new SIL.Harmony.Benchmarks project)

BenchmarkDotNet suite with a DataModelSyncBenchmarks (7 sync workloads) and an AddSnapshotsBenchmarks that isolates the persist step, [MemoryDiagnoser] enabled. Run with dotnet run -c Release --project src/SIL.Harmony.Benchmarks.

Testing

Full test suite passes: 241 passing. The only failures are the 6 DataModelPerformanceBenchmarks timing-threshold tests, which also fail on main (environmental, not caused by this change).

Notes for reviewers

The projection is SQLite-specific in a couple of spots that matter for FK correctness (GUIDs stored as uppercase TEXT; ON CONFLICT after a SELECT needs the SQLite WHERE true disambiguator in the code history — the current per-query path uses VALUES). If other providers are ever targeted, FastProjection would need revisiting.
Known limitation: an intra-batch self-reference (e.g. Word.AntonymId pointing at another new Word in the same batch) is not ordered; it's a nullable SET NULL FK and not exercised by current workloads.

Previously AddSnapshots projected each snapshot by calling FindAsync per entity, which issued one database query per snapshot (and, on an initial sync of new data, every query returned null after a round-trip). Pre-load the projected rows that already exist for the batch with a single tracked query per object type. ProjectSnapshot then resolves existing entities from the change tracker and skips the lookup entirely for entities that have no projected row yet, collapsing N queries down to roughly one per distinct object type. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019zEmz7jPRPF6Lv8h6YAWBW

Adds a BenchmarkDotNet suite that measures CrdtRepository.AddSnapshots on its own, across 7 workloads mirroring DataModelSyncBenchmarks. Expensive DB seeding runs once in a template DB; each iteration forks the DB and recomputes the snapshot batch so no EF-tracked state leaks across iterations. - SnapshotWorker.ComputeSnapshotsToPersist: returns the exact snapshot list UpdateSnapshots would persist, without writing it. - DataModelTestBase: internal CreateRepository() and CrdtConfig accessors. - BenchmarkWorkloadBuilders: shared commit builders extracted from DataModelSyncBenchmarks (+ BuildUpdateExisting). - Program.cs: run both suites via BenchmarkSwitcher (handles --filter/args). - Remove leftover Console.WriteLine debug lines from AddSnapshots. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Two experimental fast AddSnapshots implementations that keep the EF snapshot insert unchanged but populate projected tables with raw INSERT ... ON CONFLICT upserts instead of going through EF's change tracker: - FAST: one upsert command per entity row - FAST_JSON: one command per entity type, rows passed as a single JSON array expanded with SQLite json_each/json_extract FastProjection derives table/column names, primary key, the SnapshotId shadow FK, and value converters from the EF model (no per-entity code). It dedups to the latest snapshot per entity, runs deletes before upserts (children-first) then upserts (parents-first) for FK/unique-constraint safety, and reuses the caller's transaction. CrdtRepository.AddSnapshots now selects via #if FAST_JSON / #elif FAST / #else. Program.cs adds a third FAST_JSON benchmark job and DataModelSyncBenchmarks enables [MemoryDiagnoser]. Benchmarks (CreateWords, 1000): both fast paths ~35% faster and ~32% fewer allocations than baseline; per-query vs JSON-batch shows no measurable difference against in-memory SQLite. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The JSON-batch projection benchmarked identically to the per-query path against in-memory SQLite (same time and allocations), so remove it and keep only the per-query raw-SQL upsert path. FastProjection loses the useJsonBatch parameter and all json_each/json_extract code; CrdtRepository.AddSnapshots collapses to #if FAST / #else; the benchmark drops the FAST_JSON job. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Remove the EF change-tracker slow path and the #if FAST conditional so AddSnapshots always uses FastProjection. Deletes the now-dead slow-path helpers (ProjectSnapshot, GetEntityEntry, LoadExistingEntityIds, LoadExistingEntities). The benchmark collapses to a single job since FAST vs DEFAULT are now identical. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

FastProjection is now an injected singleton (registered in AddCrdtDataCore and resolved into CrdtRepository via ActivatorUtilities) instead of a static class. Its per-type projected-table SQL metadata cache moves from a static field onto an internal ConcurrentDictionary on CrdtConfig, so it's shared across repositories/contexts and tied to config lifetime. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

coderabbitai · 2026-07-21T09:56:44Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
auto_review:
drafts: true

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch reduce-sync-work

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

# Conflicts: # src/SIL.Harmony/Config/HarmonyConfig.cs # src/SIL.Harmony/SnapshotWorker.cs

Notify DI interceptors and HarmonyConfig.OnProjectedEntitiesChanged after projected SQL with the latest upsert or delete per entity.

Keep the slnx migration from main and include SIL.Harmony.Benchmarks in the solution.

Main now uses Microsoft.Testing.Platform, so solution-wide dotnet test was launching the Benchmarks exe and failing on unknown MTP flags.

hahn-kev-bot · 2026-07-21T09:56:36Z

Overview

Reduces the work done during sync (AddRangeFromSync → SnapshotWorker.UpdateSnapshots → CrdtRepository.AddSnapshots) and adds a benchmark suite to measure it. On the CreateWords workload at 10k changes this branch takes sync from ~1.96 s / 976 MB allocated down to ~0.99 s / 538 MB — roughly 2× faster and ~45% less memory.

What changed

Snapshot pre-load (SnapshotWorker / DataModel)

UpdateSnapshots now bulk-loads the relevant current snapshots (with their Commit) into a cache keyed by entity id, and SnapshotWorker reads full snapshots straight from that cache instead of issuing a FindSnapshot DB round-trip per cache hit.

Fast raw-SQL projection (FastProjection, new)

Snapshot rows are still inserted through EF unchanged, but the projected tables are now populated with hand-written raw SQL INSERT ... ON CONFLICT(pk) DO UPDATE (one upsert per entity row) instead of going through EF's change tracker (FindAsync / SetValues / graph tracking).
Everything the SQL needs — table/column names (correctly delimited), primary key, the SnapshotId shadow FK, value converters — is derived from the EF model, so there is no per-entity code.
It dedups to the latest snapshot per entity, runs deletes before upserts (children-first) then upserts (parents-first) for FK/unique-constraint safety, and reuses the caller's transaction.
FastProjection is an injectable singleton; its per-type SQL metadata cache lives on an internal ConcurrentDictionary on CrdtConfig, shared across repositories/contexts. This replaces the previous EF change-tracker projection path in CrdtRepository.AddSnapshots, which is removed.

Benchmarks (new SIL.Harmony.Benchmarks project)

BenchmarkDotNet suite with a DataModelSyncBenchmarks (7 sync workloads) and an AddSnapshotsBenchmarks that isolates the persist step, [MemoryDiagnoser] enabled. Run with dotnet run -c Release --project src/SIL.Harmony.Benchmarks.

Testing

Full test suite passes: 241 passing. The only failures are the 6 DataModelPerformanceBenchmarks timing-threshold tests, which also fail on main (environmental, not caused by this change).

Notes for reviewers

The projection is SQLite-specific in a couple of spots that matter for FK correctness (GUIDs stored as uppercase TEXT; ON CONFLICT after a SELECT needs the SQLite WHERE true disambiguator in the code history — the current per-query path uses VALUES). If other providers are ever targeted, FastProjection would need revisiting.
Known limitation: an intra-batch self-reference (e.g. Word.AntonymId pointing at another new Word in the same batch) is not ordered; it's a nullable SET NULL FK and not exercised by current workloads.

Previously AddSnapshots projected each snapshot by calling FindAsync per entity, which issued one database query per snapshot (and, on an initial sync of new data, every query returned null after a round-trip). Pre-load the projected rows that already exist for the batch with a single tracked query per object type. ProjectSnapshot then resolves existing entities from the change tracker and skips the lookup entirely for entities that have no projected row yet, collapsing N queries down to roughly one per distinct object type. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019zEmz7jPRPF6Lv8h6YAWBW

Adds a BenchmarkDotNet suite that measures CrdtRepository.AddSnapshots on its own, across 7 workloads mirroring DataModelSyncBenchmarks. Expensive DB seeding runs once in a template DB; each iteration forks the DB and recomputes the snapshot batch so no EF-tracked state leaks across iterations. - SnapshotWorker.ComputeSnapshotsToPersist: returns the exact snapshot list UpdateSnapshots would persist, without writing it. - DataModelTestBase: internal CreateRepository() and CrdtConfig accessors. - BenchmarkWorkloadBuilders: shared commit builders extracted from DataModelSyncBenchmarks (+ BuildUpdateExisting). - Program.cs: run both suites via BenchmarkSwitcher (handles --filter/args). - Remove leftover Console.WriteLine debug lines from AddSnapshots. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Two experimental fast AddSnapshots implementations that keep the EF snapshot insert unchanged but populate projected tables with raw INSERT ... ON CONFLICT upserts instead of going through EF's change tracker: - FAST: one upsert command per entity row - FAST_JSON: one command per entity type, rows passed as a single JSON array expanded with SQLite json_each/json_extract FastProjection derives table/column names, primary key, the SnapshotId shadow FK, and value converters from the EF model (no per-entity code). It dedups to the latest snapshot per entity, runs deletes before upserts (children-first) then upserts (parents-first) for FK/unique-constraint safety, and reuses the caller's transaction. CrdtRepository.AddSnapshots now selects via #if FAST_JSON / #elif FAST / #else. Program.cs adds a third FAST_JSON benchmark job and DataModelSyncBenchmarks enables [MemoryDiagnoser]. Benchmarks (CreateWords, 1000): both fast paths ~35% faster and ~32% fewer allocations than baseline; per-query vs JSON-batch shows no measurable difference against in-memory SQLite. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The JSON-batch projection benchmarked identically to the per-query path against in-memory SQLite (same time and allocations), so remove it and keep only the per-query raw-SQL upsert path. FastProjection loses the useJsonBatch parameter and all json_each/json_extract code; CrdtRepository.AddSnapshots collapses to #if FAST / #else; the benchmark drops the FAST_JSON job. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Remove the EF change-tracker slow path and the #if FAST conditional so AddSnapshots always uses FastProjection. Deletes the now-dead slow-path helpers (ProjectSnapshot, GetEntityEntry, LoadExistingEntityIds, LoadExistingEntities). The benchmark collapses to a single job since FAST vs DEFAULT are now identical. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

FastProjection is now an injected singleton (registered in AddCrdtDataCore and resolved into CrdtRepository via ActivatorUtilities) instead of a static class. Its per-type projected-table SQL metadata cache moves from a static field onto an internal ConcurrentDictionary on CrdtConfig, so it's shared across repositories/contexts and tied to config lifetime. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

coderabbitai · 2026-07-21T09:56:44Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
auto_review:
drafts: true

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch reduce-sync-work

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

# Conflicts: # src/SIL.Harmony/Config/HarmonyConfig.cs # src/SIL.Harmony/SnapshotWorker.cs

Notify DI interceptors and HarmonyConfig.OnProjectedEntitiesChanged after projected SQL with the latest upsert or delete per entity.

Keep the slnx migration from main and include SIL.Harmony.Benchmarks in the solution.

Main now uses Microsoft.Testing.Platform, so solution-wide dotnet test was launching the Benchmarks exe and failing on unknown MTP flags.

hahn-kev-bot · 2026-07-21T09:56:36Z

Overview

Reduces the work done during sync (AddRangeFromSync → SnapshotWorker.UpdateSnapshots → CrdtRepository.AddSnapshots) and adds a benchmark suite to measure it. On the CreateWords workload at 10k changes this branch takes sync from ~1.96 s / 976 MB allocated down to ~0.99 s / 538 MB — roughly 2× faster and ~45% less memory.

What changed

Snapshot pre-load (SnapshotWorker / DataModel)

UpdateSnapshots now bulk-loads the relevant current snapshots (with their Commit) into a cache keyed by entity id, and SnapshotWorker reads full snapshots straight from that cache instead of issuing a FindSnapshot DB round-trip per cache hit.

Fast raw-SQL projection (FastProjection, new)

Snapshot rows are still inserted through EF unchanged, but the projected tables are now populated with hand-written raw SQL INSERT ... ON CONFLICT(pk) DO UPDATE (one upsert per entity row) instead of going through EF's change tracker (FindAsync / SetValues / graph tracking).
Everything the SQL needs — table/column names (correctly delimited), primary key, the SnapshotId shadow FK, value converters — is derived from the EF model, so there is no per-entity code.
It dedups to the latest snapshot per entity, runs deletes before upserts (children-first) then upserts (parents-first) for FK/unique-constraint safety, and reuses the caller's transaction.
FastProjection is an injectable singleton; its per-type SQL metadata cache lives on an internal ConcurrentDictionary on CrdtConfig, shared across repositories/contexts. This replaces the previous EF change-tracker projection path in CrdtRepository.AddSnapshots, which is removed.

Benchmarks (new SIL.Harmony.Benchmarks project)

BenchmarkDotNet suite with a DataModelSyncBenchmarks (7 sync workloads) and an AddSnapshotsBenchmarks that isolates the persist step, [MemoryDiagnoser] enabled. Run with dotnet run -c Release --project src/SIL.Harmony.Benchmarks.

Testing

Full test suite passes: 241 passing. The only failures are the 6 DataModelPerformanceBenchmarks timing-threshold tests, which also fail on main (environmental, not caused by this change).

Notes for reviewers

The projection is SQLite-specific in a couple of spots that matter for FK correctness (GUIDs stored as uppercase TEXT; ON CONFLICT after a SELECT needs the SQLite WHERE true disambiguator in the code history — the current per-query path uses VALUES). If other providers are ever targeted, FastProjection would need revisiting.
Known limitation: an intra-batch self-reference (e.g. Word.AntonymId pointing at another new Word in the same batch) is not ordered; it's a nullable SET NULL FK and not exercised by current workloads.

Previously AddSnapshots projected each snapshot by calling FindAsync per entity, which issued one database query per snapshot (and, on an initial sync of new data, every query returned null after a round-trip). Pre-load the projected rows that already exist for the batch with a single tracked query per object type. ProjectSnapshot then resolves existing entities from the change tracker and skips the lookup entirely for entities that have no projected row yet, collapsing N queries down to roughly one per distinct object type. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019zEmz7jPRPF6Lv8h6YAWBW

Adds a BenchmarkDotNet suite that measures CrdtRepository.AddSnapshots on its own, across 7 workloads mirroring DataModelSyncBenchmarks. Expensive DB seeding runs once in a template DB; each iteration forks the DB and recomputes the snapshot batch so no EF-tracked state leaks across iterations. - SnapshotWorker.ComputeSnapshotsToPersist: returns the exact snapshot list UpdateSnapshots would persist, without writing it. - DataModelTestBase: internal CreateRepository() and CrdtConfig accessors. - BenchmarkWorkloadBuilders: shared commit builders extracted from DataModelSyncBenchmarks (+ BuildUpdateExisting). - Program.cs: run both suites via BenchmarkSwitcher (handles --filter/args). - Remove leftover Console.WriteLine debug lines from AddSnapshots. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Two experimental fast AddSnapshots implementations that keep the EF snapshot insert unchanged but populate projected tables with raw INSERT ... ON CONFLICT upserts instead of going through EF's change tracker: - FAST: one upsert command per entity row - FAST_JSON: one command per entity type, rows passed as a single JSON array expanded with SQLite json_each/json_extract FastProjection derives table/column names, primary key, the SnapshotId shadow FK, and value converters from the EF model (no per-entity code). It dedups to the latest snapshot per entity, runs deletes before upserts (children-first) then upserts (parents-first) for FK/unique-constraint safety, and reuses the caller's transaction. CrdtRepository.AddSnapshots now selects via #if FAST_JSON / #elif FAST / #else. Program.cs adds a third FAST_JSON benchmark job and DataModelSyncBenchmarks enables [MemoryDiagnoser]. Benchmarks (CreateWords, 1000): both fast paths ~35% faster and ~32% fewer allocations than baseline; per-query vs JSON-batch shows no measurable difference against in-memory SQLite. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The JSON-batch projection benchmarked identically to the per-query path against in-memory SQLite (same time and allocations), so remove it and keep only the per-query raw-SQL upsert path. FastProjection loses the useJsonBatch parameter and all json_each/json_extract code; CrdtRepository.AddSnapshots collapses to #if FAST / #else; the benchmark drops the FAST_JSON job. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Remove the EF change-tracker slow path and the #if FAST conditional so AddSnapshots always uses FastProjection. Deletes the now-dead slow-path helpers (ProjectSnapshot, GetEntityEntry, LoadExistingEntityIds, LoadExistingEntities). The benchmark collapses to a single job since FAST vs DEFAULT are now identical. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

FastProjection is now an injected singleton (registered in AddCrdtDataCore and resolved into CrdtRepository via ActivatorUtilities) instead of a static class. Its per-type projected-table SQL metadata cache moves from a static field onto an internal ConcurrentDictionary on CrdtConfig, so it's shared across repositories/contexts and tied to config lifetime. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

coderabbitai · 2026-07-21T09:56:44Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
auto_review:
drafts: true

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch reduce-sync-work

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

# Conflicts: # src/SIL.Harmony/Config/HarmonyConfig.cs # src/SIL.Harmony/SnapshotWorker.cs

Notify DI interceptors and HarmonyConfig.OnProjectedEntitiesChanged after projected SQL with the latest upsert or delete per entity.

Keep the slnx migration from main and include SIL.Harmony.Benchmarks in the solution.

Main now uses Microsoft.Testing.Platform, so solution-wide dotnet test was launching the Benchmarks exe and failing on unknown MTP flags.

hahn-kev-bot · 2026-07-21T09:56:36Z

Overview

Reduces the work done during sync (AddRangeFromSync → SnapshotWorker.UpdateSnapshots → CrdtRepository.AddSnapshots) and adds a benchmark suite to measure it. On the CreateWords workload at 10k changes this branch takes sync from ~1.96 s / 976 MB allocated down to ~0.99 s / 538 MB — roughly 2× faster and ~45% less memory.

What changed

Snapshot pre-load (SnapshotWorker / DataModel)

UpdateSnapshots now bulk-loads the relevant current snapshots (with their Commit) into a cache keyed by entity id, and SnapshotWorker reads full snapshots straight from that cache instead of issuing a FindSnapshot DB round-trip per cache hit.

Fast raw-SQL projection (FastProjection, new)

Snapshot rows are still inserted through EF unchanged, but the projected tables are now populated with hand-written raw SQL INSERT ... ON CONFLICT(pk) DO UPDATE (one upsert per entity row) instead of going through EF's change tracker (FindAsync / SetValues / graph tracking).
Everything the SQL needs — table/column names (correctly delimited), primary key, the SnapshotId shadow FK, value converters — is derived from the EF model, so there is no per-entity code.
It dedups to the latest snapshot per entity, runs deletes before upserts (children-first) then upserts (parents-first) for FK/unique-constraint safety, and reuses the caller's transaction.
FastProjection is an injectable singleton; its per-type SQL metadata cache lives on an internal ConcurrentDictionary on CrdtConfig, shared across repositories/contexts. This replaces the previous EF change-tracker projection path in CrdtRepository.AddSnapshots, which is removed.

Benchmarks (new SIL.Harmony.Benchmarks project)

BenchmarkDotNet suite with a DataModelSyncBenchmarks (7 sync workloads) and an AddSnapshotsBenchmarks that isolates the persist step, [MemoryDiagnoser] enabled. Run with dotnet run -c Release --project src/SIL.Harmony.Benchmarks.

Testing

Full test suite passes: 241 passing. The only failures are the 6 DataModelPerformanceBenchmarks timing-threshold tests, which also fail on main (environmental, not caused by this change).

Notes for reviewers

The projection is SQLite-specific in a couple of spots that matter for FK correctness (GUIDs stored as uppercase TEXT; ON CONFLICT after a SELECT needs the SQLite WHERE true disambiguator in the code history — the current per-query path uses VALUES). If other providers are ever targeted, FastProjection would need revisiting.
Known limitation: an intra-batch self-reference (e.g. Word.AntonymId pointing at another new Word in the same batch) is not ordered; it's a nullable SET NULL FK and not exercised by current workloads.

Previously AddSnapshots projected each snapshot by calling FindAsync per entity, which issued one database query per snapshot (and, on an initial sync of new data, every query returned null after a round-trip). Pre-load the projected rows that already exist for the batch with a single tracked query per object type. ProjectSnapshot then resolves existing entities from the change tracker and skips the lookup entirely for entities that have no projected row yet, collapsing N queries down to roughly one per distinct object type. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019zEmz7jPRPF6Lv8h6YAWBW

Adds a BenchmarkDotNet suite that measures CrdtRepository.AddSnapshots on its own, across 7 workloads mirroring DataModelSyncBenchmarks. Expensive DB seeding runs once in a template DB; each iteration forks the DB and recomputes the snapshot batch so no EF-tracked state leaks across iterations. - SnapshotWorker.ComputeSnapshotsToPersist: returns the exact snapshot list UpdateSnapshots would persist, without writing it. - DataModelTestBase: internal CreateRepository() and CrdtConfig accessors. - BenchmarkWorkloadBuilders: shared commit builders extracted from DataModelSyncBenchmarks (+ BuildUpdateExisting). - Program.cs: run both suites via BenchmarkSwitcher (handles --filter/args). - Remove leftover Console.WriteLine debug lines from AddSnapshots. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Two experimental fast AddSnapshots implementations that keep the EF snapshot insert unchanged but populate projected tables with raw INSERT ... ON CONFLICT upserts instead of going through EF's change tracker: - FAST: one upsert command per entity row - FAST_JSON: one command per entity type, rows passed as a single JSON array expanded with SQLite json_each/json_extract FastProjection derives table/column names, primary key, the SnapshotId shadow FK, and value converters from the EF model (no per-entity code). It dedups to the latest snapshot per entity, runs deletes before upserts (children-first) then upserts (parents-first) for FK/unique-constraint safety, and reuses the caller's transaction. CrdtRepository.AddSnapshots now selects via #if FAST_JSON / #elif FAST / #else. Program.cs adds a third FAST_JSON benchmark job and DataModelSyncBenchmarks enables [MemoryDiagnoser]. Benchmarks (CreateWords, 1000): both fast paths ~35% faster and ~32% fewer allocations than baseline; per-query vs JSON-batch shows no measurable difference against in-memory SQLite. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The JSON-batch projection benchmarked identically to the per-query path against in-memory SQLite (same time and allocations), so remove it and keep only the per-query raw-SQL upsert path. FastProjection loses the useJsonBatch parameter and all json_each/json_extract code; CrdtRepository.AddSnapshots collapses to #if FAST / #else; the benchmark drops the FAST_JSON job. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Remove the EF change-tracker slow path and the #if FAST conditional so AddSnapshots always uses FastProjection. Deletes the now-dead slow-path helpers (ProjectSnapshot, GetEntityEntry, LoadExistingEntityIds, LoadExistingEntities). The benchmark collapses to a single job since FAST vs DEFAULT are now identical. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

FastProjection is now an injected singleton (registered in AddCrdtDataCore and resolved into CrdtRepository via ActivatorUtilities) instead of a static class. Its per-type projected-table SQL metadata cache moves from a static field onto an internal ConcurrentDictionary on CrdtConfig, so it's shared across repositories/contexts and tied to config lifetime. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

coderabbitai · 2026-07-21T09:56:44Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
auto_review:
drafts: true

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch reduce-sync-work

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

# Conflicts: # src/SIL.Harmony/Config/HarmonyConfig.cs # src/SIL.Harmony/SnapshotWorker.cs

Notify DI interceptors and HarmonyConfig.OnProjectedEntitiesChanged after projected SQL with the latest upsert or delete per entity.

Keep the slnx migration from main and include SIL.Harmony.Benchmarks in the solution.

Main now uses Microsoft.Testing.Platform, so solution-wide dotnet test was launching the Benchmarks exe and failing on unknown MTP flags.

hahn-kev-bot · 2026-07-21T09:56:36Z

Overview

Reduces the work done during sync (AddRangeFromSync → SnapshotWorker.UpdateSnapshots → CrdtRepository.AddSnapshots) and adds a benchmark suite to measure it. On the CreateWords workload at 10k changes this branch takes sync from ~1.96 s / 976 MB allocated down to ~0.99 s / 538 MB — roughly 2× faster and ~45% less memory.

What changed

Snapshot pre-load (SnapshotWorker / DataModel)

UpdateSnapshots now bulk-loads the relevant current snapshots (with their Commit) into a cache keyed by entity id, and SnapshotWorker reads full snapshots straight from that cache instead of issuing a FindSnapshot DB round-trip per cache hit.

Fast raw-SQL projection (FastProjection, new)

Snapshot rows are still inserted through EF unchanged, but the projected tables are now populated with hand-written raw SQL INSERT ... ON CONFLICT(pk) DO UPDATE (one upsert per entity row) instead of going through EF's change tracker (FindAsync / SetValues / graph tracking).
Everything the SQL needs — table/column names (correctly delimited), primary key, the SnapshotId shadow FK, value converters — is derived from the EF model, so there is no per-entity code.
It dedups to the latest snapshot per entity, runs deletes before upserts (children-first) then upserts (parents-first) for FK/unique-constraint safety, and reuses the caller's transaction.
FastProjection is an injectable singleton; its per-type SQL metadata cache lives on an internal ConcurrentDictionary on CrdtConfig, shared across repositories/contexts. This replaces the previous EF change-tracker projection path in CrdtRepository.AddSnapshots, which is removed.

Benchmarks (new SIL.Harmony.Benchmarks project)

BenchmarkDotNet suite with a DataModelSyncBenchmarks (7 sync workloads) and an AddSnapshotsBenchmarks that isolates the persist step, [MemoryDiagnoser] enabled. Run with dotnet run -c Release --project src/SIL.Harmony.Benchmarks.

Testing

Full test suite passes: 241 passing. The only failures are the 6 DataModelPerformanceBenchmarks timing-threshold tests, which also fail on main (environmental, not caused by this change).

Notes for reviewers

The projection is SQLite-specific in a couple of spots that matter for FK correctness (GUIDs stored as uppercase TEXT; ON CONFLICT after a SELECT needs the SQLite WHERE true disambiguator in the code history — the current per-query path uses VALUES). If other providers are ever targeted, FastProjection would need revisiting.
Known limitation: an intra-batch self-reference (e.g. Word.AntonymId pointing at another new Word in the same batch) is not ordered; it's a nullable SET NULL FK and not exercised by current workloads.

Previously AddSnapshots projected each snapshot by calling FindAsync per entity, which issued one database query per snapshot (and, on an initial sync of new data, every query returned null after a round-trip). Pre-load the projected rows that already exist for the batch with a single tracked query per object type. ProjectSnapshot then resolves existing entities from the change tracker and skips the lookup entirely for entities that have no projected row yet, collapsing N queries down to roughly one per distinct object type. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019zEmz7jPRPF6Lv8h6YAWBW

Adds a BenchmarkDotNet suite that measures CrdtRepository.AddSnapshots on its own, across 7 workloads mirroring DataModelSyncBenchmarks. Expensive DB seeding runs once in a template DB; each iteration forks the DB and recomputes the snapshot batch so no EF-tracked state leaks across iterations. - SnapshotWorker.ComputeSnapshotsToPersist: returns the exact snapshot list UpdateSnapshots would persist, without writing it. - DataModelTestBase: internal CreateRepository() and CrdtConfig accessors. - BenchmarkWorkloadBuilders: shared commit builders extracted from DataModelSyncBenchmarks (+ BuildUpdateExisting). - Program.cs: run both suites via BenchmarkSwitcher (handles --filter/args). - Remove leftover Console.WriteLine debug lines from AddSnapshots. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Two experimental fast AddSnapshots implementations that keep the EF snapshot insert unchanged but populate projected tables with raw INSERT ... ON CONFLICT upserts instead of going through EF's change tracker: - FAST: one upsert command per entity row - FAST_JSON: one command per entity type, rows passed as a single JSON array expanded with SQLite json_each/json_extract FastProjection derives table/column names, primary key, the SnapshotId shadow FK, and value converters from the EF model (no per-entity code). It dedups to the latest snapshot per entity, runs deletes before upserts (children-first) then upserts (parents-first) for FK/unique-constraint safety, and reuses the caller's transaction. CrdtRepository.AddSnapshots now selects via #if FAST_JSON / #elif FAST / #else. Program.cs adds a third FAST_JSON benchmark job and DataModelSyncBenchmarks enables [MemoryDiagnoser]. Benchmarks (CreateWords, 1000): both fast paths ~35% faster and ~32% fewer allocations than baseline; per-query vs JSON-batch shows no measurable difference against in-memory SQLite. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The JSON-batch projection benchmarked identically to the per-query path against in-memory SQLite (same time and allocations), so remove it and keep only the per-query raw-SQL upsert path. FastProjection loses the useJsonBatch parameter and all json_each/json_extract code; CrdtRepository.AddSnapshots collapses to #if FAST / #else; the benchmark drops the FAST_JSON job. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Remove the EF change-tracker slow path and the #if FAST conditional so AddSnapshots always uses FastProjection. Deletes the now-dead slow-path helpers (ProjectSnapshot, GetEntityEntry, LoadExistingEntityIds, LoadExistingEntities). The benchmark collapses to a single job since FAST vs DEFAULT are now identical. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

FastProjection is now an injected singleton (registered in AddCrdtDataCore and resolved into CrdtRepository via ActivatorUtilities) instead of a static class. Its per-type projected-table SQL metadata cache moves from a static field onto an internal ConcurrentDictionary on CrdtConfig, so it's shared across repositories/contexts and tied to config lifetime. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

coderabbitai · 2026-07-21T09:56:44Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
auto_review:
drafts: true

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch reduce-sync-work

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

# Conflicts: # src/SIL.Harmony/Config/HarmonyConfig.cs # src/SIL.Harmony/SnapshotWorker.cs

Notify DI interceptors and HarmonyConfig.OnProjectedEntitiesChanged after projected SQL with the latest upsert or delete per entity.

Keep the slnx migration from main and include SIL.Harmony.Benchmarks in the solution.

Main now uses Microsoft.Testing.Platform, so solution-wide dotnet test was launching the Benchmarks exe and failing on unknown MTP flags.

hahn-kev-bot · 2026-07-21T09:56:36Z

Overview

Reduces the work done during sync (AddRangeFromSync → SnapshotWorker.UpdateSnapshots → CrdtRepository.AddSnapshots) and adds a benchmark suite to measure it. On the CreateWords workload at 10k changes this branch takes sync from ~1.96 s / 976 MB allocated down to ~0.99 s / 538 MB — roughly 2× faster and ~45% less memory.

What changed

Snapshot pre-load (SnapshotWorker / DataModel)

UpdateSnapshots now bulk-loads the relevant current snapshots (with their Commit) into a cache keyed by entity id, and SnapshotWorker reads full snapshots straight from that cache instead of issuing a FindSnapshot DB round-trip per cache hit.

Fast raw-SQL projection (FastProjection, new)

Snapshot rows are still inserted through EF unchanged, but the projected tables are now populated with hand-written raw SQL INSERT ... ON CONFLICT(pk) DO UPDATE (one upsert per entity row) instead of going through EF's change tracker (FindAsync / SetValues / graph tracking).
Everything the SQL needs — table/column names (correctly delimited), primary key, the SnapshotId shadow FK, value converters — is derived from the EF model, so there is no per-entity code.
It dedups to the latest snapshot per entity, runs deletes before upserts (children-first) then upserts (parents-first) for FK/unique-constraint safety, and reuses the caller's transaction.
FastProjection is an injectable singleton; its per-type SQL metadata cache lives on an internal ConcurrentDictionary on CrdtConfig, shared across repositories/contexts. This replaces the previous EF change-tracker projection path in CrdtRepository.AddSnapshots, which is removed.

Benchmarks (new SIL.Harmony.Benchmarks project)

BenchmarkDotNet suite with a DataModelSyncBenchmarks (7 sync workloads) and an AddSnapshotsBenchmarks that isolates the persist step, [MemoryDiagnoser] enabled. Run with dotnet run -c Release --project src/SIL.Harmony.Benchmarks.

Testing

Full test suite passes: 241 passing. The only failures are the 6 DataModelPerformanceBenchmarks timing-threshold tests, which also fail on main (environmental, not caused by this change).

Notes for reviewers

The projection is SQLite-specific in a couple of spots that matter for FK correctness (GUIDs stored as uppercase TEXT; ON CONFLICT after a SELECT needs the SQLite WHERE true disambiguator in the code history — the current per-query path uses VALUES). If other providers are ever targeted, FastProjection would need revisiting.
Known limitation: an intra-batch self-reference (e.g. Word.AntonymId pointing at another new Word in the same batch) is not ordered; it's a nullable SET NULL FK and not exercised by current workloads.

Previously AddSnapshots projected each snapshot by calling FindAsync per entity, which issued one database query per snapshot (and, on an initial sync of new data, every query returned null after a round-trip). Pre-load the projected rows that already exist for the batch with a single tracked query per object type. ProjectSnapshot then resolves existing entities from the change tracker and skips the lookup entirely for entities that have no projected row yet, collapsing N queries down to roughly one per distinct object type. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019zEmz7jPRPF6Lv8h6YAWBW

Adds a BenchmarkDotNet suite that measures CrdtRepository.AddSnapshots on its own, across 7 workloads mirroring DataModelSyncBenchmarks. Expensive DB seeding runs once in a template DB; each iteration forks the DB and recomputes the snapshot batch so no EF-tracked state leaks across iterations. - SnapshotWorker.ComputeSnapshotsToPersist: returns the exact snapshot list UpdateSnapshots would persist, without writing it. - DataModelTestBase: internal CreateRepository() and CrdtConfig accessors. - BenchmarkWorkloadBuilders: shared commit builders extracted from DataModelSyncBenchmarks (+ BuildUpdateExisting). - Program.cs: run both suites via BenchmarkSwitcher (handles --filter/args). - Remove leftover Console.WriteLine debug lines from AddSnapshots. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Two experimental fast AddSnapshots implementations that keep the EF snapshot insert unchanged but populate projected tables with raw INSERT ... ON CONFLICT upserts instead of going through EF's change tracker: - FAST: one upsert command per entity row - FAST_JSON: one command per entity type, rows passed as a single JSON array expanded with SQLite json_each/json_extract FastProjection derives table/column names, primary key, the SnapshotId shadow FK, and value converters from the EF model (no per-entity code). It dedups to the latest snapshot per entity, runs deletes before upserts (children-first) then upserts (parents-first) for FK/unique-constraint safety, and reuses the caller's transaction. CrdtRepository.AddSnapshots now selects via #if FAST_JSON / #elif FAST / #else. Program.cs adds a third FAST_JSON benchmark job and DataModelSyncBenchmarks enables [MemoryDiagnoser]. Benchmarks (CreateWords, 1000): both fast paths ~35% faster and ~32% fewer allocations than baseline; per-query vs JSON-batch shows no measurable difference against in-memory SQLite. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The JSON-batch projection benchmarked identically to the per-query path against in-memory SQLite (same time and allocations), so remove it and keep only the per-query raw-SQL upsert path. FastProjection loses the useJsonBatch parameter and all json_each/json_extract code; CrdtRepository.AddSnapshots collapses to #if FAST / #else; the benchmark drops the FAST_JSON job. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Remove the EF change-tracker slow path and the #if FAST conditional so AddSnapshots always uses FastProjection. Deletes the now-dead slow-path helpers (ProjectSnapshot, GetEntityEntry, LoadExistingEntityIds, LoadExistingEntities). The benchmark collapses to a single job since FAST vs DEFAULT are now identical. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

FastProjection is now an injected singleton (registered in AddCrdtDataCore and resolved into CrdtRepository via ActivatorUtilities) instead of a static class. Its per-type projected-table SQL metadata cache moves from a static field onto an internal ConcurrentDictionary on CrdtConfig, so it's shared across repositories/contexts and tied to config lifetime. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

coderabbitai · 2026-07-21T09:56:44Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
auto_review:
drafts: true

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch reduce-sync-work

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

# Conflicts: # src/SIL.Harmony/Config/HarmonyConfig.cs # src/SIL.Harmony/SnapshotWorker.cs

Notify DI interceptors and HarmonyConfig.OnProjectedEntitiesChanged after projected SQL with the latest upsert or delete per entity.

Keep the slnx migration from main and include SIL.Harmony.Benchmarks in the solution.

Main now uses Microsoft.Testing.Platform, so solution-wide dotnet test was launching the Benchmarks exe and failing on unknown MTP flags.

hahn-kev-bot · 2026-07-21T09:56:36Z

Overview

Reduces the work done during sync (AddRangeFromSync → SnapshotWorker.UpdateSnapshots → CrdtRepository.AddSnapshots) and adds a benchmark suite to measure it. On the CreateWords workload at 10k changes this branch takes sync from ~1.96 s / 976 MB allocated down to ~0.99 s / 538 MB — roughly 2× faster and ~45% less memory.

What changed

Snapshot pre-load (SnapshotWorker / DataModel)

UpdateSnapshots now bulk-loads the relevant current snapshots (with their Commit) into a cache keyed by entity id, and SnapshotWorker reads full snapshots straight from that cache instead of issuing a FindSnapshot DB round-trip per cache hit.

Fast raw-SQL projection (FastProjection, new)

Snapshot rows are still inserted through EF unchanged, but the projected tables are now populated with hand-written raw SQL INSERT ... ON CONFLICT(pk) DO UPDATE (one upsert per entity row) instead of going through EF's change tracker (FindAsync / SetValues / graph tracking).
Everything the SQL needs — table/column names (correctly delimited), primary key, the SnapshotId shadow FK, value converters — is derived from the EF model, so there is no per-entity code.
It dedups to the latest snapshot per entity, runs deletes before upserts (children-first) then upserts (parents-first) for FK/unique-constraint safety, and reuses the caller's transaction.
FastProjection is an injectable singleton; its per-type SQL metadata cache lives on an internal ConcurrentDictionary on CrdtConfig, shared across repositories/contexts. This replaces the previous EF change-tracker projection path in CrdtRepository.AddSnapshots, which is removed.

Benchmarks (new SIL.Harmony.Benchmarks project)

BenchmarkDotNet suite with a DataModelSyncBenchmarks (7 sync workloads) and an AddSnapshotsBenchmarks that isolates the persist step, [MemoryDiagnoser] enabled. Run with dotnet run -c Release --project src/SIL.Harmony.Benchmarks.

Testing

Full test suite passes: 241 passing. The only failures are the 6 DataModelPerformanceBenchmarks timing-threshold tests, which also fail on main (environmental, not caused by this change).

Notes for reviewers

The projection is SQLite-specific in a couple of spots that matter for FK correctness (GUIDs stored as uppercase TEXT; ON CONFLICT after a SELECT needs the SQLite WHERE true disambiguator in the code history — the current per-query path uses VALUES). If other providers are ever targeted, FastProjection would need revisiting.
Known limitation: an intra-batch self-reference (e.g. Word.AntonymId pointing at another new Word in the same batch) is not ordered; it's a nullable SET NULL FK and not exercised by current workloads.

Previously AddSnapshots projected each snapshot by calling FindAsync per entity, which issued one database query per snapshot (and, on an initial sync of new data, every query returned null after a round-trip). Pre-load the projected rows that already exist for the batch with a single tracked query per object type. ProjectSnapshot then resolves existing entities from the change tracker and skips the lookup entirely for entities that have no projected row yet, collapsing N queries down to roughly one per distinct object type. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019zEmz7jPRPF6Lv8h6YAWBW

Adds a BenchmarkDotNet suite that measures CrdtRepository.AddSnapshots on its own, across 7 workloads mirroring DataModelSyncBenchmarks. Expensive DB seeding runs once in a template DB; each iteration forks the DB and recomputes the snapshot batch so no EF-tracked state leaks across iterations. - SnapshotWorker.ComputeSnapshotsToPersist: returns the exact snapshot list UpdateSnapshots would persist, without writing it. - DataModelTestBase: internal CreateRepository() and CrdtConfig accessors. - BenchmarkWorkloadBuilders: shared commit builders extracted from DataModelSyncBenchmarks (+ BuildUpdateExisting). - Program.cs: run both suites via BenchmarkSwitcher (handles --filter/args). - Remove leftover Console.WriteLine debug lines from AddSnapshots. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Two experimental fast AddSnapshots implementations that keep the EF snapshot insert unchanged but populate projected tables with raw INSERT ... ON CONFLICT upserts instead of going through EF's change tracker: - FAST: one upsert command per entity row - FAST_JSON: one command per entity type, rows passed as a single JSON array expanded with SQLite json_each/json_extract FastProjection derives table/column names, primary key, the SnapshotId shadow FK, and value converters from the EF model (no per-entity code). It dedups to the latest snapshot per entity, runs deletes before upserts (children-first) then upserts (parents-first) for FK/unique-constraint safety, and reuses the caller's transaction. CrdtRepository.AddSnapshots now selects via #if FAST_JSON / #elif FAST / #else. Program.cs adds a third FAST_JSON benchmark job and DataModelSyncBenchmarks enables [MemoryDiagnoser]. Benchmarks (CreateWords, 1000): both fast paths ~35% faster and ~32% fewer allocations than baseline; per-query vs JSON-batch shows no measurable difference against in-memory SQLite. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The JSON-batch projection benchmarked identically to the per-query path against in-memory SQLite (same time and allocations), so remove it and keep only the per-query raw-SQL upsert path. FastProjection loses the useJsonBatch parameter and all json_each/json_extract code; CrdtRepository.AddSnapshots collapses to #if FAST / #else; the benchmark drops the FAST_JSON job. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Remove the EF change-tracker slow path and the #if FAST conditional so AddSnapshots always uses FastProjection. Deletes the now-dead slow-path helpers (ProjectSnapshot, GetEntityEntry, LoadExistingEntityIds, LoadExistingEntities). The benchmark collapses to a single job since FAST vs DEFAULT are now identical. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

FastProjection is now an injected singleton (registered in AddCrdtDataCore and resolved into CrdtRepository via ActivatorUtilities) instead of a static class. Its per-type projected-table SQL metadata cache moves from a static field onto an internal ConcurrentDictionary on CrdtConfig, so it's shared across repositories/contexts and tied to config lifetime. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

coderabbitai · 2026-07-21T09:56:44Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
auto_review:
drafts: true

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch reduce-sync-work

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

# Conflicts: # src/SIL.Harmony/Config/HarmonyConfig.cs # src/SIL.Harmony/SnapshotWorker.cs

Notify DI interceptors and HarmonyConfig.OnProjectedEntitiesChanged after projected SQL with the latest upsert or delete per entity.

Keep the slnx migration from main and include SIL.Harmony.Benchmarks in the solution.

Main now uses Microsoft.Testing.Platform, so solution-wide dotnet test was launching the Benchmarks exe and failing on unknown MTP flags.

hahn-kevand others added 11 commits July 21, 2026 10:42

add some sync perf benchmarks
136982d

fix benchmark reduce work per iteration
83ba359

pre load snapshots
d264539

create a bunch of benchmark types
ed1c193

enable running new and old code
bef8740

hahn-kev added 5 commits July 22, 2026 10:05

fix formatting
245babb

Merge remote-tracking branch 'origin/main' into reduce-sync-work
a653441
# Conflicts: # src/SIL.Harmony/Config/HarmonyConfig.cs # src/SIL.Harmony/SnapshotWorker.cs

Add once-per-save projected entity interceptor.
e99e7e6
Notify DI interceptors and HarmonyConfig.OnProjectedEntitiesChanged after projected SQL with the latest upsert or delete per entity.

Merge origin/main into reduce-sync-work.
a8a82a7
Keep the slnx migration from main and include SIL.Harmony.Benchmarks in the solution.

Stop treating the Benchmarks project as a test host.
5e379ef
Main now uses Microsoft.Testing.Platform, so solution-wide dotnet test was launching the Benchmarks exe and failing on unknown MTP flags.

hahn-kevand others added 11 commits July 21, 2026 10:42

add some sync perf benchmarks
136982d

fix benchmark reduce work per iteration
83ba359

pre load snapshots
d264539

create a bunch of benchmark types
ed1c193

enable running new and old code
bef8740

hahn-kev added 5 commits July 22, 2026 10:05

fix formatting
245babb

Merge remote-tracking branch 'origin/main' into reduce-sync-work
a653441
# Conflicts: # src/SIL.Harmony/Config/HarmonyConfig.cs # src/SIL.Harmony/SnapshotWorker.cs

Add once-per-save projected entity interceptor.
e99e7e6
Notify DI interceptors and HarmonyConfig.OnProjectedEntitiesChanged after projected SQL with the latest upsert or delete per entity.

Merge origin/main into reduce-sync-work.
a8a82a7
Keep the slnx migration from main and include SIL.Harmony.Benchmarks in the solution.

Stop treating the Benchmarks project as a test host.
5e379ef
Main now uses Microsoft.Testing.Platform, so solution-wide dotnet test was launching the Benchmarks exe and failing on unknown MTP flags.

hahn-kevand others added 11 commits July 21, 2026 10:42

add some sync perf benchmarks
136982d

fix benchmark reduce work per iteration
83ba359

pre load snapshots
d264539

create a bunch of benchmark types
ed1c193

enable running new and old code
bef8740

hahn-kev added 5 commits July 22, 2026 10:05

fix formatting
245babb

Merge remote-tracking branch 'origin/main' into reduce-sync-work
a653441
# Conflicts: # src/SIL.Harmony/Config/HarmonyConfig.cs # src/SIL.Harmony/SnapshotWorker.cs

Add once-per-save projected entity interceptor.
e99e7e6
Notify DI interceptors and HarmonyConfig.OnProjectedEntitiesChanged after projected SQL with the latest upsert or delete per entity.

Merge origin/main into reduce-sync-work.
a8a82a7
Keep the slnx migration from main and include SIL.Harmony.Benchmarks in the solution.

Stop treating the Benchmarks project as a test host.
5e379ef
Main now uses Microsoft.Testing.Platform, so solution-wide dotnet test was launching the Benchmarks exe and failing on unknown MTP flags.

hahn-kevand others added 11 commits July 21, 2026 10:42

add some sync perf benchmarks
136982d

fix benchmark reduce work per iteration
83ba359

pre load snapshots
d264539

create a bunch of benchmark types
ed1c193

enable running new and old code
bef8740

hahn-kev added 5 commits July 22, 2026 10:05

fix formatting
245babb

Merge remote-tracking branch 'origin/main' into reduce-sync-work
a653441
# Conflicts: # src/SIL.Harmony/Config/HarmonyConfig.cs # src/SIL.Harmony/SnapshotWorker.cs

Add once-per-save projected entity interceptor.
e99e7e6
Notify DI interceptors and HarmonyConfig.OnProjectedEntitiesChanged after projected SQL with the latest upsert or delete per entity.

Merge origin/main into reduce-sync-work.
a8a82a7
Keep the slnx migration from main and include SIL.Harmony.Benchmarks in the solution.

Stop treating the Benchmarks project as a test host.
5e379ef
Main now uses Microsoft.Testing.Platform, so solution-wide dotnet test was launching the Benchmarks exe and failing on unknown MTP flags.

hahn-kevand others added 11 commits July 21, 2026 10:42

add some sync perf benchmarks
136982d

fix benchmark reduce work per iteration
83ba359

pre load snapshots
d264539

create a bunch of benchmark types
ed1c193

enable running new and old code
bef8740

hahn-kev added 5 commits July 22, 2026 10:05

fix formatting
245babb

Merge remote-tracking branch 'origin/main' into reduce-sync-work
a653441
# Conflicts: # src/SIL.Harmony/Config/HarmonyConfig.cs # src/SIL.Harmony/SnapshotWorker.cs

Add once-per-save projected entity interceptor.
e99e7e6
Notify DI interceptors and HarmonyConfig.OnProjectedEntitiesChanged after projected SQL with the latest upsert or delete per entity.

Merge origin/main into reduce-sync-work.
a8a82a7
Keep the slnx migration from main and include SIL.Harmony.Benchmarks in the solution.

Stop treating the Benchmarks project as a test host.
5e379ef
Main now uses Microsoft.Testing.Platform, so solution-wide dotnet test was launching the Benchmarks exe and failing on unknown MTP flags.

hahn-kevand others added 11 commits July 21, 2026 10:42

add some sync perf benchmarks
136982d

fix benchmark reduce work per iteration
83ba359

pre load snapshots
d264539

create a bunch of benchmark types
ed1c193

enable running new and old code
bef8740

hahn-kev added 5 commits July 22, 2026 10:05

fix formatting
245babb

Merge remote-tracking branch 'origin/main' into reduce-sync-work
a653441
# Conflicts: # src/SIL.Harmony/Config/HarmonyConfig.cs # src/SIL.Harmony/SnapshotWorker.cs

Add once-per-save projected entity interceptor.
e99e7e6
Notify DI interceptors and HarmonyConfig.OnProjectedEntitiesChanged after projected SQL with the latest upsert or delete per entity.

Merge origin/main into reduce-sync-work.
a8a82a7
Keep the slnx migration from main and include SIL.Harmony.Benchmarks in the solution.

Stop treating the Benchmarks project as a test host.
5e379ef
Main now uses Microsoft.Testing.Platform, so solution-wide dotnet test was launching the Benchmarks exe and failing on unknown MTP flags.

hahn-kevand others added 11 commits July 21, 2026 10:42

add some sync perf benchmarks
136982d

fix benchmark reduce work per iteration
83ba359

pre load snapshots
d264539

create a bunch of benchmark types
ed1c193

enable running new and old code
bef8740

hahn-kev added 5 commits July 22, 2026 10:05

fix formatting
245babb

Merge remote-tracking branch 'origin/main' into reduce-sync-work
a653441
# Conflicts: # src/SIL.Harmony/Config/HarmonyConfig.cs # src/SIL.Harmony/SnapshotWorker.cs

Add once-per-save projected entity interceptor.
e99e7e6
Notify DI interceptors and HarmonyConfig.OnProjectedEntitiesChanged after projected SQL with the latest upsert or delete per entity.

Merge origin/main into reduce-sync-work.
a8a82a7
Keep the slnx migration from main and include SIL.Harmony.Benchmarks in the solution.

Stop treating the Benchmarks project as a test host.
5e379ef
Main now uses Microsoft.Testing.Platform, so solution-wide dotnet test was launching the Benchmarks exe and failing on unknown MTP flags.

hahn-kevand others added 11 commits July 21, 2026 10:42

add some sync perf benchmarks
136982d

fix benchmark reduce work per iteration
83ba359

pre load snapshots
d264539

create a bunch of benchmark types
ed1c193

enable running new and old code
bef8740

hahn-kev added 5 commits July 22, 2026 10:05

fix formatting
245babb

Merge remote-tracking branch 'origin/main' into reduce-sync-work
a653441
# Conflicts: # src/SIL.Harmony/Config/HarmonyConfig.cs # src/SIL.Harmony/SnapshotWorker.cs

Add once-per-save projected entity interceptor.
e99e7e6
Notify DI interceptors and HarmonyConfig.OnProjectedEntitiesChanged after projected SQL with the latest upsert or delete per entity.

Merge origin/main into reduce-sync-work.
a8a82a7
Keep the slnx migration from main and include SIL.Harmony.Benchmarks in the solution.

Stop treating the Benchmarks project as a test host.
5e379ef
Main now uses Microsoft.Testing.Platform, so solution-wide dotnet test was launching the Benchmarks exe and failing on unknown MTP flags.

Uh oh!

Conversation

hahn-kev-bot commented Jul 21, 2026

Overview

What changed

Testing

Notes for reviewers

Uh oh!

coderabbitaiBot commented Jul 21, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Draft PR not reviewed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

hahn-kev-bot commented Jul 21, 2026

Overview

What changed

Testing

Notes for reviewers

Uh oh!

coderabbitaiBot commented Jul 21, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Draft PR not reviewed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

hahn-kev-bot commented Jul 21, 2026

Overview

What changed

Testing

Notes for reviewers

Uh oh!

coderabbitaiBot commented Jul 21, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Draft PR not reviewed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

hahn-kev-bot commented Jul 21, 2026

Overview

What changed

Testing

Notes for reviewers

Uh oh!

coderabbitaiBot commented Jul 21, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Draft PR not reviewed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

coderabbitaiBot commented Jul 21, 2026•
edited
Loading

coderabbitaiBot commented Jul 21, 2026•
edited
Loading

coderabbitaiBot commented Jul 21, 2026•
edited
Loading

coderabbitaiBot commented Jul 21, 2026•
edited
Loading

coderabbitaiBot commented Jul 21, 2026•
edited
Loading

coderabbitaiBot commented Jul 21, 2026•
edited
Loading