Skip to content

One embedding refresh per base at a time - #295

Merged
WaylandYang merged 2 commits into
devfrom
fix/one-embedding-refresh-per-base
Sep 4, 2026
Merged

WaylandYang merged 2 commits into
devfrom
fix/one-embedding-refresh-per-base

Conversation

@WaylandYang

Copy link
Copy Markdown
Contributor

Closes #282.

Creating a base with three packs enqueues three embed_ontology jobs. With worker concurrency at 64 they ran at the same time, each read the same set of terms still missing a vector, and each embedded all of them: 3,752 terms embedded three times in the report on #282.

Refreshes for one base now take turns. A per-base tokio::sync::Mutex (in a process-wide table) is held for the duration of refresh_scoped; a job that arrives while another is running waits, then re-reads the missing set, which by then is empty or holds only the terms added after the first job read its set. Waiting rather than skipping means nothing is left for a later refresh to pick up. Different bases are unaffected.

A unit test pins the behaviour: two refreshes of one base serialise, another base is not blocked.

cargo fmt, cargo clippy -D warnings, cargo test -p utopia-server ontology_index pass.

🤖 Generated with Claude Code

WaylandYang and others added 2 commits September 4, 2026 10:35
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Wayland Yang <wayland0916@gmail.com>
@WaylandYang
WaylandYang merged commit 7dc2af3 into dev Sep 4, 2026
4 checks passed
@WaylandYang
WaylandYang deleted the fix/one-embedding-refresh-per-base branch September 4, 2026 02:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Installing several packs embeds the whole ontology once per pack

1 participant