Symptom
Creating a base with three packs (schema.org, W3C Org, PROV-O) produced this, within two seconds:
23:20:04 本体向量已补齐 kb_id=01a06992… count=3752
23:20:05 本体向量已补齐 kb_id=01a06992… count=3752
23:20:06 本体向量已补齐 kb_id=01a06992… count=3752
count is what that run embedded, not what exists. The same 3,752 terms were sent to the embedding endpoint three times: 11,256 calls where 3,752 would do. With five packs it would be five times. This is paid work, on the first minute of every new base.
Cause
owl_import.rs:755 enqueues one embed_ontology job per imported pack. jobs::enqueue does not deduplicate, and the worker runs with concurrency 64, so all three jobs start together. Each calls ontology_index::refresh, which reads types_needing_embedding (correctly: only terms whose embedded text does not match) — but all three read it before any of them has written, so each sees the full set and embeds it. The "补齐" logic is idempotent in result and triple in cost.
Fix options
- Single-flight per base. In
refresh_scoped, take a per-kb_id lock (an in-process Mutex<HashSet<Uuid>> is enough; the jobs are in one process) and return early if a refresh for that base is already running. The later jobs then find nothing stale and exit. Smallest change, no schema.
- Dedupe at enqueue. Skip enqueueing
embed_ontology when a queued or running job for the same kb_id exists. Cleaner queue, but a running job that has already read its stale set will still miss terms the next pack adds, so the import path would need to re-enqueue after that job finishes.
- Claim rows. Mark terms as being embedded before the call, so concurrent runs partition instead of duplicate. Most robust, most code.
The first option plus keeping the per-pack enqueue is probably right: the extra jobs cost one query each, and any terms a later pack adds after the running refresh read its set are picked up by the last job in line.
Test
A database-backed test in crates/utopia-store/tests cannot see the server's in-process lock, so this one belongs in utopia-server: import two packs into a fresh base, run two refresh calls concurrently against a counting fake embed client, assert the total embedded equals the number of distinct terms.
Symptom
Creating a base with three packs (schema.org, W3C Org, PROV-O) produced this, within two seconds:
countis what that run embedded, not what exists. The same 3,752 terms were sent to the embedding endpoint three times: 11,256 calls where 3,752 would do. With five packs it would be five times. This is paid work, on the first minute of every new base.Cause
owl_import.rs:755enqueues oneembed_ontologyjob per imported pack.jobs::enqueuedoes not deduplicate, and the worker runs with concurrency 64, so all three jobs start together. Each callsontology_index::refresh, which readstypes_needing_embedding(correctly: only terms whose embedded text does not match) — but all three read it before any of them has written, so each sees the full set and embeds it. The "补齐" logic is idempotent in result and triple in cost.Fix options
refresh_scoped, take a per-kb_idlock (an in-processMutex<HashSet<Uuid>>is enough; the jobs are in one process) and return early if a refresh for that base is already running. The later jobs then find nothing stale and exit. Smallest change, no schema.embed_ontologywhen a queued or running job for the samekb_idexists. Cleaner queue, but a running job that has already read its stale set will still miss terms the next pack adds, so the import path would need to re-enqueue after that job finishes.The first option plus keeping the per-pack enqueue is probably right: the extra jobs cost one query each, and any terms a later pack adds after the running refresh read its set are picked up by the last job in line.
Test
A database-backed test in
crates/utopia-store/testscannot see the server's in-process lock, so this one belongs inutopia-server: import two packs into a fresh base, run tworefreshcalls concurrently against a counting fake embed client, assert the total embedded equals the number of distinct terms.