Uh oh!
There was an error while loading. Please reload this page.
feat(memory): add livekit-memory — sub-10ms in-process semantic memory - #725
feat(memory): add livekit-memory — sub-10ms in-process semantic memory#725Piyussh01 wants to merge 2 commits into
Conversation
|
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
| digest = hashlib.blake2b(token.encode("utf-8"), digest_size=8).digest() | ||
| h = int.from_bytes(digest, "little") | ||
| idx = h % self._dims | ||
| sign = 1.0 if (h >> 1) & 1 else -1.0 |
There was a problem hiding this comment.
🟡 HashingEmbedder sign bit is fully determined by bucket index, defeating signed feature hashing
In HashingEmbedder._token_index_sign, the sign is derived from bit 1 of the hash ((h >> 1) & 1), while the bucket index is h % self._dims. When dims is a power of 2 (which the default 256 is, and all test values like 64 are), h % dims equals h & (dims-1), meaning bit 1 is part of the bits that determine the bucket. This makes the sign a deterministic function of the bucket — tokens that collide into the same bucket always get the same sign, so collision contributions never cancel. Empirically verified: with 10,000 test tokens, 0 out of 256 (or 64) buckets ever see mixed signs. The fix is to use a high bit for the sign (e.g., (h >> 32) & 1), which is independent of the low bits used for the bucket.
| sign=1.0if (h>>1) &1else-1.0 | |
| sign=1.0if (h>>32) &1else-1.0 |
Was this helpful? React with 👍 or 👎 to provide feedback.
| def add(self, key: int, vector: np.ndarray) -> None: | ||
| # usearch upserts when a key already exists. | ||
| self._index.add(key, np.ascontiguousarray(vector, dtype=np.float32)) |
There was a problem hiding this comment.
🚩 UsearchIndex.add() claims upsert semantics that may not hold in all usearch versions
The comment at livekit-memory/livekit/memory/_index.py:202 says "usearch upserts when a key already exists," but in many usearch v2.x versions, Index.add() allows duplicate keys by default rather than replacing. If this assumption is wrong, _put() at livekit-memory/livekit/memory/store.py:181 would create duplicate vectors in the HNSW index when updating an existing item's text/embedding within the same namespace (since the old vector is only removed when the namespace changes, per store.py:165-167). This would cause search to potentially return stale vectors. Could not verify against the actual usearch version since it wasn't installed in the test environment. Worth confirming with a test that exercises upsert on UsearchIndex specifically.
Was this helpful? React with 👍 or 👎 to provide feedback.
xianshijing-lk
commented
Jun 25, 2026
cc @theomonnom |
Piyussh01
commented
Jul 4, 2026
following up on this @xianshijing-lk i think it'll be a nice addition to livekits product roadmap. we can try moving away from static embedding in the future |
Problem
Voice agents need a user's context mid-turn, under a tight latency budget. A remote vector DB round-trip or a transformer query-embed on CPU (~10ms median, ~50ms p99) is enough to break a live conversation loop.
Approach
A new self-contained workspace package
livekit-memory(livekit.memory) providing in-process semantic memory:Model2VecEmbedder) — token-lookup + mean-pool, no transformer forward pass. ~0.03ms per short query.BruteForceIndex(one normalized matmul, sub-ms to ~100k vectors), auto-upgrading toUsearchIndex(HNSW) past ~100k.factsnamespace (pinned user profile) plus an ANN-indexed semantic collection, with acontext()helper that returns one prompt-ready string per turn.Evidence (measured, Apple M4 Pro, 384d)
~30x under the 10ms budget at a million vectors. Transformer embedders (fastembed MiniLM ~9.4ms median / ~53ms p99) confirm why the static embedder is required.
Scope & safety
rtc/api/protocol.model2vec/usearch/fastembedare optional extras, env-gated to Python >= 3.10.HashingEmbedder+ brute-force is the default, so tests run offline with no model downloads.make check(ruff format, ruff lint, mypy --strict) and adds 11 tests.Caveats
onnxextra (CallableEmbedder+ fastembed) is the opt-in escape hatch for higher recall at higher latency.Happy to discuss whether this belongs here vs.
livekit/agents, and to trim to the pure-numpy core if preferred.