Uh oh!
There was an error while loading. Please reload this page.
Graph-IVF: add SPFresh-style LIRE maintenance with bounded merge planning - #1379
Draft
xwj-ox wants to merge 5 commits into
Draft
Graph-IVF: add SPFresh-style LIRE maintenance with bounded merge planning#1379xwj-ox wants to merge 5 commits into
xwj-ox wants to merge 5 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reference Issues/PRs
Includes the LIRE implementation from #1369 together with the subsequent batch-correctness fixes and split-batch telemetry. The comparison base is
u/adkrishnan/graph-ivfat90cb5cafc75199d407873141b3029610e33f653d; the treatment head is287b086f48563a2d3312c70b615011b7f1c99489.What does this implement/fix? Briefly explain your changes.
The existing online builder independently bisects overflowing postings, then reassigns every point in each local parent-and-neighbor region. Delete maintenance scatters retired-posting members among preselected survivors and can leave a destination overfull until a later insertion. This change implements SPFresh-style LIRE maintenance to reduce unnecessary reassignment and handle overflow after either inserts or merges.
region_points,npa_candidates, andoperation_indexto split CSV telemetry while preserving the existing column prefix. Attributeclusters_updatedacross a complete cascade batch and distinguish delete-triggered split batches. UpdateONLINE.mdand add regression coverage.Any other comments?
Validation on Windows at treatment head
287b086f:cargo fmt --all --checkpassed.cargo test -p diskann-graphivf --lockedpassed: 96 unit tests and 11 integration tests; no doctests.cargo clippy -p diskann-graphivf --all-targets --locked -- -D warningspassed.git diff --check 90cb5cafc75199d407873141b3029610e33f653d..287b086f48563a2d3312c70b615011b7f1c99489passed.Regression tests cover duplicate-vector balance, same-batch neighbor arrivals, co-split children, projected merge capacity and reservations, merging underfull victims with each other, graph-search widening and exact fallback, merge-induced overflow, ID-budget exhaustion, and telemetry serialization.
Compatibility and scope:
SplitEventgains public fields, so downstream Rust struct literals need updating. CSV readers should select columns by header.reassign_neighborsbounds the nearby postings examined by split filters and seeds the merge search budget; it does not cap the final routing domain. Graph mode remains approximate, and the filter region is local. The bounded training fit is an adaptation, not a reproduction of SPFresh's full storage system.clusters_updatedmeasures logical postings, not physical SSD writes. Parent migration counts and NPA-candidate counts are separate diagnostics, not a strict funnel.