Skip to content

Graph-IVF: add SPFresh-style LIRE maintenance with bounded merge planning - #1379

Draft
xwj-ox wants to merge 5 commits into
u/adkrishnan/graph-ivffrom
codex/graph-ivf-lire-correctness-fixes
Draft

Graph-IVF: add SPFresh-style LIRE maintenance with bounded merge planning#1379
xwj-ox wants to merge 5 commits into
u/adkrishnan/graph-ivffrom
codex/graph-ivf-lire-correctness-fixes

Conversation

@xwj-ox

Copy link
Copy Markdown
  • Does this PR have a descriptive title that could go in our release notes?
  • Does this PR add any new dependencies?
  • Does this PR modify any existing APIs?
  • Is the change to the API backwards compatible?
  • Should this result in any changes to our documentation, either updating existing docs or adding new ones?

Reference Issues/PRs

Includes the LIRE implementation from #1369 together with the subsequent batch-correctness fixes and split-batch telemetry. The comparison base is u/adkrishnan/graph-ivf at 90cb5cafc75199d407873141b3029610e33f653d; the treatment head is 287b086f48563a2d3312c70b615011b7f1c99489.

What does this implement/fix? Briefly explain your changes.

The existing online builder independently bisects overflowing postings, then reassigns every point in each local parent-and-neighbor region. Delete maintenance scatters retired-posting members among preselected survivors and can leave a destination overfull until a later insertion. This change implements SPFresh-style LIRE maintenance to reduce unnecessary reassignment and handle overflow after either inserts or merges.

  • Fit each split parent independently using capacity-constrained binary clustering. Initialize with one random member and its farthest member, then repair training assignments using distance-preference margins. Final placement uses the nearer child and preserves the fitted assignment for exact distance ties, including duplicate vectors.
  • Apply LIRE necessary-condition filters to parent and nearby postings, including points routed to neighbors in the same insert batch. Publish all children and retire all split parents before routing the deduplicated candidate points against the configured live-centroid router. Keep a local child or surviving source when the graph result is no closer, and relocate only points whose assignment changes.
  • Plan capacity-compatible merge targets using the existing centroid graph, bounded candidate expansion, exact candidate reranking, and a packed nearest-feasible exact fallback. Account for same-batch deletes and capacity reserved by earlier victims. Retry skipped victims against the eventual retirement set and protect any skipped victim selected as a survivor.
  • Retire the planned merge victims before final routing. Materialize the capacity-safe target, apply final global-domain relocations, and run the same split cascade for any resulting overflow. Without an explicit cluster cap, report centroid-ID exhaustion instead of successfully returning an overfull partition.
  • Append region_points, npa_candidates, and operation_index to split CSV telemetry while preserving the existing column prefix. Attribute clusters_updated across a complete cascade batch and distinguish delete-triggered split batches. Update ONLINE.md and add regression coverage.

Any other comments?

Validation on Windows at treatment head 287b086f:

  • cargo fmt --all --check passed.
  • cargo test -p diskann-graphivf --locked passed: 96 unit tests and 11 integration tests; no doctests.
  • cargo clippy -p diskann-graphivf --all-targets --locked -- -D warnings passed.
  • git diff --check 90cb5cafc75199d407873141b3029610e33f653d..287b086f48563a2d3312c70b615011b7f1c99489 passed.

Regression tests cover duplicate-vector balance, same-batch neighbor arrivals, co-split children, projected merge capacity and reservations, merging underfull victims with each other, graph-search widening and exact fallback, merge-induced overflow, ID-budget exhaustion, and telemetry serialization.

Compatibility and scope:

  • No dependencies are added. SplitEvent gains public fields, so downstream Rust struct literals need updating. CSV readers should select columns by header.
  • reassign_neighbors bounds the nearby postings examined by split filters and seeds the merge search budget; it does not cap the final routing domain. Graph mode remains approximate, and the filter region is local. The bounded training fit is an adaptation, not a reproduction of SPFresh's full storage system.
  • The centroid-selection recall diagnostic now accepts exact IDs before applying a floating-point tie tolerance. This does not change query top-k selection, but changes the diagnostic's boundary treatment.
  • clusters_updated measures logical postings, not physical SSD writes. Parent migration counts and NPA-candidate counts are separate diagnostics, not a strict funnel.
  • Online clustering remains synchronous and memory resident. No new full MSTuring30M ABBA campaign was run as part of this submission; this PR makes no new benchmark-gain claim. Reports and slides are excluded.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@xwj-ox@xiaoweiJiang-Suzhou