Hi!
The FreshDiskANN paper outlines the StreamingMerge procedure. In combing through the codebase (main @ f8ef303), there doesn't appear to be a singular entrypoint that allows a caller to utilize the FreshDiskANN API contract without being aware of all the types of indices.
test_streaming_scenario.cpp outlines how to build an in-memory index that supports inserts and deletes.build_stitched_index.cpp outlines how to merge indicessearch_disk_index.cpp demonstrates how to run a search across an index that is stored on disk.
Given a client that provides a memory budget and no starting list of vectors, my reading of the paper would indicate the following needs to be done in a wrapping class:
- create an empty, streaming enabled, in-memory index that holds writes - this is outlined in
test_streaming_scenario.cpp and is the only sink for insertions.
- create an empty, SSD resident index, which is demonstrated by
build_disk_index.cpp... this index would not have a true build phase as there is nothing to add.
- once the mutable index in [1] is full, merge [1] and [2] using the routine outlined in
merge_shards within disk_utils.h - during the merge process, we would have already created a new mutable in-memory index for any in-flight writes + deletes.
- separately, maintain a list of deletions that are used for filtering within all live indices.
I would be happy to submit a patch that unifies the above in such a way that a caller can just create an Index and not have to worry about RO-TempIndex, RW-TempIndex and the SSD-Resident Index; however, I would like to confirm that my read on the current codebase is correct in that there is no singular entrypoint for this.
Hi!
The FreshDiskANN paper outlines the StreamingMerge procedure. In combing through the codebase (main @ f8ef303), there doesn't appear to be a singular entrypoint that allows a caller to utilize the FreshDiskANN API contract without being aware of all the types of indices.
test_streaming_scenario.cppoutlines how to build an in-memory index that supports inserts and deletes.build_stitched_index.cppoutlines how to merge indicessearch_disk_index.cppdemonstrates how to run a search across an index that is stored on disk.Given a client that provides a memory budget and no starting list of vectors, my reading of the paper would indicate the following needs to be done in a wrapping class:
test_streaming_scenario.cppand is the only sink for insertions.build_disk_index.cpp... this index would not have a true build phase as there is nothing to add.merge_shardswithin disk_utils.h - during the merge process, we would have already created a new mutable in-memory index for any in-flight writes + deletes.I would be happy to submit a patch that unifies the above in such a way that a caller can just create an Index and not have to worry about RO-TempIndex, RW-TempIndex and the SSD-Resident Index; however, I would like to confirm that my read on the current codebase is correct in that there is no singular entrypoint for this.