[PoC] Initial support for Optane PM

[Mark Hildebrand (hildebrandmw)]

Hi there!

This proof of concept PR implements a small set of changes to explore using Optane PM to store the dataset and graph data structures for benchmarking purposes.

Additional arguments are added to the build_memory_index and search_memory_index binaries. These are:

• The directory to use for the PM mapping. The current PM programming model involves mounting the PM DIMMs as a DAX (Direct Access) filesystem. When files are memory mapped from this directory, no page cache is used and instead loads and stores to the mapped virtual memory addresses go straight to the underlying media. If you don't want to use PM, this argument can be left as "null"
• Boolean flag indicating whether the dataset should be placed in PM (true) or DRAM (false).
• Boolean flag indicating whether the graph should be placed in PM (true) or DRAM (false).

Limitations

This PR is mainly meant for benchmarking purposes/start a discussion. As such, it only supports Linux based operating systems.

This change also adds an additional dependency of libmemkind to support heap allocations in either DRAM or PM. On Ubuntu systems, this library can be installed system wide with

apt install libmemkind-dev libmemkind0

Other distributions require building libmemkind from scratch (which fortunately is not too difficult).

Furthermore, though it uses persistent memory as if it were volatile memory. If robust persistence is required, a library like PMDK would have to be used. The change set would be larger to support the transactional semantics of PMDK.

Other Notes

The implementation of the pmem_allocator as a lazily initialized global is definitely a bit awkward (also definitely not particularly flexible). An alternative could be to replace the raw pointer for the dataset with something like a unique_ptr with the custom allocator - but this could lead to combinatorial expansion in Index template parameters to support the Cartesian product of datatypes/graph allocators/dataset allocators, which also seems less than ideal. Since allocations are not on the critical path, we could use something like std::function to wrap the allocation/free functions. I'm open to suggestions 😃

I'm more than happy to answer any questions, make changes to this PR, and provide any assistance in exploring this direction. Thanks!!

[Harsha Simhadri (harsha-simhadri)]

Mark, Thanks a lot for this contribution. A few questions from the compatibility side:

1. You modified the index.h file. Is the modified file still usable for non-Optance use cases?
2. Will the modified file compile with MSVC/Windows? I am not expecting all of your contribution to compile in MSVC, but we should be able to demarcate current use cases which should compile on Windows, and the Optane use cases which can be limited to linux.

[Mark Hildebrand (hildebrandmw)]

Hi Harsha,

Thanks for the questions!

1. Yes, the modified index.h file is still valid for non-Optane use (and this should be the default behavior as well.) In places where function signatures are modified, I tried to not break the existing API by using default arguments preserving the original behavior.
2. Unfortunately, the code in this state will not build on Windows due to the memkind dependency. However, given enough #ifdefs in the right places, I'm pretty sure we can get it to compile on both OS'es. Would you like us to go ahead and make those modifications?

[Mark Hildebrand (hildebrandmw)]

Hi Harsha,

We've added some changes that should let the code work compile and run on both Linux and Windows (the Windows version does not support Optane, but should otherwise work just like before). Please let us know if you run into any issues!

[Jordan Maples (JordanMaples)]

closing - cleaning up old prs