tcmalloc: large alloc error

[HannanKan]

I run diskann on my server with 128g main memory and 1T SSD.
My dataset has 10M points and each of them is 200-dim.
I build diskann and run with scripts
./tests/build_disk_index float mips /home/user1/ann/dataset/data10m/data10m /home/user1/ann/dataset/index/index 100 100 64g 128g 64 20
I vary the last parameter from 2 to 100(2, 20, 50,100). I always get "tcmalloc: large alloc " error followed by segment fault (core dump).

Look forward to your help!

[ShikharJ]

Hannankan (@HannanKan) Could you please paste the error output? I have experienced tcmalloc errors in the past, and they were almost always due to installation issues. Can you check that the installation steps you're following look like this? Also, for 10M points in 200 dimensions, I don't think you'll need DiskANN per se. You can make do with Vamana. Just make use of the build_memory_index and search_memory_index functionality?

[HannanKan]

I followed steps in repo's README totally during environment setup(so I use mkl 2019.4-070 version). Actually, in my business, I have to query billions of vectors. Currently, I just use small datasets to verify. I will show you my complete logs.
To debug, I change datasets to 1m and reduce threads to 16. My running parameter is
./tests/build_disk_index float ps /home/node/ann/dataset/data1m/data1m /home/node/ann/dataset/index/index_pq20 100 100 64g 128g 16 50 >> log 2>&1 &
Input data layout is checked and is ok.
I use gdb to accidently find that there is a vector whose capacity is super larger, 2*10^14 magnitude.
when I set thread number to 1, it works.

[HannanKan]

error log: https://github.com/HannanKan/diskann_err_log/blob/main/errlog1

[ShikharJ]

Maybe try 64000000 instead of 64g? I don't think the parser works for subscripts.

[HannanKan]

err log: https://github.com/HannanKan/diskann_err_log/blob/main/64g-64000000.
still cannot work.
Can I sent my data and running parameter to you so you can reproduce my error or detect my problem？

[ShikharJ]

Can you please share the final command that you're running? The data isn't needed.

[HannanKan]

./tests/build_disk_index float mips /home/node/ann/dataset/data1m/data1m /home/node/ann/dataset/index/index_pq20 100 100 64g 128g 64 50
/home/node/ann/dataset/data1m/data1m: path to my base data. 1 million vectors
/home/node/ann/dataset/index/index_pq20: path to put index

[HannanKan]

Can you please share the final command that you're running? The data isn't needed.

Hi! Can you please tell me if you have found solution to my problem?

[ShikharJ]

Hannankan (@HannanKan) I was able to try out with 1M points and 128-dimensions (SIFT1M) with the same parameters that you're using, and I'm able to build normally. I'm guessing this is an installation / RAM limit issue regarding tcmalloc. Maybe try uninstalling and re-installing gperftools? In the meantime, I'll try out a bigger dataset.

[HannanKan]

Shikhar Jaiswal (@ShikharJ) I truly thank you for your help. After 3 steps, I can build and use DiskANN(other problem stated in another issue).

• remove and reinstall g++
• remove and reinstall gperftools
• remove mkl 2019.4-070 and reinstall intel-mkl-64bit-2020.0-088

After that, my search time (with index) is super super slow.(QPS 30.41, Xeon(R) Gold 5218 2.3GHz, 1T SSD 128g memory, 64 thread. 10m data)

[ShikharJ]

Hannankan (@HannanKan) Glad to hear that helped. Could you post the final build and search commands you're running, along with the search output?

[HannanKan]

I use 10m data point and 100k queries;
index build phase command:
./tests/build_disk_index float mips /home/node/ann/dataset/data10m/data10m /home/node/ann/dataset/index/index 100 100 64g 128g 64 0
build phase takes 3840s(~1h)
build parameters are listed as such

#base data	R(degree)	L(search_list_size)	B(search_memory)	M(build_memory)	T(num_threads)	PQ(bytes_per_points)
10m	100	100	64g	128g	64	0

search command:
./tests/search_disk_index float mips ../../../dataset/index/index 5000 8 4 ../../../dataset/query/query.public.100K.fbin ../../../dataset/GT_10M/text2image-10M 10 ../../../dataset/result/result 10
search parameters is listed:

#query data	cached_nodes	num_threads	beamwidth	recall@k	search_list_size
100k	5000	8	4	10	10

output is as follows:

L	Beamwidth	QPS	Mean Latency	99.9 Latency	Mean IOs	CPU (s)	Recall@10
10	4	27.38	583099.56	1561211.00	21.09	6296.62	64.66

[ShikharJ]

Hannankan (@HannanKan) Apologies for the late reply. The output is indeed weird, in the sense that I can generate much better values using the in-memory index itself. Could you paste the whole search command output please?

[HannanKan]

I thank you so much. After replacing HDD with SSD, search time reduces sharply.

[ShikharJ]

Hannankan (@HannanKan) Great! I'm glad that I could be of help, and I hope DiskANN is useful to the task that you're aiming to accomplish. I'm closing this issue for now then.