Optimize the peak memory usage when loading dataset.

[Light-V]

I've made a small optimization in data loading process that reduces peak memory usage when handling large datasets.
Previously, we were using np.array() to convert large datasets from h5py objects into NumPy arrays. This operation, while straightforward, was causing a spike in memory usage due to the creation of a new array copy, which could lead to OOM errors when working with particularly large datasets.
To mitigate this issue, I have replaced np.array() with np.asarray() in the relevant sections of the code. The np.asarray() function, unlike np.array(), will attempt to pass through the input data without creating a new array copy if the input is already a NumPy array. This behavior helps to reduce unnecessary memory allocation and can be particularly effective in scenarios where memory is a constraint.

[maumueller]

Thanks @Light-V. Did you observe a change in index building time/search performance with this change? I have made very bad experiences without this cast, but this was a long time ago.

[Light-V]

@maumueller Thank you for your suggestion. I will give it a try and post the performance comparison later.

[Light-V]

Hi, @maumueller

I have run ann-benchmark on qsg-ngt with these two different ways to load dataset. And here's the search result:

diff result.1 result.2
3c3
< 0: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 3.000) 0.994 9433.409

0: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 3.000) 0.994 9616.273
5c5
< 1: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 0.000) 0.461 40888.565

---

1: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 0.000) 0.462 41178.560
7c7
< 2: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 1.500) 0.922 19599.222

---

2: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 1.500) 0.923 19792.332
9c9
< 3: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 2.000) 0.730 24640.807

---

3: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 2.000) 0.732 24812.362
11c11
< 4: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 2.000) 0.980 13130.141

---

4: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 2.000) 0.981 13396.878
13c13
< 5: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 1.500) 0.626 28672.532

---

5: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 1.500) 0.628 28876.686
15c15
< 6: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 5.000) 0.994 7805.384

---

6: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 5.000) 0.994 7890.528
17c17
< 7: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 1.500) 0.954 16309.917

---

7: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 1.500) 0.956 16625.485
19c19
< 8: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 3.000) 0.955 14523.425

---

8: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 3.000) 0.955 14687.466
21c21
< 9: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 0.000) 0.813 27601.332

---

9: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 0.000) 0.814 27835.597
23c23
< 10: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 8.000) 1.000 2951.642

---

10: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 8.000) 1.000 3008.100
25c25
< 11: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 3.000) 0.998 7070.455

---

11: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 3.000) 0.999 7199.444
27c27
< 12: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 1.200) 0.696 28557.780

---

12: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 1.200) 0.698 28513.283
29c29
< 13: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 1.200) 0.875 22116.188

---

13: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 1.200) 0.876 22458.581
31c31
< 14: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 3.000) 0.980 11920.713

---

14: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 3.000) 0.981 12182.230
33c33
< 15: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 5.000) 1.000 4511.494

---

15: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 5.000) 1.000 4587.789
35c35
< 16: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 2.000) 0.991 10027.577

---

16: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 2.000) 0.992 10132.040
37c37
< 17: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 2.000) 0.845 21594.956

---

17: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 2.000) 0.844 21941.602
39c39
< 18: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 1.200) 0.910 18668.998

---

18: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 1.200) 0.912 19092.833
41c41
< 19: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 1.200) 0.538 31914.098

---

19: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 1.200) 0.536 32022.208
43c43
< 20: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 0.000) 0.625 36063.251

---

20: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 0.000) 0.628 36227.625
45c45
< 21: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 1.200) 0.803 25712.051

---

21: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 1.200) 0.804 25995.405
47c47
< 22: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 10.000) 0.999 4170.250

---

22: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 10.000) 0.999 4243.505
49c49
< 23: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 2.000) 0.911 19146.093

---

23: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 2.000) 0.911 19330.184
51c51
< 24: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 0.000) 0.740 32339.626

---

24: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 0.000) 0.740 32329.880
53c53
< 25: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 2.000) 0.955 16193.192

---

25: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 2.000) 0.956 16318.249
55c55
< 26: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 3.000) 0.841 19298.415

---

26: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 3.000) 0.840 19437.719
57c57
< 27: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 20.000) 1.000 2212.972

---

27: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 20.000) 1.000 2235.706
59c59
< 28: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 1.500) 0.771 25701.573

---

28: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 1.500) 0.772 25698.865
61c61
< 29: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 1.500) 0.860 22676.155

---

29: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 1.500) 0.861 23154.836
63c63
< 30: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 3.000) 0.916 16764.549

---

30: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 3.000) 0.913 16869.750

And the build time is 3791.798057794571s and 3794.1561596393585s. I think there is no significant different between them.