[Feature]: Support HNSW SQ

[xiaofan-luan]

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe.

SQ8 and PQ are widely used in ANN search. If you want to understand more about quantization, Faiss is probably one of the best code bases to explore.

HNSW is the fastest index in the open source world, so why not make it work together with SQ and PQ to accelerate it further?

Let me know if anyone is interested and we can offer more help on it

Describe the solution you'd like.

No response

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response

[noble-8]

Hi @xiaofan-luan
I am interested in contributing!
and would like to know how to help! let me know how to get started.

[xiaofan-luan]

cool man! I though @liliu-z could offer you some help

[xiaofan-luan]

you have any experience on cpp and some any idea about HSNW algorithm yet?

[noble-8]

Hi
I had taken a cpp course in college, I have primarily worked as a Java developer ~3 ish years, so i feel that I can onboard quickly. I feel that I am comfortable working on cpp.
I am really new to this algorithm but getting up to speed. I am going through this documentation here:https://www.pinecone.io/learn/hnsw/
feel free to point me to other resources.
I hope this is not a dealbreaker!

[xiaofan-luan]

1. you can start from read code from HNSW in knowhere https://github.com/milvus-io/knowhere/tree/c05c8767f43eaa855c13654804d0bea9cc42c7de/src/index/hnsw
2. Once you understand hnsw, then the next step would be under stand how to do PQ, SQ. Faiss document will give your general ideas and milvus has all faiss code you can utilize https://github.com/facebookresearch/faiss/wiki.
3. Add index parameters for milvus to support PQ, SQ, which will be a trivial task

[liliu-z]

Hi @noble-8 this is on our roadmap and please feel free to make a PR for https://github.com/milvus-io/knowhere . I suggest we can start from SQ8 which is easier to implement. And more than welcome to open another issue in Knowhere for further detailed communication.

[liliu-z]

/assign @liliu-z

[noble-8]

Sounds good. Will do!