[FEA] Product quantization preprocessing API

[cjnolet]

We need a separate product quantization API that is decoupled from IVF but can still be composed into IVF.

Ideally this API would follow FAISS or Scikit-learn'a transformer estimators.

[QDXG-CXK]

Hi! I'm really interested in helping to implement this feature.

From what I understand, the request involves:

1. Extracting the product quantization code from ivf-pq and exposing public API.
2. Redesigning the ivf-pq to use the PQ API, eliminating duplicated code.
3. [Possibly in the future] Creating a base class for ivf that can be flexibly combined with SQ, PQ, BinaryQ and other quantization methods. (related to issues [FEA] Scalar quantization API #106 and [FEA] Binary quantization API #139 )

Is there a concrete plan for this yet? I noticed that some discussion has taken place in #211 ,but I didn't find a clear conclusion there. As mentioned in this comment and this one, the PQ API should at least cover training, encoding and searching. It seems like CuVS prefers a stateless style rather than holding trained quantizer in a index like FAISS does.

Looking forward to collaborating on this!

[QDXG-CXK]

Hi Maintainers! I'm still very interested in contributing to this feature and would appreciate any feedback.

[cjnolet]

Hi @QDXG-CXK,

My sincerest apologies for the delayed reply. It has taken us quite awhile to think through and plan for what an API like this might look like. We've also opted to start with binary and scalar quantization APIs first so that we can establish a preprocessing namespace.

We have also been reaching out to the community to learn more about the use-cases and why users would be interested in such an API. it turns out, the encoding of the vectors is the most important piece. For this reason we propose having a struct/simple c++ object that can hold the codebooks, centroids, rotation matrix, etc... and having a function to train the quantizer (which sets the relevant state on the object) and then a transform() function which accepts the trained object and encodes a bunch of vectors into the PQ space.

[QDXG-CXK]

Thanks @cjnolet for the explanation!

I noticed that SQ and BQ have been merged into the main branch and can serve as useful references. I'd like to start working on this immediately.

It seems only train() and transform() are the focus at this stage, while related computations (like AQD or SQD) are still under planning. Please let me know if there are any updates on the computational aspects - I'd be happy to contribute or provide feedback.