Add support for PQ preprocessing API #1278
Conversation
Signed-off-by: Mickael Ide <mide@nvidia.com>
github-project-automation
bot
moved this to Todo
in Vector Search, ML, & Data Mining Release Board
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
lowener
added
feature request
| @@ -0,0 +1,164 @@ | |||
| /* | |||
| * Copyright (c) 2024-2025, NVIDIA CORPORATION. | |||
There was a problem hiding this comment.
Please only include the current year in new files
Signed-off-by: Mickael Ide <mide@nvidia.com>
cjnolet
moved this from Todo
to In Progress
in Vector Search, ML, & Data Mining Release Board
| pq_params.add_data_on_build = false; | ||
| pq_params.max_train_points_per_pq_code = params.max_train_points_per_pq_code; | ||
|
|
||
| auto pq_index = cuvs::neighbors::ivf_pq::build(res, pq_params, dataset); |
There was a problem hiding this comment.
I've tried using ivf build + pq-centeres api to get the generated centroids as part of generating pq for diskann. It appears I get significant lower recall values. I think it because the pq strategy used by ivf has 2 quantizers and also ivf uses balanced kmeans which can lower the recall.
There was a problem hiding this comment.
Based on my understanding, the current VPQ also has two quantizers. @lowener correct me if I am missing something, but based on what I had seen in DiskANN, the raw vectors are PQ quantized directly, there is no coarse quantizer (VQ centroids).
Signed-off-by: Mickael Ide <mide@nvidia.com>
Signed-off-by: Mickael Ide <mide@nvidia.com>
Signed-off-by: Mickael Ide <mide@nvidia.com>
Signed-off-by: Mickael Ide <mide@nvidia.com>
Signed-off-by: Mickael Ide <mide@nvidia.com>
Signed-off-by: Mickael Ide <mide@nvidia.com>
Signed-off-by: Mickael Ide <mide@nvidia.com>
Signed-off-by: Mickael Ide <mide@nvidia.com>
Signed-off-by: Mickael Ide <mide@nvidia.com>
Signed-off-by: Mickael Ide <mide@nvidia.com>
Signed-off-by: Mickael Ide <mide@nvidia.com>
Related issue: #107
This PR adds support for a PQ preprocessing API. It gives access to
train()andtransform()function that can be used to transform a dataset into PQ codes. It is re-using the VPQ functions from CAGRA-Q.