Skip to content

[BUG] neighbors::refine can potentially undo pre-filtering #1796

Open
@achirkin

Description

@achirkin

When fixing #1793 , I realized that the refinement operation can undo the filtering if the number of valid top-k-results is less than k.

So far, we mark invalid results with infinite distance in IVF-Flat and IVF-PQ. But the neighbors::refine can "refine" the distance back to a small value!
It's not a problem with the fused IVF-PQ similarity kernel, because the invalid results do not make it to the outputs; but the non-fused kernel only marks the bad values with inftinite distance and relies on matrix::select_k to filter out invalid results.
matrix::select_k should normally treat infinity as normal values, so the invalid key-value pairs may make it all the way to the refine operation. Moreover, we may get infinities due to low-precision float overflows; refining these is an expected behavior.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    • Status

      Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions