-
Notifications
You must be signed in to change notification settings - Fork 114
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix few bugs on binary index with Faiss HNSW #1850
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -190,6 +190,19 @@ public static ValidationException validateKnnField( | |
return exception; | ||
} | ||
|
||
String vectorDataType = (String) fieldMap.get(VECTOR_DATA_TYPE_FIELD); | ||
if (VectorDataType.BINARY.toString().equalsIgnoreCase(vectorDataType)) { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. we should convert the datatype to enum and then use There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Because vectorDataType can be null, I choose this approach. Why should we convert it to the data type? |
||
exception.addValidationError( | ||
String.format( | ||
Locale.ROOT, | ||
"Field \"%s\" is of data type %s. Only FLOAT or BYTE is supported.", | ||
field, | ||
VectorDataType.BINARY | ||
) | ||
); | ||
return exception; | ||
} | ||
|
||
// Return if dimension does not need to be checked | ||
if (expectedDimension < 0) { | ||
return null; | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -476,8 +476,15 @@ private boolean canDoExactSearch(final int filterIdsCount) { | |
if (isExactSearchThresholdSettingSet(filterThresholdValue)) { | ||
return filterThresholdValue >= filterIdsCount; | ||
} | ||
|
||
// if no setting is set, then use the default max distance computation value to see if we can do exact search. | ||
return KNNConstants.MAX_DISTANCE_COMPUTATIONS >= filterIdsCount * knnQuery.getQueryVector().length; | ||
/** | ||
* TODO we can have a different MAX_DISTANCE_COMPUTATIONS for binary index as computation cost for binary index | ||
* is cheaper than computation cost for non binary vector | ||
*/ | ||
return KNNConstants.MAX_DISTANCE_COMPUTATIONS >= filterIdsCount * (knnQuery.getVectorDataType() == VectorDataType.FLOAT | ||
? knnQuery.getQueryVector().length | ||
: knnQuery.getByteQueryVector().length); | ||
heemin32 marked this conversation as resolved.
Show resolved
Hide resolved
Comment on lines
+486
to
+487
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't like the fact that we have two vectors. This will create a lot of branching in the code. We should have gone with generics or something with a different type of query. Something similar to lucene. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We are already aware of it and captured it in #1810 |
||
} | ||
|
||
/** | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can
fieldMap.get(VECTOR_DATA_TYPE_FIELD)
be null?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It can be null.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if it can be null then the type casting to string will cause a NPE. Please handle this gracefully.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It won't cause a NPE.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why?? casting a null to string cause NPE. I am missing something here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tested it and it didn't cause NPE. String is of Object and String can be null.