Add calculate-recall parameter to vector search and skip calculating recall if number clients > cpu cores #626
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Finn Roblin <finnrobl@amazon.com>
Signed-off-by: Finn Roblin <finnrobl@amazon.com>
finnroblin
requested review from
IanHoang,
gkamat,
beaioun,
cgchinmay,
rishabh6788 and
VijayanB
as code owners
Signed-off-by: Finn Roblin <finnrobl@amazon.com>
IanHoang
reviewed
Sep 18, 2024
| }) | ||
|
|
||
| def _get_should_calculate_recall(params: dict) -> bool: | ||
| # set in global config (benchmark.ini) and passed by AsyncExecutor |
There was a problem hiding this comment.
@finnroblin Could you elaborate on this comment? What changes should the user be making to the benchmark.ini to enable this?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
When the number of search clients is greater than the number of cpu cores recall drops unexpectedly. @VijayanB is investigating the cause, but this PR throws a warning if num_clients > the cpu cores available and skips calculating the recall. It also adds a parameter
calculate-recallto specify whether or not to calculate recall (eithercalculate-recall: trueorcalculate-recall: false).Testing
Unit tests.
Also tested on mac with 12 cores. Skips recall calculation if num_clients > 12, calculates recall otherwise.
calculate-recallparameter works as expected. Will add unit tests in a second PR by EOD.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.