Load knn vectors format with mmapfs #78724
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Before the format used niofs. The current knn vectors implementation is based on the HNSW algorithm, which is designed for the case where the graph and vectors are be held in memory. Switching to mmapfs from niofs made a huge difference in ANN benchmarks, speeding up some searches over 3x.
jtibshirani
added
:Search/Search
|
Example benchmark results on nio (NIOFSDirectory) mmap (MMapDirectory) |
|
Pinging @elastic/es-search (Team:Search) |
mayya-sharipova
approved these changes
Oct 6, 2021
There was a problem hiding this comment.
@jtibshirani Oh very nice, indeed impressive gains, especially with higher recall, more than 3x times! Great notice, Julie!.
wjp719
added a commit
to wjp719/elasticsearch
that referenced
this pull request
Oct 7, 2021
…' into feature/data_stream_support_routing * wjp/feature/data_stream_support_routing: (44 commits) Revert "Adjust /_cat/templates not to request all metadata (elastic#78812)" Allow indices lookup to be built lazily (elastic#78745) [DOCS] Document default security in alpha2 (elastic#78227) Add cluster applier stats (elastic#77552) Fix failing URLDecodeProcessorTests::testProcessor test (elastic#78690) Upgrade to lucene snapshot ba75dc5e6bf (elastic#78817) Adjust /_cat/templates not to request all metadata (elastic#78812) Simplify build plugin license handling (elastic#77009) Fix SearchableSnapshotsBlobStoreCacheIntegTests.testBlobStoreCache (elastic#78616) Improve Docker image caching and testing (elastic#78552) Load knn vectors format with mmapfs (elastic#78724) Fix date math zone test to use negative minutes (elastic#78796) Changing name of shards field in node/stats api to shard_stats (elastic#78531) [DOCS] Fix system index refs in restore tutorial (elastic#78582) Add previously removed settings back for 8.0 (elastic#78784) TSDB: Fix template name in test Add a system property to forcibly format everything (elastic#78768) Revert "Adding config so that some tests will break if over-the-wire encryption fails (elastic#78409)" (elastic#78787) Must date math test failure Adding config so that some tests will break if over-the-wire encryption fails (elastic#78409) ...
jtibshirani
added
:Search Relevance/Vectors
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before the format used niofs. The current knn vectors implementation is based on
the HNSW algorithm, which is designed for the case where the graph and vectors
are be held in memory. Switching to mmapfs from niofs made a big difference in
ANN benchmarks, speeding up some searches over 3x.
Relates to #78473.