Add min score linear retriever #129359
Merged
Add min score linear retriever #129359
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Small changes in BlobContainer interface and wrapper. Relates ES-11815
…129054) The reason the test fails is that operations contained _seq_no field with different doc value types (with no skippers and with skippers) and this isn't allowed, since field types need to be consistent in a Lucene index. The initial operations were generated not knowing about the fact the index mode was set to logsdb or time_series. Causing the operations to not have doc value skippers. However when replaying the operations via following engine, the operations did have doc value skippers. The fix is to set `index.seq_no.index_options` to `points_and_doc_values`, so that the initial operations are indexed without doc value skippers. This test doesn't gain anything from storing seqno with doc value skippers, so there is no loss of testing coverage. Closes elastic#128541
This ensures we package an aggregation zip with all artifacts we want to publish to maven central as part of a release. Running zipAggregation will produce a zip file in the build/nmcp/zip folder. The content of this zip is meant to match the maven artifacts we have currently declared as dra maven artifacts.
Runs a sanity check after loading a block of values. Previously we were doing a quick check if assertions were enabled. Now we do two quick checks all the time. Better - we attach information about how a block was loaded when there's a problem. Relates to elastic#128959
The functionality in `PhaseCacheManagement` was already project-aware, but these tests were still using deprecated methods.
This adds some testing tools for verifying vector recall and latency
directly without having to spin up an entire ES node and running a rally
track.
Its pretty barebones and takes inspiration from lucene-util, but I
wanted access to our own formats and tooling to make our lives easier.
Here is an example config file. This will build the initial index, run
queries at num_candidates: 50, then again at num_candidates 100 (without
reindexing, and re-using the cached nearest neighbors).
```
[{
"doc_vectors" : "path",
"query_vectors" : "path",
"num_docs" : 10000,
"num_queries" : 10,
"index_type" : "hnsw",
"num_candidates" : 50,
"k" : 10,
"hnsw_m" : 16,
"hnsw_ef_construction" : 200,
"index_threads" : 4,
"reindex" : true,
"force_merge" : false,
"vector_space" : "maximum_inner_product",
"dimensions" : 768
},
{
"doc_vectors" : "path",
"query_vectors" : "path",
"num_docs" : 10000,
"num_queries" : 10,
"index_type" : "hnsw",
"num_candidates" : 100,
"k" : 10,
"hnsw_m" : 16,
"hnsw_ef_construction" : 200,
"vector_space" : "maximum_inner_product",
"dimensions" : 768
}
]
```
To execute:
```
./gradlew :qa:vector:checkVec --args="/Path/to/knn_tester_config.json"
```
Calling `./gradlew :qa:vector:checkVecHelp` gives some guidance on how
to use it, additionally providing a way to run it via java directly
(useful to bypass gradlew guff).
Add a spec test of `LOOKUP JOIN` against a time series index.
This is part of an iterative process to make ILM project-aware.
…t {lookup-join.LookupJoinOnTimeSeriesIndex ASYNC} elastic#129078
…9076) The `ClusterState` parameter of the `asyncPredicate` is not used anywhere.
…t {lookup-join.LookupJoinOnTimeSeriesIndex SYNC} elastic#129082
…est {p0=upgraded_cluster/70_ilm/Test Lifecycle Still There And Indices Are Still Managed} elastic#129097
…est {p0=upgraded_cluster/90_ml_data_frame_analytics_crud/Get mixed cluster outlier_detection job} elastic#129098
…ollowedWithEnvironmentVariableFiles elastic#128867
…27613) This PR introduces 3 new settings: indices.merge.disk.check_interval, indices.merge.disk.watermark.high, and indices.merge.disk.watermark.high.max_headroom that control if the threadpool merge executor starts executing new merges when the disk space is getting low. The intent of this change is to avoid the situation where in-progress merges exhaust the available disk space on the node's local filesystem. To this end, the thread pool merge executor periodically monitors the available disk space, as well as the current disk space estimates required by all in-progress (currently running) merges on the node, and will NOT schedule any new merges if the disk space is getting low (by default below the 5% limit of the total disk space, or 100 GB, whichever is smaller (same as the disk allocation flood stage level)).
…tic#128735) This PR introduces a new include_vectors option to the _source retrieval context. When set to false, vectors are excluded from the returned _source. This is especially efficient when used with synthetic source, as it avoids loading vector fields entirely. By default, vectors remain included unless explicitly excluded.
…kSpaceTests testAvailableDiskSpaceMonitorWhenFileSystemStatErrors elastic#129149
…ic#129033) * Add transport version for ML inference Mistral chat completion * Add changelog for Mistral Chat Completion version fix * Revert "Add changelog for Mistral Chat Completion version fix" This reverts commit 7a57416.
All we care about is if reindex is true or false. We shouldn't worry about force merge. Because if reindex is true, we will create the directory, if its false, we won't.
…kSpaceTests testUnavailableBudgetBlocksNewMergeTasksFromStartingExecution elastic#129148
* Google Vertex AI completion model, response entity and tests * Fixed GoogleVertexAiServiceTest for Service configuration * Changelog * Removed downcasting and using `moveToFirstToken` * Create GoogleVertexAiChatCompletionResponseHandler for streaming and non streaming responses * Added unit tests * PR feedback * Removed googlevertexaicompletion model. Using just GoogleVertexAiChatCompletionModel for completion and chat completion * Renamed uri -> nonStreamingUri. Added streamingUri and getters in GoogleVertexAiChatCompletionModel * Moved rateLimitGroupHashing to subclasses of GoogleVertexAiModel * Fixed rate limit has of GoogleVertexAiRerankModel and refactored uri for GoogleVertexAiUnifiedChatCompletionRequest --------- Co-authored-by: lhoet-google <lhoet@google.com> Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com>
mridula-s109
added
>enhancement
auto-backport
|
Hi @mridula-s109, I've created a changelog YAML for you. |
There was a problem hiding this comment.
Pull Request Overview
This PR introduces support for a new min_score parameter in the linear retriever to filter out low-relevance documents based on a configurable threshold. The changes include updates to YAML tests to cover different min_score scenarios, adjustments to the LinearRetrieverBuilder to integrate min_score with proper error validation, and updates to feature lists and changelog documentation.
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| x-pack/plugin/rank-rrf/src/yamlRestTest/resources/rest-api-spec/test/linear/10_linear_retriever.yml | Adds YAML tests to validate min_score functionality across various scenarios |
| x-pack/plugin/rank-rrf/src/main/java/org/elasticsearch/xpack/rank/linear/LinearRetrieverBuilder.java | Introduces the min_score parameter with validation and integrates it into result filtering |
| x-pack/plugin/rank-rrf/src/main/java/org/elasticsearch/xpack/rank/RankRRFFeatures.java | Updates test feature set to include the new min_score node feature |
| docs/changelog/129359.yaml | Documents the changelog entry for the min_score enhancement |
Comments suppressed due to low confidence (2)
x-pack/plugin/rank-rrf/src/main/java/org/elasticsearch/xpack/rank/linear/LinearRetrieverBuilder.java:141
- Consider adding a test case to validate that passing a negative minScore throws an IllegalArgumentException.
if (minScore != null && minScore < 0) {
x-pack/plugin/rank-rrf/src/main/java/org/elasticsearch/xpack/rank/linear/LinearRetrieverBuilder.java:130
- [nitpick] Consider adding Javadoc comments for the new minScore parameter to clarify its purpose and usage in the constructor.
public LinearRetrieverBuilder(
|
Pinging @elastic/es-search-relevance (Team:Search Relevance) |
kderusso
approved these changes
Jun 12, 2025
afoucret
previously requested changes
Jun 12, 2025
...lugin/rank-rrf/src/main/java/org/elasticsearch/xpack/rank/linear/LinearRetrieverBuilder.java
Show resolved
Hide resolved
afoucret
dismissed
their stale review
Don't want to block the merge process, even if recommend to have a look at capabilities instead of node features.
💔 Backport failedThe backport operation could not be completed due to the following error: You can use sqren/backport to manually backport by running |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces support for the min_score parameter in the linear retriever. The min_score filter ensures that only documents with a relevance score above a specified threshold are retrieved, improving result quality and filtering out low-relevance matches.
Added min_score support in the linear retriever.
Updated the retrieval logic to apply the min_score threshold.
Based on an example from the Elasticsearch Guide (8.18), extended it to demonstrate min_score usage.
Related PR - Fixing minscore propagation issue in this PR made it easier to incorporate it in linear.