-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adds a new parameter, max_analyzer_offset, for the highlighter #3893
Conversation
I couldn't run the check on windows, it was failing even when I run it with the main branch I executed the line below on powershell.
Finally, how can I signed the commits? |
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
I see that tests are failing for multi cluster searches. However this is expected, as these other clusters did not support this feature. Is there an way to check te cluster version and only runs this when all clusters support it? |
For DCO, amend your PRs with |
Do you need to add something like this to your test specification (substituting the correct version of course)? |
…max_analyzer_offset. When this parameter is provided the highlight stops in its value. This prevents the highlighter to go beyond the index maxAnalyzedOffset. Signed-off-by: Hauck <joaoh14@gmail.com>
Signed-off-by: Hauck <joaoh14@gmail.com>
Signed-off-by: Hauck <joaoh14@gmail.com>
Signed-off-by: Hauck <joaoh14@gmail.com>
Signed-off-by: Hauck <joaoh14@gmail.com>
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
Signed-off-by: Hauck <joaoh14@gmail.com>
Gradle Check (Jenkins) Run Completed with:
|
Signed-off-by: Hauck <joaoh14@gmail.com>
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
Codecov Report
@@ Coverage Diff @@
## main #3893 +/- ##
============================================
+ Coverage 70.57% 70.66% +0.08%
- Complexity 56679 56795 +116
============================================
Files 4563 4573 +10
Lines 272755 273288 +533
Branches 40040 40084 +44
============================================
+ Hits 192505 193116 +611
+ Misses 64014 63905 -109
- Partials 16236 16267 +31
|
Signed-off-by: Hauck <joaoh14@gmail.com>
Gradle Check (Jenkins) Run Completed with:
|
server/src/main/java/org/opensearch/search/fetch/subphase/highlight/UnifiedHighlighter.java
Outdated
Show resolved
Hide resolved
Gradle Check (Jenkins) Run Completed with:
|
Do you think this can be back-ported to 2.x? |
Gradle Check (Jenkins) Run Completed with:
|
rest-api-spec/src/main/resources/rest-api-spec/test/search.highlight/30_max_analyzed_offset.yml
Outdated
Show resolved
Hide resolved
I think so. Since it's a new parameter it's fully backward compatible. @reta do you see any reason not to backport? |
Signed-off-by: Hauck <joaoh14@gmail.com>
Gradle Check (Jenkins) Run Completed with:
|
@andrross all yours to final review/merge |
* #3842 adds a new parameter to the highlighter, the max_analyzer_offset. When this parameter is provided the highlight stops in its value. This prevents the highlighter to go beyond the index maxAnalyzedOffset. Signed-off-by: Hauck <joaoh14@gmail.com> * Adds a test for the new parameter Signed-off-by: Hauck <joaoh14@gmail.com> * Fix the test add in the previous commit; Signed-off-by: Hauck <joaoh14@gmail.com> * This was checking against the wrong field Signed-off-by: Hauck <joaoh14@gmail.com> * Only runs the test for the correct version Signed-off-by: Hauck <joaoh14@gmail.com> * Skips the test in Elasticsearch as well; Signed-off-by: Hauck <joaoh14@gmail.com> * Remove elastic 3.0 to test Signed-off-by: Hauck <joaoh14@gmail.com> * Skips all versions Signed-off-by: Hauck <joaoh14@gmail.com> * Remove unnecessary fields as pointed by @reta Signed-off-by: Hauck <joaoh14@gmail.com> * Compute if fieldMaxAnalyzedIsNotValid in the constructor as suggest by @reta Signed-off-by: Hauck <joaoh14@gmail.com> * As discussed, it is better to throws different exceptions for when the fieldMaxAnalyzed is not valid and for when it is disabled; Signed-off-by: Hauck <joaoh14@gmail.com> * hint what to do to allow highlight of bigger documents Signed-off-by: Hauck <joaoh14@gmail.com> * Let the user define the new parameter globally for all fields highlighted Signed-off-by: Hauck <joaoh14@gmail.com> * Change the fieldMaxAnalyzedOffset Integer in order to use null when it is absent in highlight. This allows the error messages to much more precise, showing invalid for all negative numbers; Signed-off-by: Hauck <joaoh14@gmail.com> * Update javadocs and implements the stream methods for the new fields; Signed-off-by: Hauck <joaoh14@gmail.com> * builder.field do not accept null, so check before calling the method is necessary Signed-off-by: Hauck <joaoh14@gmail.com> * Only send and read the new fields if the version supports it Signed-off-by: Hauck <joaoh14@gmail.com> * the previous commit was checking the wrong field Signed-off-by: Hauck <joaoh14@gmail.com> * Check for version 3.0.0 instead of current version Signed-off-by: Hauck <joaoh14@gmail.com> * Update server/src/main/java/org/apache/lucene/search/uhighlight/CustomUnifiedHighlighter.java Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Hauck <joaoh14@gmail.com> * Execute the test after version 3.0.0 Signed-off-by: Hauck <joaoh14@gmail.com> Co-authored-by: Andriy Redko <drreta@gmail.com> (cherry picked from commit 931813f)
No, I don't see any reasons not to backport, it is good change I think. |
…earch-project#3893) * opensearch-project#3842 adds a new parameter to the highlighter, the max_analyzer_offset. When this parameter is provided the highlight stops in its value. This prevents the highlighter to go beyond the index maxAnalyzedOffset. Signed-off-by: Hauck <joaoh14@gmail.com> * Adds a test for the new parameter Signed-off-by: Hauck <joaoh14@gmail.com> * Fix the test add in the previous commit; Signed-off-by: Hauck <joaoh14@gmail.com> * This was checking against the wrong field Signed-off-by: Hauck <joaoh14@gmail.com> * Only runs the test for the correct version Signed-off-by: Hauck <joaoh14@gmail.com> * Skips the test in Elasticsearch as well; Signed-off-by: Hauck <joaoh14@gmail.com> * Remove elastic 3.0 to test Signed-off-by: Hauck <joaoh14@gmail.com> * Skips all versions Signed-off-by: Hauck <joaoh14@gmail.com> * Remove unnecessary fields as pointed by @reta Signed-off-by: Hauck <joaoh14@gmail.com> * Compute if fieldMaxAnalyzedIsNotValid in the constructor as suggest by @reta Signed-off-by: Hauck <joaoh14@gmail.com> * As discussed, it is better to throws different exceptions for when the fieldMaxAnalyzed is not valid and for when it is disabled; Signed-off-by: Hauck <joaoh14@gmail.com> * hint what to do to allow highlight of bigger documents Signed-off-by: Hauck <joaoh14@gmail.com> * Let the user define the new parameter globally for all fields highlighted Signed-off-by: Hauck <joaoh14@gmail.com> * Change the fieldMaxAnalyzedOffset Integer in order to use null when it is absent in highlight. This allows the error messages to much more precise, showing invalid for all negative numbers; Signed-off-by: Hauck <joaoh14@gmail.com> * Update javadocs and implements the stream methods for the new fields; Signed-off-by: Hauck <joaoh14@gmail.com> * builder.field do not accept null, so check before calling the method is necessary Signed-off-by: Hauck <joaoh14@gmail.com> * Only send and read the new fields if the version supports it Signed-off-by: Hauck <joaoh14@gmail.com> * the previous commit was checking the wrong field Signed-off-by: Hauck <joaoh14@gmail.com> * Check for version 3.0.0 instead of current version Signed-off-by: Hauck <joaoh14@gmail.com> * Update server/src/main/java/org/apache/lucene/search/uhighlight/CustomUnifiedHighlighter.java Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Hauck <joaoh14@gmail.com> * Execute the test after version 3.0.0 Signed-off-by: Hauck <joaoh14@gmail.com> Co-authored-by: Andriy Redko <drreta@gmail.com> (cherry picked from commit 931813f) Signed-off-by: Hauck <joaoh14@gmail.com>
…hlighter (#4031) * Adds a new parameter, max_analyzer_offset, for the highlighter (#3893) * #3842 adds a new parameter to the highlighter, the max_analyzer_offset. When this parameter is provided the highlight stops in its value. This prevents the highlighter to go beyond the index maxAnalyzedOffset. Signed-off-by: Hauck <joaoh14@gmail.com> * Adds a test for the new parameter Signed-off-by: Hauck <joaoh14@gmail.com> * Fix the test add in the previous commit; Signed-off-by: Hauck <joaoh14@gmail.com> * This was checking against the wrong field Signed-off-by: Hauck <joaoh14@gmail.com> * Only runs the test for the correct version Signed-off-by: Hauck <joaoh14@gmail.com> * Skips the test in Elasticsearch as well; Signed-off-by: Hauck <joaoh14@gmail.com> * Remove elastic 3.0 to test Signed-off-by: Hauck <joaoh14@gmail.com> * Skips all versions Signed-off-by: Hauck <joaoh14@gmail.com> * Remove unnecessary fields as pointed by @reta Signed-off-by: Hauck <joaoh14@gmail.com> * Compute if fieldMaxAnalyzedIsNotValid in the constructor as suggest by @reta Signed-off-by: Hauck <joaoh14@gmail.com> * As discussed, it is better to throws different exceptions for when the fieldMaxAnalyzed is not valid and for when it is disabled; Signed-off-by: Hauck <joaoh14@gmail.com> * hint what to do to allow highlight of bigger documents Signed-off-by: Hauck <joaoh14@gmail.com> * Let the user define the new parameter globally for all fields highlighted Signed-off-by: Hauck <joaoh14@gmail.com> * Change the fieldMaxAnalyzedOffset Integer in order to use null when it is absent in highlight. This allows the error messages to much more precise, showing invalid for all negative numbers; Signed-off-by: Hauck <joaoh14@gmail.com> * Update javadocs and implements the stream methods for the new fields; Signed-off-by: Hauck <joaoh14@gmail.com> * builder.field do not accept null, so check before calling the method is necessary Signed-off-by: Hauck <joaoh14@gmail.com> * Only send and read the new fields if the version supports it Signed-off-by: Hauck <joaoh14@gmail.com> * the previous commit was checking the wrong field Signed-off-by: Hauck <joaoh14@gmail.com> * Check for version 3.0.0 instead of current version Signed-off-by: Hauck <joaoh14@gmail.com> * Update server/src/main/java/org/apache/lucene/search/uhighlight/CustomUnifiedHighlighter.java Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Hauck <joaoh14@gmail.com> * Execute the test after version 3.0.0 Signed-off-by: Hauck <joaoh14@gmail.com> Co-authored-by: Andriy Redko <drreta@gmail.com> (cherry picked from commit 931813f) Signed-off-by: Hauck <joaoh14@gmail.com> * Fix the version tests Signed-off-by: Hauck <joaoh14@gmail.com> Co-authored-by: Andriy Redko <drreta@gmail.com>
* #3842 adds a new parameter to the highlighter, the max_analyzer_offset. When this parameter is provided the highlight stops in its value. This prevents the highlighter to go beyond the index maxAnalyzedOffset. Signed-off-by: Hauck <joaoh14@gmail.com> * Adds a test for the new parameter Signed-off-by: Hauck <joaoh14@gmail.com> * Fix the test add in the previous commit; Signed-off-by: Hauck <joaoh14@gmail.com> * This was checking against the wrong field Signed-off-by: Hauck <joaoh14@gmail.com> * Only runs the test for the correct version Signed-off-by: Hauck <joaoh14@gmail.com> * Skips the test in Elasticsearch as well; Signed-off-by: Hauck <joaoh14@gmail.com> * Remove elastic 3.0 to test Signed-off-by: Hauck <joaoh14@gmail.com> * Skips all versions Signed-off-by: Hauck <joaoh14@gmail.com> * Remove unnecessary fields as pointed by @reta Signed-off-by: Hauck <joaoh14@gmail.com> * Compute if fieldMaxAnalyzedIsNotValid in the constructor as suggest by @reta Signed-off-by: Hauck <joaoh14@gmail.com> * As discussed, it is better to throws different exceptions for when the fieldMaxAnalyzed is not valid and for when it is disabled; Signed-off-by: Hauck <joaoh14@gmail.com> * hint what to do to allow highlight of bigger documents Signed-off-by: Hauck <joaoh14@gmail.com> * Let the user define the new parameter globally for all fields highlighted Signed-off-by: Hauck <joaoh14@gmail.com> * Change the fieldMaxAnalyzedOffset Integer in order to use null when it is absent in highlight. This allows the error messages to much more precise, showing invalid for all negative numbers; Signed-off-by: Hauck <joaoh14@gmail.com> * Update javadocs and implements the stream methods for the new fields; Signed-off-by: Hauck <joaoh14@gmail.com> * builder.field do not accept null, so check before calling the method is necessary Signed-off-by: Hauck <joaoh14@gmail.com> * Only send and read the new fields if the version supports it Signed-off-by: Hauck <joaoh14@gmail.com> * the previous commit was checking the wrong field Signed-off-by: Hauck <joaoh14@gmail.com> * Check for version 3.0.0 instead of current version Signed-off-by: Hauck <joaoh14@gmail.com> * Update server/src/main/java/org/apache/lucene/search/uhighlight/CustomUnifiedHighlighter.java Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Hauck <joaoh14@gmail.com> * Execute the test after version 3.0.0 Signed-off-by: Hauck <joaoh14@gmail.com> Co-authored-by: Andriy Redko <drreta@gmail.com> (cherry picked from commit 931813f)
* #3842 adds a new parameter to the highlighter, the max_analyzer_offset. When this parameter is provided the highlight stops in its value. This prevents the highlighter to go beyond the index maxAnalyzedOffset. Signed-off-by: Hauck <joaoh14@gmail.com> * Adds a test for the new parameter Signed-off-by: Hauck <joaoh14@gmail.com> * Fix the test add in the previous commit; Signed-off-by: Hauck <joaoh14@gmail.com> * This was checking against the wrong field Signed-off-by: Hauck <joaoh14@gmail.com> * Only runs the test for the correct version Signed-off-by: Hauck <joaoh14@gmail.com> * Skips the test in Elasticsearch as well; Signed-off-by: Hauck <joaoh14@gmail.com> * Remove elastic 3.0 to test Signed-off-by: Hauck <joaoh14@gmail.com> * Skips all versions Signed-off-by: Hauck <joaoh14@gmail.com> * Remove unnecessary fields as pointed by @reta Signed-off-by: Hauck <joaoh14@gmail.com> * Compute if fieldMaxAnalyzedIsNotValid in the constructor as suggest by @reta Signed-off-by: Hauck <joaoh14@gmail.com> * As discussed, it is better to throws different exceptions for when the fieldMaxAnalyzed is not valid and for when it is disabled; Signed-off-by: Hauck <joaoh14@gmail.com> * hint what to do to allow highlight of bigger documents Signed-off-by: Hauck <joaoh14@gmail.com> * Let the user define the new parameter globally for all fields highlighted Signed-off-by: Hauck <joaoh14@gmail.com> * Change the fieldMaxAnalyzedOffset Integer in order to use null when it is absent in highlight. This allows the error messages to much more precise, showing invalid for all negative numbers; Signed-off-by: Hauck <joaoh14@gmail.com> * Update javadocs and implements the stream methods for the new fields; Signed-off-by: Hauck <joaoh14@gmail.com> * builder.field do not accept null, so check before calling the method is necessary Signed-off-by: Hauck <joaoh14@gmail.com> * Only send and read the new fields if the version supports it Signed-off-by: Hauck <joaoh14@gmail.com> * the previous commit was checking the wrong field Signed-off-by: Hauck <joaoh14@gmail.com> * Check for version 3.0.0 instead of current version Signed-off-by: Hauck <joaoh14@gmail.com> * Update server/src/main/java/org/apache/lucene/search/uhighlight/CustomUnifiedHighlighter.java Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Hauck <joaoh14@gmail.com> * Execute the test after version 3.0.0 Signed-off-by: Hauck <joaoh14@gmail.com> Co-authored-by: Andriy Redko <drreta@gmail.com> (cherry picked from commit 931813f)
May I ask if this fix was introduced in OpenSearch 2.5? Thanks in advance. |
As per issue labels, it should be available from |
Unfortunately I'm having this issue on both clusters of OpenSearch 2.5 and 2.7 on AWS, this is the error message that appears on Discover section when using whatever query:
|
In your search query, are you passing the option "max_analyzer_offset": 100000? |
Stupid question perhaps, but do I understand correctly that this is a setting that needs to be set per-query? How do we ensure this value is set when executing queries from OpenSearch Dasbhoards? |
Description
Adds a new parameter, max_analyzer_offset, for the highlighter. This new parameter can be used to avoid aborting when highlighting large fields, or just to speedup the highlight doing it only on the beginning of the field.
Issues Resolved
This pull request closes #3842
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.