Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Backport 2.x] Do not evaluate shard_size and shard_min_doc_count at segment slice level #9211

Merged
merged 1 commit into from
Aug 10, 2023

Conversation

jed326
Copy link
Collaborator

@jed326 jed326 commented Aug 9, 2023

Description

Backport for #8860 #9085

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions
Copy link
Contributor

github-actions bot commented Aug 9, 2023

Gradle Check (Jenkins) Run Completed with:

@opensearch-trigger-bot
Copy link
Contributor

Compatibility status:

> Task :checkCompatibility
Checking compatibility for: https://github.com/opensearch-project/opensearch-oci-object-storage.git with ref: 2.x
Skipping compatibility check for https://github.com/opensearch-project/opensearch-oci-object-storage.git
Incompatible components: [https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git]
Components skipped due to git failures: [https://github.com/opensearch-project/opensearch-oci-object-storage.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/ml-commons.git]

BUILD SUCCESSFUL in 29m 36s

@jed326 jed326 changed the title Do not evaluate shard_size and shard_min_doc_count at segment slice level [Backport 2.x] Do not evaluate shard_size and shard_min_doc_count at segment slice level Aug 9, 2023
@opensearch-trigger-bot
Copy link
Contributor

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer.git]
Components skipped due to git failures: [https://github.com/opensearch-project/opensearch-oci-object-storage.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/ml-commons.git]

BUILD SUCCESSFUL in 33m 9s

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@jed326
Copy link
Collaborator Author

jed326 commented Aug 10, 2023

@reta @sohami here is the backport PR for #8860.

Most recent gradle check failed with known flakey tests.

…evel (opensearch-project#9085)

* Use BucketCountThresholds in InternalTerms and InternalAggregations and do not apply shard level thresholds at slice level for Concurrent Segment Search

Signed-off-by: Jay Deng <jayd0104@gmail.com>

* Addressing comments

Signed-off-by: Jay Deng <jayd0104@gmail.com>

* Re-introduce shardSize member to InternalMultiTerms and InternalMappedTerms

Signed-off-by: Jay Deng <jayd0104@gmail.com>

* Introduce LocalBucketCountThresholds for local size and min_doc_count values

Signed-off-by: Jay Deng <jayd0104@gmail.com>

---------

Signed-off-by: Jay Deng <jayd0104@gmail.com>
@opensearch-trigger-bot
Copy link
Contributor

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/performance-analyzer.git]
Components skipped due to git failures: [https://github.com/opensearch-project/opensearch-oci-object-storage.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

BUILD SUCCESSFUL in 33m 13s

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.search.SearchWeightedRoutingIT.testStrictWeightedRoutingWithCustomString_FailOpenEnabled

@codecov
Copy link

codecov bot commented Aug 10, 2023

Codecov Report

Merging #9211 (993fdf1) into 2.x (ed3cfce) will decrease coverage by 0.05%.
The diff coverage is 74.69%.

@@             Coverage Diff              @@
##                2.x    #9211      +/-   ##
============================================
- Coverage     70.85%   70.81%   -0.05%     
+ Complexity    57522    57460      -62     
============================================
  Files          4756     4757       +1     
  Lines        271788   271790       +2     
  Branches      40086    40090       +4     
============================================
- Hits         192588   192476     -112     
- Misses        62703    62859     +156     
+ Partials      16497    16455      -42     
Files Changed Coverage Δ
.../search/aggregations/bucket/terms/DoubleTerms.java 75.00% <ø> (-1.93%) ⬇️
...ch/search/aggregations/bucket/terms/LongTerms.java 67.14% <ø> (ø)
.../search/aggregations/bucket/terms/StringTerms.java 85.71% <ø> (ø)
...h/aggregations/bucket/terms/UnsignedLongTerms.java 48.57% <ø> (ø)
...earch/search/aggregations/InternalAggregation.java 81.81% <33.33%> (+0.64%) ⬆️
...arch/aggregations/AggregationCollectorManager.java 78.57% <50.00%> (-4.77%) ⬇️
...ket/terms/GlobalOrdinalsStringTermsAggregator.java 83.45% <50.00%> (+0.17%) ⬆️
...rch/aggregations/bucket/terms/TermsAggregator.java 78.88% <50.00%> (-1.35%) ⬇️
...earch/aggregations/bucket/terms/InternalTerms.java 87.50% <61.53%> (-1.35%) ⬇️
...gations/bucket/terms/UnmappedSignificantTerms.java 33.33% <66.66%> (ø)
... and 17 more

... and 443 files with indirect coverage changes

@reta reta merged commit f28c80b into opensearch-project:2.x Aug 10, 2023
11 checks passed
@jed326 jed326 deleted the backport-9085-to-2.x branch September 13, 2023 19:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants