Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize innerhits query performance #16937

Merged
merged 1 commit into from
Jan 22, 2025
Merged

Optimize innerhits query performance #16937

merged 1 commit into from
Jan 22, 2025

Conversation

kkewwei
Copy link
Contributor

@kkewwei kkewwei commented Jan 3, 2025 •
edited
Loading

Description

During I experimentation with concurrent execution in the innerhit phase, I discovered another logic that requires optimization.

It is known that TermQuery is never cached in QueryCache, but innerhit phase extensively utilizes TermQuery, resulting in inefficient(https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/search/UsageTrackingQueryCachingPolicy.java#L57). The following improvements can be made:

Related Issues

Resolves #16878 (comment)

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions github-actions bot added enhancement Enhancement or improvement to existing feature or request Search:Performance labels Jan 3, 2025
@kkewwei kkewwei force-pushed the optimize_innerhits_query branch from b79e7d9 to a1c5a45 Compare January 3, 2025 12:25
@kkewwei
Copy link
Contributor Author

kkewwei commented Jan 3, 2025

{"run-benchmark-test": "id_5"}

@github-actions github-actions bot mentioned this pull request Jan 3, 2025
Closed
@kkewwei
Copy link
Contributor Author

kkewwei commented Jan 3, 2025

I found that InnerHitsIT integration test can fully covered my changes, and it seems difficult to add unit test.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 3, 2025

❌ Gradle check result for a1c5a45: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@kkewwei
Copy link
Contributor Author

kkewwei commented Jan 13, 2025 •
edited
Loading

@kkewwei Looking at the perf numbers from #16937 (comment) it seems the nested-date-histo query is seeing some regression, could you double check on that? Feel free to share some of your own benchmarking results as well.

I'll re-trigger these benchmarks as well.

@jed326 I have checked that nested-date-histo query will not call what I changed, I'm doing a deeper check.

@kkewwei kkewwei force-pushed the optimize_innerhits_query branch from b1d2f72 to a2f5579 Compare January 15, 2025 15:05
@kkewwei
Copy link
Contributor Author

kkewwei commented Jan 15, 2025 •
edited
Loading

@jed326 @msfroh, I dug deeper, and found that nested-date-histo query has nothing to do with my modification, All my changes are related to FetchPhase, while nested-date-histo only execute QueryPhase(size=0). So the query has no reference value for my changes.

Furthermore, I put the dataset nested(3.5gb) installed in the Opensearch on my MacBook, and test the nested-date-histo, which also verifies the conclusion above.

Please help run benchmark test: id_16 again.

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for a2f5579: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@jed326
Copy link
Collaborator

jed326 commented Jan 15, 2025

{"run-benchmark-test": "id_16"}

@jed326
Copy link
Collaborator

jed326 commented Jan 15, 2025

All my changes are related to FetchPhase, while nested-date-histo only execute QueryPhase(size=0).

Thanks @kkewwei , that makes sense. Unfortunately we don't have any nightly runs for the nested workload so there's no historical results for us to compare to either.

@github-actions GitHub Actions
Copy link
Contributor

The Jenkins job url is https://build.ci.opensearch.org/job/benchmark-pull-request/2109/ . Final results will be published once the job is completed.

jed326
jed326 reviewed Jan 15, 2025
View reviewed changes
Copy link
Collaborator

@jed326 jed326 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some cleanup related comments but otherwise LGTM, will let @msfroh do another pass too

server/src/main/java/org/opensearch/search/aggregations/bucket/BucketsAggregator.java Outdated Show resolved Hide resolved
server/src/main/java/org/opensearch/search/aggregations/bucket/nested/NestedAggregator.java Outdated Show resolved Hide resolved
@opensearch-ci-bot
Copy link
Collaborator

Benchmark Results

Benchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-pull-request/2109/

Metric Task Value Unit
Cumulative indexing time of primary shards 17.9177 min
Min cumulative indexing time across primary shards 17.9177 min
Median cumulative indexing time across primary shards 17.9177 min
Max cumulative indexing time across primary shards 17.9177 min
Cumulative indexing throttle time of primary shards 0 min
Min cumulative indexing throttle time across primary shards 0 min
Median cumulative indexing throttle time across primary shards 0 min
Max cumulative indexing throttle time across primary shards 0 min
Cumulative merge time of primary shards 3.74107 min
Cumulative merge count of primary shards 8
Min cumulative merge time across primary shards 3.74107 min
Median cumulative merge time across primary shards 3.74107 min
Max cumulative merge time across primary shards 3.74107 min
Cumulative merge throttle time of primary shards 1.60065 min
Min cumulative merge throttle time across primary shards 1.60065 min
Median cumulative merge throttle time across primary shards 1.60065 min
Max cumulative merge throttle time across primary shards 1.60065 min
Cumulative refresh time of primary shards 0.50895 min
Cumulative refresh count of primary shards 23
Min cumulative refresh time across primary shards 0.50895 min
Median cumulative refresh time across primary shards 0.50895 min
Max cumulative refresh time across primary shards 0.50895 min
Cumulative flush time of primary shards 1.21608 min
Cumulative flush count of primary shards 9
Min cumulative flush time across primary shards 1.21608 min
Median cumulative flush time across primary shards 1.21608 min
Max cumulative flush time across primary shards 1.21608 min
Total Young Gen GC time 1.606 s
Total Young Gen GC count 55
Total Old Gen GC time 0 s
Total Old Gen GC count 0
Store size 3.36191 GB
Translog size 5.12227e-08 GB
Heap used for segments 0 MB
Heap used for doc values 0 MB
Heap used for terms 0 MB
Heap used for norms 0 MB
Heap used for points 0 MB
Heap used for stored fields 0 MB
Segment count 28
Min Throughput index-append 34724.6 docs/s
Mean Throughput index-append 35439.6 docs/s
Median Throughput index-append 35271.2 docs/s
Max Throughput index-append 36956.3 docs/s
50th percentile latency index-append 455.602 ms
90th percentile latency index-append 599.439 ms
99th percentile latency index-append 2626.21 ms
99.9th percentile latency index-append 8034.82 ms
100th percentile latency index-append 9498.26 ms
50th percentile service time index-append 455.587 ms
90th percentile service time index-append 599.588 ms
99th percentile service time index-append 2626.21 ms
99.9th percentile service time index-append 8034.82 ms
100th percentile service time index-append 9498.26 ms
error rate index-append 0 %
Min Throughput wait-until-merges-finish 105.1 ops/s
Mean Throughput wait-until-merges-finish 105.1 ops/s
Median Throughput wait-until-merges-finish 105.1 ops/s
Max Throughput wait-until-merges-finish 105.1 ops/s
100th percentile latency wait-until-merges-finish 9.19464 ms
100th percentile service time wait-until-merges-finish 9.19464 ms
error rate wait-until-merges-finish 0 %
Min Throughput randomized-nested-queries 19.88 ops/s
Mean Throughput randomized-nested-queries 19.93 ops/s
Median Throughput randomized-nested-queries 19.94 ops/s
Max Throughput randomized-nested-queries 19.96 ops/s
50th percentile latency randomized-nested-queries 46.1434 ms
90th percentile latency randomized-nested-queries 67.022 ms
99th percentile latency randomized-nested-queries 80.4837 ms
99.9th percentile latency randomized-nested-queries 93.3667 ms
100th percentile latency randomized-nested-queries 107.144 ms
50th percentile service time randomized-nested-queries 45.434 ms
90th percentile service time randomized-nested-queries 66.1227 ms
99th percentile service time randomized-nested-queries 79.5363 ms
99.9th percentile service time randomized-nested-queries 92.568 ms
100th percentile service time randomized-nested-queries 106.376 ms
error rate randomized-nested-queries 0 %
Min Throughput randomized-term-queries 24.99 ops/s
Mean Throughput randomized-term-queries 25 ops/s
Median Throughput randomized-term-queries 25 ops/s
Max Throughput randomized-term-queries 25 ops/s
50th percentile latency randomized-term-queries 12.2434 ms
90th percentile latency randomized-term-queries 27.3462 ms
99th percentile latency randomized-term-queries 46.0607 ms
100th percentile latency randomized-term-queries 49.9128 ms
50th percentile service time randomized-term-queries 11.2763 ms
90th percentile service time randomized-term-queries 26.2106 ms
99th percentile service time randomized-term-queries 45.1469 ms
100th percentile service time randomized-term-queries 48.9333 ms
error rate randomized-term-queries 0 %
Min Throughput randomized-sorted-term-queries 15.99 ops/s
Mean Throughput randomized-sorted-term-queries 15.99 ops/s
Median Throughput randomized-sorted-term-queries 15.99 ops/s
Max Throughput randomized-sorted-term-queries 15.99 ops/s
50th percentile latency randomized-sorted-term-queries 14.8429 ms
90th percentile latency randomized-sorted-term-queries 22.271 ms
99th percentile latency randomized-sorted-term-queries 28.4499 ms
100th percentile latency randomized-sorted-term-queries 29.2802 ms
50th percentile service time randomized-sorted-term-queries 13.7724 ms
90th percentile service time randomized-sorted-term-queries 21.2691 ms
99th percentile service time randomized-sorted-term-queries 27.4151 ms
100th percentile service time randomized-sorted-term-queries 28.5315 ms
error rate randomized-sorted-term-queries 0 %
Min Throughput match-all 5 ops/s
Mean Throughput match-all 5 ops/s
Median Throughput match-all 5 ops/s
Max Throughput match-all 5 ops/s
50th percentile latency match-all 3.97398 ms
90th percentile latency match-all 4.38565 ms
99th percentile latency match-all 4.76561 ms
100th percentile latency match-all 8.605 ms
50th percentile service time match-all 2.80019 ms
90th percentile service time match-all 3.11581 ms
99th percentile service time match-all 3.28809 ms
100th percentile service time match-all 7.59863 ms
error rate match-all 0 %
Min Throughput nested-date-histo 1 ops/s
Mean Throughput nested-date-histo 1 ops/s
Median Throughput nested-date-histo 1 ops/s
Max Throughput nested-date-histo 1 ops/s
50th percentile latency nested-date-histo 1557.82 ms
90th percentile latency nested-date-histo 1595.16 ms
99th percentile latency nested-date-histo 1623.74 ms
100th percentile latency nested-date-histo 1717.43 ms
50th percentile service time nested-date-histo 1556.47 ms
90th percentile service time nested-date-histo 1593.22 ms
99th percentile service time nested-date-histo 1622.89 ms
100th percentile service time nested-date-histo 1716.05 ms
error rate nested-date-histo 0 %
Min Throughput randomized-nested-queries-with-inner-hits_default 17.98 ops/s
Mean Throughput randomized-nested-queries-with-inner-hits_default 17.99 ops/s
Median Throughput randomized-nested-queries-with-inner-hits_default 17.99 ops/s
Max Throughput randomized-nested-queries-with-inner-hits_default 17.99 ops/s
50th percentile latency randomized-nested-queries-with-inner-hits_default 46.4734 ms
90th percentile latency randomized-nested-queries-with-inner-hits_default 67.6089 ms
99th percentile latency randomized-nested-queries-with-inner-hits_default 80.8365 ms
99.9th percentile latency randomized-nested-queries-with-inner-hits_default 91.6713 ms
100th percentile latency randomized-nested-queries-with-inner-hits_default 94.7892 ms
50th percentile service time randomized-nested-queries-with-inner-hits_default 45.5613 ms
90th percentile service time randomized-nested-queries-with-inner-hits_default 66.8789 ms
99th percentile service time randomized-nested-queries-with-inner-hits_default 79.811 ms
99.9th percentile service time randomized-nested-queries-with-inner-hits_default 90.491 ms
100th percentile service time randomized-nested-queries-with-inner-hits_default 93.7555 ms
error rate randomized-nested-queries-with-inner-hits_default 0 %
Min Throughput randomized-nested-queries-with-inner-hits_default_big_size 16 ops/s
Mean Throughput randomized-nested-queries-with-inner-hits_default_big_size 16 ops/s
Median Throughput randomized-nested-queries-with-inner-hits_default_big_size 16 ops/s
Max Throughput randomized-nested-queries-with-inner-hits_default_big_size 16 ops/s
50th percentile latency randomized-nested-queries-with-inner-hits_default_big_size 53.6835 ms
90th percentile latency randomized-nested-queries-with-inner-hits_default_big_size 75.7195 ms
99th percentile latency randomized-nested-queries-with-inner-hits_default_big_size 90.0311 ms
99.9th percentile latency randomized-nested-queries-with-inner-hits_default_big_size 99.7014 ms
100th percentile latency randomized-nested-queries-with-inner-hits_default_big_size 124.62 ms
50th percentile service time randomized-nested-queries-with-inner-hits_default_big_size 52.7549 ms
90th percentile service time randomized-nested-queries-with-inner-hits_default_big_size 74.7934 ms
99th percentile service time randomized-nested-queries-with-inner-hits_default_big_size 89.0985 ms
99.9th percentile service time randomized-nested-queries-with-inner-hits_default_big_size 98.524 ms
100th percentile service time randomized-nested-queries-with-inner-hits_default_big_size 122.851 ms
error rate randomized-nested-queries-with-inner-hits_default_big_size 0 %

@opensearch-ci-bot
Copy link
Collaborator

Benchmark Baseline Comparison Results

Benchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-compare/36/

Metric Task Baseline Contender Diff Unit
Cumulative indexing time of primary shards 18.1561 17.9177 -0.23837 min
Min cumulative indexing time across primary shard 18.1561 17.9177 -0.23837 min
Median cumulative indexing time across primary shard 18.1561 17.9177 -0.23837 min
Max cumulative indexing time across primary shard 18.1561 17.9177 -0.23837 min
Cumulative indexing throttle time of primary shards 0 0 0 min
Min cumulative indexing throttle time across primary shard 0 0 0 min
Median cumulative indexing throttle time across primary shard 0 0 0 min
Max cumulative indexing throttle time across primary shard 0 0 0 min
Cumulative merge time of primary shards 3.55243 3.74107 0.18863 min
Cumulative merge count of primary shards 10 8 -2
Min cumulative merge time across primary shard 3.55243 3.74107 0.18863 min
Median cumulative merge time across primary shard 3.55243 3.74107 0.18863 min
Max cumulative merge time across primary shard 3.55243 3.74107 0.18863 min
Cumulative merge throttle time of primary shards 1.23895 1.60065 0.3617 min
Min cumulative merge throttle time across primary shard 1.23895 1.60065 0.3617 min
Median cumulative merge throttle time across primary shard 1.23895 1.60065 0.3617 min
Max cumulative merge throttle time across primary shard 1.23895 1.60065 0.3617 min
Cumulative refresh time of primary shards 0.4725 0.50895 0.03645 min
Cumulative refresh count of primary shards 25 23 -2
Min cumulative refresh time across primary shard 0.4725 0.50895 0.03645 min
Median cumulative refresh time across primary shard 0.4725 0.50895 0.03645 min
Max cumulative refresh time across primary shard 0.4725 0.50895 0.03645 min
Cumulative flush time of primary shards 1.20917 1.21608 0.00692 min
Cumulative flush count of primary shards 10 9 -1
Min cumulative flush time across primary shard 1.20917 1.21608 0.00692 min
Median cumulative flush time across primary shard 1.20917 1.21608 0.00692 min
Max cumulative flush time across primary shard 1.20917 1.21608 0.00692 min
Total Young Gen GC time 1.773 1.606 -0.167 s
Total Young Gen GC count 58 55 -3
Total Old Gen GC time 0 0 0 s
Total Old Gen GC count 0 0 0
Store size 3.38136 3.36191 -0.01945 GB
Translog size 5.12227e-08 5.12227e-08 0 GB
Heap used for segments 0 0 0 MB
Heap used for doc values 0 0 0 MB
Heap used for terms 0 0 0 MB
Heap used for norms 0 0 0 MB
Heap used for points 0 0 0 MB
Heap used for stored fields 0 0 0 MB
Segment count 30 28 -2
Min Throughput index-append 34049.7 34724.6 674.887 docs/s
Mean Throughput index-append 34980.1 35439.6 459.505 docs/s
Median Throughput index-append 34997.7 35271.2 273.52 docs/s
Max Throughput index-append 35965.3 36956.3 990.938 docs/s
50th percentile latency index-append 455.325 455.602 0.27692 ms
90th percentile latency index-append 634.377 599.439 -34.9383 ms
99th percentile latency index-append 2686.66 2626.21 -60.45 ms
99.9th percentile latency index-append 8984.2 8034.82 -949.381 ms
100th percentile latency index-append 9647.02 9498.26 -148.762 ms
50th percentile service time index-append 455.274 455.587 0.31327 ms
90th percentile service time index-append 634.3 599.588 -34.7112 ms
99th percentile service time index-append 2686.66 2626.21 -60.45 ms
99.9th percentile service time index-append 8984.2 8034.82 -949.381 ms
100th percentile service time index-append 9647.02 9498.26 -148.762 ms
error rate index-append 0 0 0 %
Min Throughput wait-until-merges-finish 63.4674 105.099 41.632 ops/s
Mean Throughput wait-until-merges-finish 63.4674 105.099 41.632 ops/s
Median Throughput wait-until-merges-finish 63.4674 105.099 41.632 ops/s
Max Throughput wait-until-merges-finish 63.4674 105.099 41.632 ops/s
100th percentile latency wait-until-merges-finish 15.422 9.19464 -6.22736 ms
100th percentile service time wait-until-merges-finish 15.422 9.19464 -6.22736 ms
error rate wait-until-merges-finish 0 0 0 %
Min Throughput randomized-nested-queries 19.9056 19.8774 -0.02812 ops/s
Mean Throughput randomized-nested-queries 19.9471 19.9316 -0.01548 ops/s
Median Throughput randomized-nested-queries 19.9518 19.9377 -0.01412 ops/s
Max Throughput randomized-nested-queries 19.9679 19.9582 -0.00965 ops/s
50th percentile latency randomized-nested-queries 42.5904 46.1434 3.55298 ms
90th percentile latency randomized-nested-queries 64.2178 67.022 2.80421 ms
99th percentile latency randomized-nested-queries 79.7933 80.4837 0.69031 ms
99.9th percentile latency randomized-nested-queries 99.9294 93.3667 -6.5627 ms
100th percentile latency randomized-nested-queries 101.745 107.144 5.399 ms
50th percentile service time randomized-nested-queries 41.6684 45.434 3.76557 ms
90th percentile service time randomized-nested-queries 63.0539 66.1227 3.06879 ms
99th percentile service time randomized-nested-queries 78.7359 79.5363 0.8004 ms
99.9th percentile service time randomized-nested-queries 98.6026 92.568 -6.03463 ms
100th percentile service time randomized-nested-queries 100.815 106.376 5.56124 ms
error rate randomized-nested-queries 0 0 0 %
Min Throughput randomized-term-queries 24.995 24.9939 -0.00111 ops/s
Mean Throughput randomized-term-queries 24.9958 24.9954 -0.00035 ops/s
Median Throughput randomized-term-queries 24.9958 24.9954 -0.00045 ops/s
Max Throughput randomized-term-queries 24.9964 24.9962 -0.00025 ops/s
50th percentile latency randomized-term-queries 8.76846 12.2434 3.47495 ms
90th percentile latency randomized-term-queries 21.5016 27.3462 5.8446 ms
99th percentile latency randomized-term-queries 32.1076 46.0607 13.953 ms
100th percentile latency randomized-term-queries 35.4878 49.9128 14.425 ms
50th percentile service time randomized-term-queries 7.62894 11.2763 3.64737 ms
90th percentile service time randomized-term-queries 20.6557 26.2106 5.55484 ms
99th percentile service time randomized-term-queries 31.1342 45.1469 14.0127 ms
100th percentile service time randomized-term-queries 34.7178 48.9333 14.2155 ms
error rate randomized-term-queries 0 0 0 %
Min Throughput randomized-sorted-term-queries 15.9973 15.9883 -0.00896 ops/s
Mean Throughput randomized-sorted-term-queries 15.9978 15.9901 -0.00767 ops/s
Median Throughput randomized-sorted-term-queries 15.9978 15.9903 -0.00755 ops/s
Max Throughput randomized-sorted-term-queries 15.9981 15.9915 -0.00663 ops/s
50th percentile latency randomized-sorted-term-queries 12.6395 14.8429 2.20342 ms
90th percentile latency randomized-sorted-term-queries 17.9121 22.271 4.35896 ms
99th percentile latency randomized-sorted-term-queries 24.7461 28.4499 3.70378 ms
100th percentile latency randomized-sorted-term-queries 25.2983 29.2802 3.98194 ms
50th percentile service time randomized-sorted-term-queries 11.6037 13.7724 2.16871 ms
90th percentile service time randomized-sorted-term-queries 16.9642 21.2691 4.30496 ms
99th percentile service time randomized-sorted-term-queries 23.6991 27.4151 3.716 ms
100th percentile service time randomized-sorted-term-queries 24.7119 28.5315 3.81961 ms
error rate randomized-sorted-term-queries 0 0 0 %
Min Throughput match-all 5.00327 5.00322 -5e-05 ops/s
Mean Throughput match-all 5.00385 5.00379 -6e-05 ops/s
Median Throughput match-all 5.00382 5.00375 -7e-05 ops/s
Max Throughput match-all 5.00456 5.0045 -6e-05 ops/s
50th percentile latency match-all 4.30607 3.97398 -0.33209 ms
90th percentile latency match-all 4.69927 4.38565 -0.31363 ms
99th percentile latency match-all 4.99247 4.76561 -0.22687 ms
100th percentile latency match-all 5.2292 8.605 3.3758 ms
50th percentile service time match-all 3.14069 2.80019 -0.34049 ms
90th percentile service time match-all 3.40906 3.11581 -0.29324 ms
99th percentile service time match-all 3.56543 3.28809 -0.27734 ms
100th percentile service time match-all 3.96887 7.59863 3.62976 ms
error rate match-all 0 0 0 %
Min Throughput nested-date-histo 0.994236 0.996097 0.00186 ops/s
Mean Throughput nested-date-histo 0.99802 0.997841 -0.00018 ops/s
Median Throughput nested-date-histo 0.998221 0.998034 -0.00019 ops/s
Max Throughput nested-date-histo 0.998852 0.998686 -0.00017 ops/s
50th percentile latency nested-date-histo 1441.07 1557.82 116.747 ms
90th percentile latency nested-date-histo 1464.12 1595.16 131.036 ms
99th percentile latency nested-date-histo 3507.82 1623.74 -1884.07 ms
100th percentile latency nested-date-histo 3939.14 1717.43 -2221.71 ms
50th percentile service time nested-date-histo 1439.3 1556.47 117.162 ms
90th percentile service time nested-date-histo 1461.24 1593.22 131.985 ms
99th percentile service time nested-date-histo 2685.54 1622.89 -1062.64 ms
100th percentile service time nested-date-histo 2698.42 1716.05 -982.365 ms
error rate nested-date-histo 0 0 0 %
Min Throughput randomized-nested-queries-with-inner-hits_default 17.9868 17.9786 -0.00818 ops/s
Mean Throughput randomized-nested-queries-with-inner-hits_default 17.9927 17.9882 -0.00441 ops/s
Median Throughput randomized-nested-queries-with-inner-hits_default 17.9934 17.9893 -0.00407 ops/s
Max Throughput randomized-nested-queries-with-inner-hits_default 17.9956 17.9928 -0.00275 ops/s
50th percentile latency randomized-nested-queries-with-inner-hits_default 43.6423 46.4734 2.83114 ms
90th percentile latency randomized-nested-queries-with-inner-hits_default 64.3459 67.6089 3.26303 ms
99th percentile latency randomized-nested-queries-with-inner-hits_default 75.5677 80.8365 5.26879 ms
99.9th percentile latency randomized-nested-queries-with-inner-hits_default 81.1421 91.6713 10.5292 ms
100th percentile latency randomized-nested-queries-with-inner-hits_default 81.4173 94.7892 13.3719 ms
50th percentile service time randomized-nested-queries-with-inner-hits_default 42.5791 45.5613 2.98222 ms
90th percentile service time randomized-nested-queries-with-inner-hits_default 63.412 66.8789 3.4669 ms
99th percentile service time randomized-nested-queries-with-inner-hits_default 74.7049 79.811 5.10609 ms
99.9th percentile service time randomized-nested-queries-with-inner-hits_default 80.3584 90.491 10.1325 ms
100th percentile service time randomized-nested-queries-with-inner-hits_default 80.9312 93.7555 12.8243 ms
error rate randomized-nested-queries-with-inner-hits_default 0 0 0 %
Min Throughput randomized-nested-queries-with-inner-hits_default_big_size 15.9999 16.0003 0.00044 ops/s
Mean Throughput randomized-nested-queries-with-inner-hits_default_big_size 16.0001 16.0006 0.00053 ops/s
Median Throughput randomized-nested-queries-with-inner-hits_default_big_size 16.0001 16.0005 0.00046 ops/s
Max Throughput randomized-nested-queries-with-inner-hits_default_big_size 16.0003 16.0012 0.0009 ops/s
50th percentile latency randomized-nested-queries-with-inner-hits_default_big_size 52.7272 53.6835 0.9563 ms
90th percentile latency randomized-nested-queries-with-inner-hits_default_big_size 74.159 75.7195 1.56051 ms
99th percentile latency randomized-nested-queries-with-inner-hits_default_big_size 84.2327 90.0311 5.79837 ms
99.9th percentile latency randomized-nested-queries-with-inner-hits_default_big_size 97.3411 99.7014 2.3603 ms
100th percentile latency randomized-nested-queries-with-inner-hits_default_big_size 99.6463 124.62 24.9742 ms
50th percentile service time randomized-nested-queries-with-inner-hits_default_big_size 51.8048 52.7549 0.95008 ms
90th percentile service time randomized-nested-queries-with-inner-hits_default_big_size 73.3024 74.7934 1.49092 ms
99th percentile service time randomized-nested-queries-with-inner-hits_default_big_size 82.5657 89.0985 6.53281 ms
99.9th percentile service time randomized-nested-queries-with-inner-hits_default_big_size 95.8108 98.524 2.71323 ms
100th percentile service time randomized-nested-queries-with-inner-hits_default_big_size 98.7517 122.851 24.0992 ms
error rate randomized-nested-queries-with-inner-hits_default_big_size 0 0 0 %

@rishabh6788
Copy link
Contributor

All my changes are related to FetchPhase, while nested-date-histo only execute QueryPhase(size=0).

Thanks @kkewwei , that makes sense. Unfortunately we don't have any nightly runs for the nested workload so there's no historical results for us to compare to either.

@jed326 We do have nightly baseline runs for nested workload now, they are are used to run comparison against baseline results. In the comparison results I can see that even though there is not any improvement in p50 and p90 service_time for nested_date_histo, p99 and p100 are showing significant improvement.

@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Jan 13, 2025
Open
@kkewwei
Optimize innerhits query performance
acae7f1 
Signed-off-by: kkewwei <kewei.11@bytedance.com>
Signed-off-by: kkewwei <kkewwei@163.com>
@kkewwei kkewwei force-pushed the optimize_innerhits_query branch from a2f5579 to acae7f1 Compare January 16, 2025 01:01
@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for acae7f1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@kkewwei
Copy link
Contributor Author

kkewwei commented Jan 16, 2025

❌ Gradle check result for acae7f1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

org.opensearch.indices.replication.SegmentReplicationAllocationIT.testSingleIndexShardAllocation #14327
org.opensearch.versioning.ConcurrentSeqNoVersioningIT.testSeqNoCASLinearizability #2499

@github-actions GitHub Actions
Copy link
Contributor

❌ Gradle check result for acae7f1:

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions GitHub Actions
Copy link
Contributor

❕ Gradle check result for acae7f1: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

This was referenced Jan 17, 2025
Open
Open
@kkewwei
Copy link
Contributor Author

kkewwei commented Jan 17, 2025

❕ Gradle check result for acae7f1: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock #14289

@jed326
Copy link
Collaborator

jed326 commented Jan 17, 2025

@msfroh would you like to take another pass at this?

@msfroh msfroh merged commit 92088be into opensearch-project:main Jan 22, 2025
37 of 39 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Jan 22, 2025
@github-actions
Optimize innerhits query performance (#16937)
d67b79b 
Signed-off-by: kkewwei <kewei.11@bytedance.com>
Signed-off-by: kkewwei <kkewwei@163.com>
(cherry picked from commit 92088be)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@opensearch-trigger-bot opensearch-trigger-bot bot mentioned this pull request Jan 22, 2025
Merged
@kkewwei kkewwei deleted the optimize_innerhits_query branch January 22, 2025 02:52
jed326 pushed a commit that referenced this pull request Jan 22, 2025
@opensearch-trigger-bot @github-actions
Optimize innerhits query performance (#16937) (#17084)
73e1b92 
(cherry picked from commit 92088be)

Signed-off-by: kkewwei <kewei.11@bytedance.com>
Signed-off-by: kkewwei <kkewwei@163.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Feb 6, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jed326 jed326 jed326 approved these changes

@msfroh msfroh msfroh approved these changes

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

@andrross andrross Awaiting requested review from andrross andrross is a code owner

@ashking94 ashking94 Awaiting requested review from ashking94 ashking94 is a code owner

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

@dblock dblock Awaiting requested review from dblock dblock is a code owner

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

@gbbafna gbbafna Awaiting requested review from gbbafna gbbafna is a code owner

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 is a code owner

@nknize nknize Awaiting requested review from nknize nknize is a code owner

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

@reta reta Awaiting requested review from reta reta is a code owner

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

@sachinpkale sachinpkale Awaiting requested review from sachinpkale sachinpkale is a code owner

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

@shwetathareja shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

@sohami sohami Awaiting requested review from sohami sohami is a code owner

@VachaShah VachaShah Awaiting requested review from VachaShah VachaShah is a code owner

@jainankitk jainankitk Awaiting requested review from jainankitk jainankitk is a code owner

@linuxpi linuxpi Awaiting requested review from linuxpi linuxpi is a code owner

Assignees
No one assigned
Labels
backport 2.x Backport to 2.x branch enhancement Enhancement or improvement to existing feature or request Search:Performance v2.19.0 Issues and PRs related to version 2.19.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature Request] Concurrent executing query in InnerHitsPhase
5 participants
@kkewwei @rishabh6788 @opensearch-ci-bot @jed326 @msfroh