Expand streaming Aggregations to Cardinality Aggregator #19484

harshavamsi · 2025-09-30T19:01:56Z

Description

Expands streaming aggregations to cardinality aggregator. This PR ensures that we only use the Ordinals Collector while streaming and resets the value after each new batch.

Related Issues

Resolves #19515

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2025-09-30T21:59:32Z

✅ Gradle check result for 990a850: SUCCESS

codecov · 2025-09-30T22:00:10Z

Codecov Report

❌ Patch coverage is 71.42857% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.96%. Comparing base (ac6dfa1) to head (53a7264).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...regations/metrics/StreamCardinalityAggregator.java	72.58%	10 Missing and 7 partials ⚠️
...egations/metrics/CardinalityAggregatorFactory.java	50.00%	2 Missing ⚠️
.../search/aggregations/BucketCollectorProcessor.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #19484      +/-   ##
============================================
- Coverage     73.09%   72.96%   -0.13%     
+ Complexity    70553    70508      -45     
============================================
  Files          5716     5717       +1     
  Lines        322926   322995      +69     
  Branches      46770    46780      +10     
============================================
- Hits         236032   235676     -356     
- Misses        67882    68368     +486     
+ Partials      19012    18951      -61

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2025-09-30T23:41:39Z

❕ Gradle check result for c1979ea: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2025-10-01T02:42:35Z

❌ Gradle check result for bfe824e: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-10-01T23:05:22Z

❕ Gradle check result for 4b6dd24: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

rishabhmaurya · 2025-10-02T01:32:18Z

@harshavamsi we should add integ test similar to the ones we have for streaming terms aggregation. We should do it for both numeric and cardinality aggs.

github-actions · 2025-10-02T03:01:27Z

❕ Gradle check result for 94ce824: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

github-actions · 2025-10-03T09:30:19Z

❌ Gradle check result for 53a7264: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-10-03T15:32:08Z

✅ Gradle check result for 53a7264: SUCCESS

...-rpc/src/internalClusterTest/java/org/opensearch/streaming/aggregation/SubAggregationIT.java

...r/src/main/java/org/opensearch/search/aggregations/metrics/CardinalityAggregatorFactory.java

rishabhmaurya · 2025-10-03T19:03:58Z

In order to allow cardinality aggregator, we also need to enable it at RestSearchAction level -

OpenSearch/server/src/main/java/org/opensearch/rest/action/search/RestSearchAction.java

Line 463 in 86ada27

static boolean canUseStreamSearch(SearchRequest searchRequest) {

rishabhmaurya · 2025-10-03T23:18:58Z

In order to allow cardinality aggregator, we also need to enable it at RestSearchAction level -

I just checked, it seems like we don't need to explicitly check here. So looks good.

* Add streaming cardinality aggregator Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Add reset Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Add reset Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Add more tests Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * add more tests Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * add integ tests Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> --------- Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> (cherry picked from commit 9923617) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

rishabhmaurya · 2025-10-03T23:32:57Z

created a follow up #19518 to handle other underlying issues

) * Add streaming cardinality aggregator * Add reset * Add reset * Add more tests * add more tests * add integ tests --------- (cherry picked from commit 9923617) Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

mkhludnev · 2025-10-06T19:48:22Z

...c/test/java/org/opensearch/search/aggregations/metrics/StreamCardinalityAggregatorTests.java

+
+    public void testBasicCardinality() throws Exception {
+        try (Directory directory = newDirectory()) {
+            try (IndexWriter indexWriter = new IndexWriter(directory, new IndexWriterConfig())) {


Caveat! These tests should be improved with RandomIndexWriter to verify the work on segmented index.

…roject#19484) * Add streaming cardinality aggregator Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Add reset Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Add reset Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * Add more tests Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * add more tests Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> * add integ tests Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com> --------- Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

harshavamsi requested review from a team, Bukhtawar, CEHENKLE, Rishikesh1159, anasalkouz, andrross, ashking94, cwperks, dbwiddis, gbbafna, jed326, kotwanikunal, mch2, msfroh, owaiskazi19, reta, sachinpkale, saratvemulapalli, shwetathareja and sohami as code owners September 30, 2025 19:01

opensearch-ci-bot mentioned this pull request Oct 1, 2025

[AUTOCUT] Gradle Check Flaky Test Report for IndexServiceTests #14407

Open

rishabhmaurya added the v3.3.0 label Oct 1, 2025

harshavamsi closed this Oct 2, 2025

add integ tests

53a7264

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

harshavamsi force-pushed the expand_streaming_agg_cardinality branch from 1913943 to 53a7264 Compare October 3, 2025 07:27

harshavamsi closed this Oct 3, 2025

harshavamsi reopened this Oct 3, 2025

harshavamsi changed the title ~~Expand streaming agg cardinality~~ Expand streaming Aggregations to Cardinality Aggregator Oct 3, 2025

rishabhmaurya reviewed Oct 3, 2025

View reviewed changes

...-rpc/src/internalClusterTest/java/org/opensearch/streaming/aggregation/SubAggregationIT.java Show resolved Hide resolved

rishabhmaurya reviewed Oct 3, 2025

View reviewed changes

...-rpc/src/internalClusterTest/java/org/opensearch/streaming/aggregation/SubAggregationIT.java Show resolved Hide resolved

rishabhmaurya reviewed Oct 3, 2025

View reviewed changes

...r/src/main/java/org/opensearch/search/aggregations/metrics/CardinalityAggregatorFactory.java Show resolved Hide resolved

rishabhmaurya reviewed Oct 3, 2025

View reviewed changes

...r/src/main/java/org/opensearch/search/aggregations/metrics/CardinalityAggregatorFactory.java Show resolved Hide resolved

rishabhmaurya approved these changes Oct 3, 2025

View reviewed changes

rishabhmaurya merged commit 9923617 into opensearch-project:main Oct 3, 2025
63 of 66 checks passed

rishabhmaurya added this to Performance Roadmap Oct 3, 2025

github-project-automation bot moved this to Todo in Performance Roadmap Oct 3, 2025

rishabhmaurya moved this from Todo to Done in Performance Roadmap Oct 3, 2025

rishabhmaurya added the backport 3.3 Backport to 3.3 branch label Oct 3, 2025

opensearch-trigger-bot bot mentioned this pull request Oct 3, 2025

[Backport 3.3] Expand streaming Aggregations to Cardinality Aggregator #19517

Merged

rishabhmaurya mentioned this pull request Oct 3, 2025

[BUG] Streaming aggregation with request cache and other fixes #19518

Closed

rishabhmaurya mentioned this pull request Oct 4, 2025

Disable request cache for streaming and other fixes #19520

Merged

1 task

mkhludnev reviewed Oct 6, 2025

View reviewed changes

opensearch-ci-bot mentioned this pull request Oct 9, 2025

[AUTOCUT] Gradle Check Flaky Test Report for ResourceAwareTasksTests #14293

Open

rishabhmaurya mentioned this pull request Oct 10, 2025

[RFC] Streaming Aggregation - A Memory-Efficient Approach #16774

Open

BrewTestBot mentioned this pull request Oct 15, 2025

opensearch 3.3.0 Homebrew/homebrew-core#249627

Merged

Expand streaming Aggregations to Cardinality Aggregator #19484

Expand streaming Aggregations to Cardinality Aggregator #19484

Uh oh!

Conversation

harshavamsi commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions bot commented Sep 30, 2025

Uh oh!

codecov bot commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Sep 30, 2025

Uh oh!

github-actions bot commented Oct 1, 2025

Uh oh!

github-actions bot commented Oct 1, 2025

Uh oh!

rishabhmaurya commented Oct 2, 2025

Uh oh!

github-actions bot commented Oct 2, 2025

Uh oh!

github-actions bot commented Oct 3, 2025

Uh oh!

github-actions bot commented Oct 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rishabhmaurya commented Oct 3, 2025

Uh oh!

rishabhmaurya commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

rishabhmaurya commented Oct 3, 2025

Uh oh!

mkhludnev Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

harshavamsi commented Sep 30, 2025 •

edited

Loading

codecov bot commented Sep 30, 2025 •

edited

Loading

rishabhmaurya commented Oct 3, 2025 •

edited

Loading