Skip to content

[ESQL] Fix TopN grouping aggregates bug with non-qualifying intermediate states #129633

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

ivancea
Copy link
Contributor

@ivancea ivancea commented Jun 18, 2025

Continuation of #127148

When datanodes send the STATS intermediate states to the coordinator, it aggregates them.
Now, however, the TopN groups sent by a datanode may not be acceptable in the coordinator (Because it has better values already), so it will discard such values.

However, the engine wasn't handling intermediate groups with nulls (TopNBlockHash uses nulls to discard unused groups).
See https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/GroupingAggregator.java#L47

This code isn't connected with the query yet, so there's no bug in production

@ivancea ivancea requested a review from nik9000 June 18, 2025 13:48
@ivancea ivancea added >non-issue Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) :Analytics/ES|QL AKA ESQL v9.1.0 labels Jun 18, 2025
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-analytical-engine (Team:Analytics)

@ivancea ivancea marked this pull request as draft June 18, 2025 13:52
@ivancea ivancea requested a review from Copilot June 19, 2025 09:25
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR fixes a bug in the TopN grouping aggregates where intermediate states with nulls were mishandled, ensuring that outdated TopN sets received from datanodes are correctly discarded by the coordinator.

  • Added new intermediate input methods to aggregation functions.
  • Updated tests to cover scenarios where discarding of non-qualifying TopN sets is required.
  • Refactored code and generator logic for improved consistency in handling different types of group id blocks.

Reviewed Changes

Copilot reviewed 9 out of 87 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
x-pack/plugin/esql/compute/src/test/java/org/elasticsearch/compute/operator/HashAggregationOperatorTests.java Added testTopNNullsIntermediateDiscards to validate proper discarding of outdated TopN states.
x-pack/plugin/esql/compute/src/test/java/org/elasticsearch/compute/aggregation/GroupingAggregatorFunctionTestCase.java Updated to include and test new intermediate input methods, with minor internal assertion changes.
x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/ToPartialGroupingAggregatorFunction.java Added method overrides for addIntermediateInput using IntArrayBlock and IntBigArrayBlock.
x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/GroupingAggregatorFunction.java Introduced new method signatures for intermediate input handling.
x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/FromPartialGroupingAggregatorFunction.java Added intermediate input delegation to the underlying aggregator.
x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/FilteredGroupingAggregatorFunction.java Forwarded intermediate input calls to the next aggregator in the chain.
x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/CountGroupingAggregatorFunction.java Removed redundant Math.toIntExact conversions and added intermediate input implementations.
x-pack/plugin/esql/compute/gen/src/main/java/org/elasticsearch/compute/gen/GroupingAggregatorImplementer.java Updated code generator to produce appropriate addIntermediateInput methods based on the group id type.

delegate.addIntermediateInput(positionOffset + offset, blockFactory.newIntArrayVector(chunk, count), page);
int firstValueIndex = groupIds.getFirstValueIndex(position);
int valueCount = groupIds.getValueCount(position);
assert valueCount == 1; // Multi-values make chunking more complex, and it's not a real case yet
Copy link
Preview

Copilot AI Jun 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] Consider replacing the plain 'assert' statement with a proper JUnit assertion (e.g., assertEquals(1, valueCount)) to provide a clearer error message if this assumption fails.

Suggested change
assert valueCount == 1; // Multi-values make chunking more complex, and it's not a real case yet
assertEquals(1, valueCount, "Multi-values make chunking more complex, and it's not a real case yet");

Copilot uses AI. Check for mistakes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Analytics/ES|QL AKA ESQL >non-issue Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) v9.1.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants