-
Notifications
You must be signed in to change notification settings - Fork 25.3k
[ESQL] Fix TopN grouping aggregates bug with non-qualifying intermediate states #129633
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Pinging @elastic/es-analytical-engine (Team:Analytics) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR fixes a bug in the TopN grouping aggregates where intermediate states with nulls were mishandled, ensuring that outdated TopN sets received from datanodes are correctly discarded by the coordinator.
- Added new intermediate input methods to aggregation functions.
- Updated tests to cover scenarios where discarding of non-qualifying TopN sets is required.
- Refactored code and generator logic for improved consistency in handling different types of group id blocks.
Reviewed Changes
Copilot reviewed 9 out of 87 changed files in this pull request and generated 2 comments.
Show a summary per file
File | Description |
---|---|
x-pack/plugin/esql/compute/src/test/java/org/elasticsearch/compute/operator/HashAggregationOperatorTests.java | Added testTopNNullsIntermediateDiscards to validate proper discarding of outdated TopN states. |
x-pack/plugin/esql/compute/src/test/java/org/elasticsearch/compute/aggregation/GroupingAggregatorFunctionTestCase.java | Updated to include and test new intermediate input methods, with minor internal assertion changes. |
x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/ToPartialGroupingAggregatorFunction.java | Added method overrides for addIntermediateInput using IntArrayBlock and IntBigArrayBlock. |
x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/GroupingAggregatorFunction.java | Introduced new method signatures for intermediate input handling. |
x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/FromPartialGroupingAggregatorFunction.java | Added intermediate input delegation to the underlying aggregator. |
x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/FilteredGroupingAggregatorFunction.java | Forwarded intermediate input calls to the next aggregator in the chain. |
x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/CountGroupingAggregatorFunction.java | Removed redundant Math.toIntExact conversions and added intermediate input implementations. |
x-pack/plugin/esql/compute/gen/src/main/java/org/elasticsearch/compute/gen/GroupingAggregatorImplementer.java | Updated code generator to produce appropriate addIntermediateInput methods based on the group id type. |
delegate.addIntermediateInput(positionOffset + offset, blockFactory.newIntArrayVector(chunk, count), page); | ||
int firstValueIndex = groupIds.getFirstValueIndex(position); | ||
int valueCount = groupIds.getValueCount(position); | ||
assert valueCount == 1; // Multi-values make chunking more complex, and it's not a real case yet |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[nitpick] Consider replacing the plain 'assert' statement with a proper JUnit assertion (e.g., assertEquals(1, valueCount)) to provide a clearer error message if this assumption fails.
assert valueCount == 1; // Multi-values make chunking more complex, and it's not a real case yet | |
assertEquals(1, valueCount, "Multi-values make chunking more complex, and it's not a real case yet"); |
Copilot uses AI. Check for mistakes.
Continuation of #127148
When datanodes send the STATS intermediate states to the coordinator, it aggregates them.
Now, however, the TopN groups sent by a datanode may not be acceptable in the coordinator (Because it has better values already), so it will discard such values.
However, the engine wasn't handling intermediate groups with nulls (TopNBlockHash uses nulls to discard unused groups).
See https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/aggregation/GroupingAggregator.java#L47
This code isn't connected with the query yet, so there's no bug in production