Add PartitionedDistinctCount aggregation function #5786
Conversation
| @@ -123,6 +123,8 @@ public static AggregationFunction getAggregationFunction(FunctionContext functio | |||
| return new DistinctCountAggregationFunction(firstArgument); | |||
| case DISTINCTCOUNTBITMAP: | |||
| return new DistinctCountBitmapAggregationFunction(firstArgument); | |||
| case PARTITIONEDDISTINCTCOUNT: | |||
There was a problem hiding this comment.
we need to start using _ separators. I think the parser now handles both automatically
There was a problem hiding this comment.
Discussed offline. We should still keep no underscore within the aggregation function type, and parser will remove the underscore from the input function name to match the function type.
| break; | ||
| case LONG: | ||
| long[] longValues = blockValSet.getLongValuesSV(); | ||
| LongOpenHashSet longSet = aggregationResultHolder.getResult(); |
There was a problem hiding this comment.
I thought roaring bitmap works for long as well?
There was a problem hiding this comment.
There is a long version roaring bitmap Roaring64NavigableMap which is still in early stage within the current version of RoaringBitmap 0.8.0 we imported. We can test its performance when we upgrade the RoaringBitmap library for the latest version and change it later as it does not involve any backward-incompatible changes.
| @@ -28,6 +28,7 @@ | |||
| MINMAXRANGE("minMaxRange"), | |||
| DISTINCTCOUNT("distinctCount"), | |||
| DISTINCTCOUNTBITMAP("distinctCountBitmap"), | |||
| PARTITIONEDDISTINCTCOUNT("partitionedDistinctCount"), | |||
There was a problem hiding this comment.
Shall we change the function name to something like SEGMENTEDDISTINCTCOUT or let PARTITIONEDDISTINCTCOUNT to take a parameter of MergeLevel to indicate segment level or instance level merge?
There was a problem hiding this comment.
Good point. Renamed to SEGMENTPARTITIONEDDISTINCTCOUNT
Jackie-Jiang
force-pushed
the
partitioned_distinct_count
branch
from
fa12da2 to
c441c8a
Compare
Add a new
PartitionedDistinctCountAggregationFunctionto calculate the number of distinct values when values are partitioned for each segment.This function calculates the exact number of distinct values (using raw value instead of hash code) within the segment, then simply sums up the results from different segments to get the final result.