Non-windowed updating aggregates using datafusion. #588
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The main functionality this provides is the ability to run aggregates without windows, emitting update and retract messages that can be written to a debezium sink.
The logic for calculating the aggregate is done in UpdatingAggregatingFunc. This operator has three different versions of the aggregate exec, with three different modes:
Partial: Takes input and emits partial aggregate representations.
CombinePartial: Merge multiple partials into a single partial. This mode was added in https://github.com/ArroyoSystems/arrow-datafusion/pull/1/files.
Final: The final aggregation that finishes any aggregates, expecting partials as input.
These are combined with the new LastKeyValueView. This is a simple key-value map that uses the _timestamp field as expiration time. For any group by tuple there'll be at most one live entry in the map. Writes to state include a
_generation
field in parquet, which is used to ensure we restore the newest value.In the operator data is fed into the partial exec until it is time to flush, which happens under the following conditions:
Flushing follows the following steps:
"f"
) has any matches. These will become retracts. Then, write the new data to that final table. The retracts will be emitted before the appends.In order to make progress between flushes, the partial exec is advanced. We panic in handle_future_result() because the input will never have been closed on that exec.
Some other things that were changed in this PR: