Move Regr_* functions to use UDAF #10898
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Closes apache#10883 and is part of apache#8708
github-actions
bot
added
logical-expr
alamb
approved these changes
Jun 13, 2024
|
I took the liberty of running cargo fmt and regenerated the vendored data |
Merged
|
The macos failure https://github.com/apache/datafusion/actions/runs/9502875641/job/26191967199?pr=10898 appears to be related to #10904 (a flaky fuzz test) |
|
Merged up to get the fix for #10904 |
|
🎉 |
|
Thanks @eejbyfeldt and @alamb |
5 tasks
findepi
pushed a commit
to findepi/datafusion
that referenced
this pull request
Jul 16, 2024
* Move Regr_* functions to use UDAF Closes apache#10883 and is part of apache#8708 * Format and regen * tweak error check --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Michael-J-Ward
added a commit
to Michael-J-Ward/datafusion-python
that referenced
this pull request
Jul 24, 2024
Michael-J-Ward
added a commit
to Michael-J-Ward/datafusion-python
that referenced
this pull request
Jul 24, 2024
Michael-J-Ward
added a commit
to Michael-J-Ward/datafusion-python
that referenced
this pull request
Jul 25, 2024
andygrove
pushed a commit
to apache/datafusion-python
that referenced
this pull request
Jul 31, 2024
* chore: update datafusion deps * feat: impl ExecutionPlan::static_name() for DatasetExec This required trait method was added upstream [0] and recommends to simply forward to `static_name`. [0]: apache/datafusion#10266 * feat: update first_value and last_value wrappers. Upstream signatures were changed for the new new `AggregateBuilder` api [0]. This simply gets the code to work. We should better incorporate that API into `datafusion-python`. [0] apache/datafusion#10560 * migrate count to UDAF Builtin Count was removed upstream. TBD whether we want to re-implement `count_star` with new API. Ref: apache/datafusion#10893 * migrate approx_percentile_cont, approx_distinct, and approx_median to UDAF Ref: approx_distinct apache/datafusion#10851 Ref: approx_median apache/datafusion#10840 Ref: approx_percentile_cont and _with_weight apache/datafusion#10917 * migrate avg to UDAF Ref: apache/datafusion#10964 * migrage corr to UDAF Ref: apache/datafusion#10884 * migrate grouping to UDAF Ref: apache/datafusion#10906 * add alias `mean` for UDAF `avg` * migrate stddev to UDAF Ref: apache/datafusion#10827 * remove rust alias for stddev The python wrapper now provides stddev_samp alias. * migrage var_pop to UDAF Ref: apache/datafusion#10836 * migrate regr_* functions to UDAF Ref: apache/datafusion#10898 * migrate bitwise functions to UDAF The functions now take a single expression instead of a Vec<_>. Ref: apache/datafusion#10930 * add missing variants for ScalarValue with todo * fix typo in approx_percentile_cont * add distinct arg to count * comment out failing test `approx_percentile_cont` is now returning a DoubleArray instead of an IntArray. This may be a bug upstream; it requires further investigation. * update tests to expect lowercase `sum` in query plans This was changed upstream. Ref: apache/datafusion#10831 * update ScalarType data_type map * add docs dependency pickleshare * re-implement count_star * lint: ruff python lint * lint: rust cargo fmt * include name of window function in error for find_window_fn * refactor `find_window_fn` for debug clarity * search default aggregate functions by both name and aliases The alias list no longer includes the name of the function. Ref: apache/datafusion#10658 * fix markdown in find_window_fn docs * parameterize test_window_functions `first_value` and `last_value` are currently failing and marked as xfail. * add test ids to test_simple_select tests marked xfail * update find_window_fn to search built-ins first The behavior of `first_value` and `last_value` UDAFs currently does not match the built-in behavior. This allowed me to remove `marks=pytest.xfail` from the window tests. * improve first_call and last_call use of the builder API * remove trailing todos * fix examples/substrait.py * chore: remove explicit aliases from functions.rs Ref: #779 * remove `array_fn!` aliases * remove alias rules for `expr_fn_vec!` * remove alias rules from `expr_fn!` macro * remove unnecessary pyo3 var-arg signatures in functions.rs * remove pyo3 signatures that provided defaults for first_value and last_value * parametrize test_string_functions * test regr_ function wrappers Closes #778
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #10883 and is part of #8708
Rationale for this change
The motivation is in #8708
What changes are included in this PR?
This moves Regr to be implemented using UDAF as decided in #8708. To avoid duplicating to much code I had to modify an existing macros and add new one to avoid creating a lot of boilerplate for each one. Let me know if there is some better way of doing it.
Are these changes tested?
Existing tests cases.
Are there any user-facing changes?
I think so since we moved the implementations to a different package.