Tracking Upgrade to Datafusion 38 #690
Comments
Michael-J-Ward
added a commit
to Michael-J-Ward/datafusion-python
that referenced
this issue
May 13, 2024
|
I have a functioning upgrade PR open. Once again, the guiding "design" decision was "get to compiling and passing tests" as quickly as possible. If I removed something that you'd prefer to see implemented as a workaround, please let me know. |
andygrove
pushed a commit
that referenced
this issue
May 14, 2024
* chore: upgrade datafusion Deps Ref #690 * update concat and concat_ws to use datafusion_functions Moved in apache/datafusion#10089 * feat: upgrade functions.rs Upstream is continuing it's migration to UDFs. Ref apache/datafusion#10098 Ref apache/datafusion#10372 * fix ScalarUDF import * feat: remove deprecated suppors_filter_pushdown and impl supports_filters_pushdown Deprecated function removed in apache/datafusion#9923 * use `unnest_columns_with_options` instead of deprecated `unnest_column_with_option` * remove ScalarFunction wrappers These relied on upstream BuiltinScalarFunction, which are now removed. Ref apache/datafusion#10098 * update dataframe `test_describe` `null_count` was fixed upstream. Ref apache/datafusion#10260 * remove PyDFField and related methods DFField was removed upstream. Ref: apache/datafusion#9595 * bump `datafusion-python` package version to 38.0.0 * re-implement `PyExpr::column_name` The previous implementation relied on `DFField` which was removed upstream. Ref: apache/datafusion#9595
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Upstream Changelog
Changes
Incomplete list.
DFFieldwas removed in refactor: make dfschema wrap schemaref datafusion#9595BuiltinScalarFunctionwas removed in Move coalesce to datafusion-functions and remove BuiltInScalarFunction datafusion#10098 along with moredatafusion-functionsmigrationcov_sampwas moved todatafusion_functions_aggregatein MoveCovariance(Sample)covar/covar_sampto be a User Defined Aggregate Function datafusion#10372The text was updated successfully, but these errors were encountered: