EPIC: Contribute Dask-SQL codebase to Apache Arrow DataFusion Python

[jdye64]

Is your feature request related to a problem? Please describe.
Dask-SQL currently supports our own custom set of Rust PyO3 bindings for Apache Arrow DataFusion. Since we started that effort interest has grown in that community around offering their own set of Python bindings for Arrow DataFusion. It seems sensible to me to contribute the bindings that we have and gain the development support from that community and alleviate our developer time for features and enhancements.

This EPIC is setup to track the effort of moving code to Arrow DataFusion Python and then refactoring our codebase to subsequently use it.

While the PRs will mostly be simple in nature there is likely to be several. The choice was made to do several PRs in favor of a single large PR so reviewing would be more quick and easy and to help identify any possible regressions that might present themselves in a more cornered manner.

I will attempt to keep this list up to date with PRs relevant to this effort and their status

Arrow DataFusion Python - Worklog

• Analyze table bindings apache/datafusion-python#204
• Empty relation bindings apache/datafusion-python#208
• wrap display_name and canonical_name functions apache/datafusion-python#214
• Add PyAlias bindings apache/datafusion-python#216
• Add bindings for scalar_variable apache/datafusion-python#218
• Bindings for LIKE type expressions apache/datafusion-python#220
• Bool expr bindings apache/datafusion-python#223
• Between bindings apache/datafusion-python#229
• Add bindings for GetIndexedField apache/datafusion-python#227
• Add bindings for case, cast, and trycast apache/datafusion-python#232
• add remaining expr bindings apache/datafusion-python#233
• Extension bindings apache/datafusion-python#266
• Subquery alias bindings apache/datafusion-python#269
• Create memory table apache/datafusion-python#271
• Create view bindings apache/datafusion-python#273
• Re-export Datafusion dependencies apache/datafusion-python#277
• Add DROP TABLE bindings
• Add REPARTITION bindings
• Improve build command so that python bindings can be built "out of band", meaning projects like Dask-SQL can build the python bindings and link to their via their own Cargo build process

Dask-SQL - Worklog

• [ENH] Add Arrow DataFusion Python as Cargo dependency #1084
• Get conda build working with new dependencies
• Passing test_analyze.py
• Passing test_cmd.py
• Passing test_compatibility.py
• Passing test_complex.py
• Passing test_create.py
• Passing test_distributeby.py
• Passing test_explain.py
• Passing test_filter.py
• Passing test_fugue.py
• Passing test_function.py
• Passing test_groupby.py
• Passing test_hive.py
• Passing test_intake.py
• Passing test_jdbc.py
• Passing test_join.py
• Passing test_model.py
• Passing test_over.py
• Passing test_postgres.py
• Passing test_rex.py
• Passing test_sample.py
• Passing test_schema.py
• Passing test_select.py
• Passing test_server.py
• Passing test_show.py
• Passing test_sort.py
• Passing test_sqlite.py
• Passing test_union.py