Add arrow datafusion python dependency #1085

jdye64 · 2023-03-14T12:40:50Z

Closed with #1084

…hon bindings for DataFusion and bare minimum refactoring we need in our repo to accomodate them

…n_python

…t in every build stage

jdye64 · 2023-03-15T13:49:02Z

Waiting on oxidecomputer/typify#221 to merge so that conda builds can work with new setup.

…sion_python

…sql into add_arrow_datafusion_python

… needed from updating datafusion version to 20.0.0

…ccidentally deleted it earlier

…n_python

dask_sql/mappings.py

charlesbluca

Thanks @jdye64 🎉 a few comments around how we include protoc in CI/packaging and some follow up around the timedelta changes

.github/workflows/conda.yml

.github/workflows/rust.yml

.github/workflows/test.yml

dask_sql/mappings.py

dask_sql/physical/rex/core/literal.py

tests/integration/test_groupby.py

tests/integration/test_select.py

dask_planner/.cargo/config.toml

dask_planner/Cargo.toml

dask_planner/src/sql/logical.rs

dask_planner/src/sql/optimizer.rs

…sql into add_arrow_datafusion_python

…t to arrow-datafusion-python repo

…python

… 30 days

ayushdg · 2023-05-03T20:41:18Z

tests/integration/test_rex.py

@@ -926,8 +926,8 @@ def test_timestampdiff(c):
            "hours": [46, 6648, 591744],
            "days": [1, 277, 24656],
            "weeks": [0, 39, 3522],
-            "months": [0, 9, 810],
-            "quarters": [0, 3, 270],
+            "months": [0, 9, 821],


Since this behavior is different from the results we get from mysql and a few implementations that support timestampdiff, I'm not sure whether we want to keep the existing logic or modify it. I didn't find an easy way to get a timestampdiff with datafusion/postgresql

ayushdg

Thanks a lot for these changes @jdye64!

charlesbluca · 2023-05-04T16:54:26Z

pyproject.toml

+[build-system]
+requires = ["setuptools", "wheel", "setuptools-rust"]
+
+[tool.isort]
+profile = "black"
+
+[tool.maturin]
+include = [
+    { path = "Cargo.lock", format = "sdist" }
+]
+exclude = [".github/**", "ci/**", ".asf.yaml"]
+# Require Cargo.lock is up to date
+locked = true


Noting that I've been thinking about consolidating as much of our project metadata into pyproject.toml as possible (similar to other RAPIDS/Dask projects) which would make this file less stubby

+1 to this. I've also been thinking of simplifying the setup.py scripts and moving as much as possible into the pyproject.toml file

jdye64 added 2 commits March 13, 2023 17:29

Introduce Arrow-DataFusion-Python, the new Apache Arrow community Pyt…

14932ad

…hon bindings for DataFusion and bare minimum refactoring we need in our repo to accomodate them

Merge remote-tracking branch 'upstream/main' into add_arrow_datafusio…

6ccf087

…n_python

jdye64 requested review from andygrove, ayushdg, charlesbluca and galipremsagar as code owners March 14, 2023 12:40

jdye64 marked this pull request as draft March 14, 2023 13:23

jdye64 added 12 commits March 14, 2023 12:17

point to test upstream branch with custom build command

62d49bd

update conda build recipe to include rust compiler and setuptools-rus…

cdc236b

…t in every build stage

update cargo lock

9604b3e

lower setuptools-rust version for a test

dece095

Merge branch 'main' into add_arrow_datafusion_python

a48e587

Add actions for Rust environment setup

5c141b2

fix workflow syntax issue

605fd92

Add RUSTFMT which output

ded1595

Set rustfmt environment variable and validate it sticks

291f117

Updated Cargo.lock

5800478

Add RUSTFMT to the export script_env variables for the conda build

b4ce6bd

Set default to stable

e8e037e

jdye64 removed the request for review from andygrove March 15, 2023 13:49

jdye64 mentioned this pull request Mar 15, 2023

Execute substrait plans #1041

Open

jdye64 changed the base branch from main to arrow_datafusion_python_refactor March 16, 2023 20:45

jdye64 added 7 commits March 16, 2023 16:46

Merge branch 'arrow_datafusion_python_refactor' into add_arrow_datafu…

b59a7d8

…sion_python

Point to typify version with rustfmt fix, and update conda build file

35bc094

Merge branch 'add_arrow_datafusion_python' of github.com:jdye64/dask-…

4e6228b

…sql into add_arrow_datafusion_python

Point to version of arrow-datafusion-python with the typify rustfmt fix

82f7f97

Refactoring: not from arrow-datafusion-python but general refactoring…

3635939

… needed from updating datafusion version to 20.0.0

Add setuptools-rust back to conda meta.yaml recipe from where I had a…

d41b3cd

…ccidentally deleted it earlier

Remove rust install step and also don't use container amd64/rust

38cfcdf

Updates for testing

24e67bf

jdye64 mentioned this pull request Apr 25, 2023

Bump pyo3 from 0.18.2 to 0.18.3 in /dask_planner #1118

Closed

jdye64 added 4 commits April 26, 2023 09:43

Merge remote-tracking branch 'upstream/main' into add_arrow_datafusio…

febb742

…n_python

Adjustments to fix failing pytests

c74d944

More Pytest fixes and Rust test fixes

dc3a162

Remove mistakenly left inplace breakpoint() calls

43efde1

ayushdg reviewed Apr 26, 2023

View reviewed changes

dask_sql/mappings.py Outdated Show resolved Hide resolved

Return np.timedelta instead of pd.timedelta for interval_month_day_nano

6e8738a

jdye64 marked this pull request as ready for review April 26, 2023 23:12

charlesbluca reviewed Apr 27, 2023

View reviewed changes

ayushdg requested changes Apr 27, 2023

View reviewed changes

dask_planner/.cargo/config.toml Show resolved Hide resolved

dask_planner/Cargo.toml Show resolved Hide resolved

dask_planner/src/sql/logical.rs Outdated Show resolved Hide resolved

dask_planner/src/sql/optimizer.rs Outdated Show resolved Hide resolved

jdye64 and others added 6 commits April 27, 2023 16:27

protoc changes and small formatting review suggestions

7b81fb0

Merge branch 'add_arrow_datafusion_python' of github.com:jdye64/dask-…

479e908

…sql into add_arrow_datafusion_python

Include protoc back in rust.yml for cargo check command

0415b0b

Merge branch 'main' into add_arrow_datafusion_python

1d84460

set avg_days_in_month -> 30. Adjust update_depedencies script to poin…

9deb9f0

…t to arrow-datafusion-python repo

Merge branch 'main' into add_arrow_datafusion_python

1689c5b

jdye64 mentioned this pull request May 2, 2023

TableScan filters to PyArrow DNF format #1130

Merged

jdye64 and others added 5 commits May 2, 2023 18:24

Add back logic for avg_days_in_month

b7f1e72

Merge remote-tracking branch 'origin/main' into add_arrow_datafusion_…

04a638b

…python

Set average days per month to 30

fe179dd

Adjust PyTest expected results to align with new avg_days_in_month of…

e1fd425

… 30 days

merge with upstream

9d76dcc

jdye64 requested review from ayushdg and charlesbluca May 3, 2023 18:49

ayushdg reviewed May 3, 2023

View reviewed changes

Revert timestampdiff changes

d08e4ed

ayushdg approved these changes May 4, 2023

View reviewed changes

charlesbluca approved these changes May 4, 2023

View reviewed changes

ayushdg merged commit 108e024 into dask-contrib:main May 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add arrow datafusion python dependency #1085

Add arrow datafusion python dependency #1085

jdye64 commented Mar 14, 2023

jdye64 commented Mar 15, 2023

charlesbluca left a comment

ayushdg May 3, 2023

ayushdg left a comment

charlesbluca May 4, 2023

ayushdg May 4, 2023

Add arrow datafusion python dependency #1085

Add arrow datafusion python dependency #1085

Conversation

jdye64 commented Mar 14, 2023

jdye64 commented Mar 15, 2023

charlesbluca left a comment

Choose a reason for hiding this comment

ayushdg May 3, 2023

Choose a reason for hiding this comment

ayushdg left a comment

Choose a reason for hiding this comment

charlesbluca May 4, 2023

Choose a reason for hiding this comment

ayushdg May 4, 2023

Choose a reason for hiding this comment