Include datafusion in the benchmark

[kszlim]

Datafusion is another stateless query engine/dataframe library I'd be interested in seeing results for.

https://github.com/apache/arrow-datafusion

[Tmonster]

Hi Kevin, thanks for the suggestion!

I currently don't have a lot of bandwidth to add a whole new solution to the benchmark, but if you want to open a PR that adds the necessary setup-datafusion.sh, ver-datafusion.sh, upg-datafusion.sh, groupby-datafusion.rs, and join-datafusion.rs then I'd be happy to review. Take a look at files in the other solution folders and that should give you a good idea of what is necessary. Although it may require more steps as datafusion doesn't have any R or python APIs, so you may also need to add/modify some files in _launcher and _helpers

See repro.sh for steps to run the benchmark either locally or on an AWS instance. If no errors are thrown for the 0.5GB & 5GB datasets I'd be happy to merge your PR and re-run the benchmark to include results for datafusion.

[kszlim]

There is actually a python api, though it's not documented well:
https://github.com/apache/arrow-datafusion-python

If i have time i'll try to port the benchmarks to it.

[MrPowers]

Looks like almost all of this work is done already: https://github.com/apache/arrow-datafusion/tree/main/benchmarks/db-benchmark

Would you like to add the PR @kszlim or would you like me to take a stab?

[kszlim]

Go ahead, I don't have the time!

[Tmonster]

@MrPowers Was just looking at this again. Looks like the db benchmark for data fusion is here now?
https://github.com/apache/arrow-datafusion-python/tree/main/benchmarks/db-benchmark
Would you still like to open a PR? Some of the files have benchmark initialization setup, so that would need to be trimmed, but I don't think it would be much work

[hkpeaks]

@kszlim I feel interest to include datafusion in coming benchmarking #13 (comment), is it support streaming (data large than memory scenario)?

[kszlim]

@kszlim I feel interest to include datafusion in coming benchmarking #13 (comment), is it support streaming (data large than memory scenario)?

The rust library does, I'm not sure if the python bindings expose it.

[Dandandan]

Closed by #18

[kszlim]

Thanks!