Description
The idea of this ticket is make my plans for DataFusion visible, largely for my own personal organizational needs, but also to:
- Try some different ways to communicate / coordinate in the community
- Help provide an interesting summary of what is happening in DataFusion this week
It would be great if anyone else who has plans like this for DataFusion could try to make them visible somehow as well
Plans for this week personally:
- sqlparser release: Release version
0.40.0datafusion-sqlparser-rs#1030 - Epic: Statistics improvements #8227 (specifically, Consolidate statistics aggregation #8229 as a step towards Introduce a way to represent constrained statistics / bounds on values in Statistics #8078)
- Final edits + draft submission for Write DataFusion paper for (SIGMOD / VLDB / ICDE) #6782 with @ozankabak @Dandandan @yjshen and @JayjeetAtGithub @sunchao and @viirya
- File follow on tickets towards making function packages more modular: [EPIC] Unify Function Interface (remove
BuiltInScalarFunction) #8045
My plans for next week
- FIFO
JoinHashMapforHashJoin#8130, in support of memory explosion in HashJoin outputs:HashJoinStreammemory tracking insufficient #7848 with @korowa
Major projects I intend to help review and coordinate
- Next steps towards function packages more modular (make all BuiltInScalarFunctions ScalarUDFs): [EPIC] Unify Function Interface (remove
BuiltInScalarFunction) #8045 with @2010YOUY01 -
ARRAYfunction hardening: [Epic] General ticket for the concept of the practical implementation ofARRAY#6980 / https://github.com/apache/arrow-datafusion/issues/7988with @jayzhan211 @Veeupup @edmondop and @xudong963 - Review any new ticket / bug reports
Algorithm for (my) prioritizing PRs
Note there are many committers who can review and merge PRs as well, so this is not the priorities of the project as a whole, just the approximate algorithm I am using
Priority:
- Bug fixes (where something is just incorrect), especially regressions (where it used to work and now does not)
- Improvements directly related to stuff needed for InfluxDB (as I am employed by them)
- Documentation and test improvements (I view these as very strategically important)
- PRs that I think are strategically important
- Other new features / additions to functionality
The current strategically important projects in my head are:
- Support User Defined Table Functions / Table Value Functions #7926
- [EPIC] Unify Function Interface (remove
BuiltInScalarFunction) #8045 - Improving performance of high cardinality aggregation
Note that adding new array functionality (e.g. #6980 and its children) is very low on my priority list, not because I don't think they are valuable, but because I think they are less valuable than the other projects going on
Thus, if you are interested in contributing to DataFusion and are interested in a fast turn around time I would recommend looking into bug fixes / test improvements.
If you propose adding new functionality, the review cycle will be longer. You can make it a shorter cycle by looking at the comments on other recent PRs and following the same model (e.g. ensure there are tests in sqllogictest for example)