DataFusion weekly project plan (Andrew Lamb) - Nov 20, 2023

[alamb]

The idea of this ticket is make my plans for DataFusion visible, largely for my own personal organizational needs, but also to:

1. Try some different ways to communicate / coordinate int he community
2. Help provide an interesting summary of what is happening in DataFusion this week

I request anyone else who has plans like this for DataFusion could try something like this (maybe tickets are the best place 🤔 )

Plans for this week personally:

• Epic: Statistics improvements #8227 (specifically, Consolidate statistics aggregation #8229 as a step towards Introduce a way to represent constrained statistics / bounds on values in Statistics #8078)
• Edits to paper (improve language and get it under the page limit) Write DataFusion paper for (SIGMOD / VLDB / ICDE) #6782 with @ozankabak @Dandandan @yjshen and @JayjeetAtGithub @sunchao and @viirya
• sqlparser release: Release version 0.40.0 datafusion-sqlparser-rs#1030

My plans for next week

• FIFO JoinHashMap for HashJoin #8130, in support of memory explosion in HashJoin outputs: HashJoinStream memory tracking insufficient #7848 with @korowa

Major projects I intend to help review and coordinate

• Make function packages more modular (make all BuiltInScalarFunctions ScalarUDFs): [EPIC] Unify Function Interface (remove BuiltInScalarFunction) #8045 with @2010YOUY01 (help Refactor: Unify Expr::ScalarFunction and Expr::ScalarUDF, introduce unresolved functions by name #8258)
• ARRAY function hardening: [Epic] General ticket for the concept of the practical implementation of ARRAY #6980 / https://github.com/apache/arrow-datafusion/issues/7988with @jayzhan211 @Veeupup @edmondop and @xudong963
• Review any new ticket / bug reports

[alamb]

REview Queue

Arrow-rs

• Use Total Ordering for Aggregates and Refactor for Better Auto-Vectorization arrow-rs#5100

DataFusion

• Don't Canonicalize Filesystem Paths in ListingTableUrl / support new external tables for files that do not (yet) exist #8014
• Refactor: Unify Expr::ScalarFunction and Expr::ScalarUDF, introduce unresolved functions by name #8258
• consistent logical & physical NTILE return types #8270
• Fix Display for List #8261 (review)
• Replace array_has/array_has_all/array_has_any macro to remove duplicate code #8263
• Refactor Interval Arithmetic Updates #8276 (4k+ lines)

sqlparser-rs

• Support CONVERT expressions datafusion-sqlparser-rs#1048
• Support global and session parts in show variables for mysql and generic dialects datafusion-sqlparser-rs#1032
• Add support for release and rollback to savepoint syntax datafusion-sqlparser-rs#1045
• Support INSERT INTO ... DEFAULT VALUES ... datafusion-sqlparser-rs#1036
• Add support for generated virtual columns with expression datafusion-sqlparser-rs#1051
• add {pre,post}_visit_query to Visitor datafusion-sqlparser-rs#1044

[alamb]

Review Queue

DataFusion

• [MINOR]: Remove unecessary orderings from the final plan #8289
• Fixing issues with for timestamp literals #8193 (comment)
• support LargeList for arrow_cast, support ScalarValue::LargeList #8290 (review)
• make array_union/array_except/array_intersect handle empty/null arrays rightly #8269 (comment)

[alamb]

DataFusion

• Fix Display for List #8261 (comment)
• Fixing issues with for timestamp literals #8193 (review)
• Calculate ordering equivalence for expressions (rather than just columns) #8281 (1k+ lines)
• Support no distinct aggregate sum/min/max in single_distinct_to_group_by rule #8266 (review)
• Materialize dictionaries in group keys #8291
• Support User Defined Table Function #8306
• Support array_distinct function. #8268
• Fix incorrect results for NOT IN subqueries with nulls #8271
• Add support for recursive CTEs #7581
• Projection Pushdown over StreamingTableExec #8299
• Improve type coercion and casting #8302

sqlparser-rs

• add {pre,post}_visit_query to Visitor datafusion-sqlparser-rs#1044
• Update sqlparser-derive to use syn 2.0 datafusion-sqlparser-rs#1040

[alamb]

Next week's work: #8329