DataFusion weekly project plan (Andrew Lamb) - Dec 4, 2023

[alamb]

Follow on to Nov 27 2023:

Boilerplate Overview

The idea of this ticket is make my plans for DataFusion visible, largely for my own personal organizational needs, but also to:

1. Try some different ways to communicate / coordinate in the community
2. Help provide an interesting summary of what is happening in DataFusion this week

It would be great if anyone else who has plans like this for DataFusion could try to make them visible somehow as well

My (personal) plans for this week

• Complete Support general pruning based on <col> = 'const' in PruningPredicate #8376
• Reproduce and file several bugs / limitations we have hit in Influx recently (udpate: Incorrect results due to repartitioning a sorted ParquetExec #8451)

Project Queue (list of future projects)

• Epic: Statistics improvements #8227 (specifically, Consolidate statistics aggregation #8229 as a step towards Introduce a way to represent constrained statistics / bounds on values in Statistics #8078)
• FIFO JoinHashMap for HashJoin #8130, in support of memory explosion in HashJoin outputs: HashJoinStream memory tracking insufficient #7848 with @korowa

Additional major projects I hope to help review and coordinate

• Next steps towards function packages more modular (make all BuiltInScalarFunctions ScalarUDFs): [EPIC] Unify Function Interface (remove BuiltInScalarFunction) #8045 with @2010YOUY01 . This week I hope to get RFC: Demonstrate what a function package might look like -- encoding expressions #8046 mergable, which would require Support Expr creation for ScalarUDF: Resolve function calls by name during planning #8157)
• ARRAY function hardening: [Epic] General ticket for the concept of the practical implementation of ARRAY #6980 / Lots of code duplication in array_functions (array_replace_all, array_replace_n, array_replace, etc) #7988
• Review any new ticket / bug reports

Algorithm for (my) prioritizing PR reviews

Note there are many committers who can review and merge PRs as well, so this is not the priorities of the project as a whole, just the approximate algorithm I am using

Priority:

1. Bug fixes (where something is just incorrect), especially regressions (where it used to work and now does not)
2. Improvements directly related to features needed for InfluxDB (as I am employed by them)
3. Documentation and test improvements (I view these as very strategically important)
4. PRs that I think are strategically important
5. Other new features / additions to functionality

The current strategically important projects in my head are:

• [EPIC] Unify Function Interface (remove BuiltInScalarFunction) #8045
• Improving performance of high cardinality aggregation

Note that adding new array functionality (e.g. #6980 and its children) is very low on my priority list, not because I don't think they are valuable, but because I think they are less valuable than the other projects going on - it would be great if other people could help pick these up too.

Thus, if you are interested in contributing to DataFusion and are interested in a fast turn around time I would recommend looking into bug fixes / test improvements / documentation / etc.

If you propose adding new functionality, the review cycle will likely be longer. You can make it a shorter cycle by looking at the comments on other recent PRs and following the same model (e.g. ensure there are tests in sqllogictest for example)

[alamb]

Review Queue:

DataFusion

• feat: ScalarValue from String #8411 (and followup Minor: Use ScalarValue::from impl for strings #8429)
• Union schema can't be a subset of the child schema #8408
• feat: support LargeList for array_has, array_has_all and array_has_any #8322
• Update sqlparser requirement from 0.39.0 to 0.40.0 #8338
• Remove macro in iter_to_array for List #8414
• Minor: convert marcro list-slice and slice to function #8424
• Fix ambiguous reference when aliasing in combination with ORDER BY #8425
• Support crossjoin in substrait. #8427
• feat: customize column default values for external tables #8415 (review)
• Support parquet_metadata for datafusion-cli #8413 (comment)
• fix: Literal in ORDER BY window definition should not be an ordinal referring to relation column #8419
• Detect when filters on unique constraints make subqueries scalar #8312
• feat: Support array_sort(list_sort) #8279
• Fix PartialOrd for ScalarValue::List/FixSizeList/LargeList #8253 (review)

[alamb]

Review Queue:

DataFusion:

• Minor: convert marcro list-slice and slice to function #8424
• Remove macro in iter_to_array for List #8414
• Bugfix: Remove df-cli specific SQL statment options before executing with DataFusion #8426
• Add alias check to optimize projections merge #8438
• Support parquet_metadata for datafusion-cli #8413
• Support array_distinct function. #8268

Arrow

• Replace ScalarBuffer in Parquet with Vec (#1849) (#5177) arrow-rs#5178

sqlparser-rs:

• add support for CALL statements datafusion-sqlparser-rs#1063
• add support for JSON_TABLE datafusion-sqlparser-rs#1062

[alamb]

DataFusion:

• Remove define_array_slice and reuse array_slice for array_pop_front/back #8401
• DRAFT: Resolve function calls by name during planning  #8447
• Minor: refactor trim to clean up duplicated code #8434
• fix: RANGE frame for corner cases with empty ORDER BY clause should be treated as constant sort #8445
• Split EmptyExec into PlaceholderRowExec #8446
• flx clippy warnings #8455
• Minor: update cast #8458 (review)
• Support parquet_metadata for datafusion-cli #8413
• Support array_distinct function. #8268
• Preserve dict_id on Field during serde roundtrip #8457
• feat: support InterleaveExecNode in the proto #8460
• Minor: refactor data_trunc to reduce duplicated code #8430
• fix: ORDER BY window definition should work on null literal #8444
• Minor: Add multi ordering test for array agg order #8439
• Enable non-uniform field type for structs created in DataFusion #8463

Arrow

• Replace ScalarBuffer in Parquet with Vec (#1849) (#5177) arrow-rs#5178
• Simplify parquet statistics generation arrow-rs#5183
• Remove SIMD Feature arrow-rs#5184

sqlparser-rs:

• add support for CALL statements datafusion-sqlparser-rs#1063
• add support for JSON_TABLE datafusion-sqlparser-rs#1062

[alamb]

DataFusion:

• Remove define_array_slice and reuse array_slice for array_pop_front/back #8401
• DRAFT: Resolve function calls by name during planning  #8447
• Minor: refactor trim to clean up duplicated code #8434
• Split EmptyExec into PlaceholderRowExec #8446
• Support array_distinct function. #8268
• Minor: refactor data_trunc to reduce duplicated code #8430
• Minor: Add multi ordering test for array agg order #8439
• Enable non-uniform field type for structs created in DataFusion #8463
• Move Coercion for MakeArray to coerce_arguments_for_signature and introduce another one for ArrayAppend #8317 (review)

Arrow

• Replace ScalarBuffer in Parquet with Vec (#1849) (#5177) arrow-rs#5178
• Simplify parquet statistics generation arrow-rs#5183
• Remove SIMD Feature arrow-rs#5184
• Blockwise IO in IPC FileReader (#5153) arrow-rs#5179
• Use Vec in ColumnReader (#5177) arrow-rs#5193

sqlparser-rs:

• add support for CALL statements datafusion-sqlparser-rs#1063
• add support for JSON_TABLE datafusion-sqlparser-rs#1062

[alamb]

Next week tracked in #8490