Weekly Plan (Andrew Lamb) April 7, 2025

[alamb]

This is an attempt to organize myself and make what I plan to work on more visible

Weekly High Level Goals (in order)

• Arrow release: Release arrow-rs / parquet major version 55.0.0 (Apr 2025) arrow-rs#7084
• Start working on testing the 47.0.0 release: Release DataFusion 47.0.0 (April 2025) #15072
• TopK Pushdown Dynamic pruning filters from TopK state (optimize ORDER BY LIMIT queries) #15037 with @adriangb
• Avoid resorting/merging [EPIC] Avoid sort for already sorted Parquet files that do not overlap values on condition #6672 with @wiedld @suremarc and @xudong963
• Get Enable parquet filter pushdown (filter_pushdown) by default #3463 ready for merge with @XiangpengHao
• Work on integrating tpch data generator with @clflushopt : Make it easier to run TPCH queries with datafusion-cli #14608

Other projects I plan to review

• Bug fixes
• Performance improvements
• Complete insta test migration [Epic] Add snapshot tests (migrate to insta for tests) #15178 with @blaginin @shruti2522 @qstommyshu and others
• Hardening external sorts: A complete solution for stable and safe sort with spill #14692 with @2010YOUY01
• Set up Spark function library pattern: feat: Add datafusion-spark crate #15168 with @shehabgamin and @andygrove
• Use UTF8 view by default Change mapping of SQL VARCHAR from Utf8 to Utf8View #15096 with @zhuqi-lucas

Background

I am putting this list on github because:

1. I like how github renders checklists w/ PR titles so it is easy to track (I currently have a local text file...)
2. I thought others might be interested from seeing what I am doing / planning to do
3. It makes me feel better that I don't have time to review all the PRs 😭

The way I am trying to prioritize PRs is in the following order

1. Bug fixes
2. Documentation / UX / API improvements (things that make DataFusion easier/better to work with)
3. Performance improvements
4. New features with wide appeal
5. New functions

Note new features and functions are deliberately at the bottom

[alamb]

Review queue (docs):

• datafusion-python 46.0.0 announcement datafusion-site#65
• User defined window functions blog post  datafusion-site#66

Review queue (perf):

• ExecutionPlan: add APIs for filter pushdown & optimizer rule to apply them #15566
• Add short circuit evaluation for AND and OR #15462
• Enhance: simplify x=x --> x IS NOT NULL OR NULL #15589
• Introduce DynamicFilterSource and DynamicPhysicalExpr #15568
• perf: Introduce sort prefix computation for early TopK exit optimization on partially sorted input (10x speedup on top10 bench) #15563
• fix!: incorrect coercion when comparing with string literals #15482
• Improve performance of last_value by implementing special GroupsAccumulator #15542
• Introduce selection vector repartitioning #15423

Review queue (stats / sorting):

• Add statistics_by_partition API to ExecutionPlan #15503
• Fix: after repartitioning, the PartitionedFile and FileGroup statistics should be inexact/recomputed #15539

Review queue (sorting / memory):

• feat: Add unique id for every memory consumer #15613
• feat: add multi level merge for sorting #15608 @rluvaton
• Cascaded spill merge and re-spill #15610 @2010YOUY01

Review queue (features):

• fix: add map coercion for binary ops #15551
• [ ]

[alamb]

I am going to be focued this week on working down the DataFusion review queue and pushing along things in arrow and I will be out next week, so I am not going to file another weekly plan ticket for this week