[DISCUSSION] DataFusion Road Map: Q3-Q4 2025

[alamb]

I suggest we use this ticket to discuss and coordinate the "roadmap" for DataFusion over the next 3-6 months

DataFusion is a community project, and unlike many other open source projects there is no "DataFusion Company" that pays contributors and determines their priority. The project's evolution and priorities is determined by those community members able and willing to devote their time and energy to driving the project forward

In the past it has been helpful to have a public discussion about features in order to:

1. Find others interested in features you may be thinking of working on
2. See who is actively looking for certain features / capabilities

So please feel free to post comments about:

1. Features you plan to work on
2. Features you would like others to work on
3. Features you are willing / able / planning to help others with

See previous discussions:

• [DISCUSSION] 2025 Q1-Q2 Roadmap #14580
• [DISCUSSION] 2024 Q4 / 2025 Q1 Roadmap #13274

[alamb]

My plans:

🚀 performance (dynamic filter pushdown)

• parquet predicate pushdown with @XiangpengHao @zhuqi-lucas Enable parquet filter pushdown by default #3463
• dynamic filter pushdown with @adriangb and @berkaysynnada: Dynamic pruning filters from TopK state (optimize ORDER BY LIMIT queries) #15037

Variant / semi structured data support

I plan to help @PinkCrow007, and others work on variant. This likely requires doing so in arrow-rs / parquet and then moving on to features in datafusion like expression pushdown and user defined types

• [EPIC] [Parquet] Implement Variant type support in Parquet arrow-rs#6736
• Support Push down expression evaluation in TableProviders #14993
• Extension Types / User Defined Types #12644

I also plan to continue releases, help review code for important projects, blog writing, etc

[Rachelint]

My list (still mainly about aggregation performance, it has falled much behind duckdb on clickbench...):

• Push forward [EPIC] Improve aggregate performance with adaptive sizing in accumulators / avoiding reallocations in accumulators #7065 , which has been proved to be really promising in poc (10%~20% faster for related queries in clickbench).
  The sketch pr Implement intermediate result blocked approach to aggregation memory management #15591 is already ready for review, after this big one merged. The implementations of GroupValuess and GroupAccumulators can be worked in parallel, and push it forward fast.

• Optimizations on optimizer side, such as Linear Aggregate Functions Optimization #15633 and POC: Eliminate unnecessary group by keys (q35 in clickbench 1.35x faster) #13617

• Help push forward other working promising opmizations such as Support zero copy hash repartitioning for Hash Aggregate #15383 and Enable parquet filter pushdown by default #3463

[alamb]

My list (still mainly about aggregation performance, it has falled much behind duckdb on clickbench...):

Welcome back!

[jonathanc-n]

I would like to help push forward the CMU effort for variant types.

• I would also like to try to get this back up: support simple/cross lateral joins #14595 (@skyzh would you be interested in continuing to work on this?)
• Implementing Merge statement: POC: Parse to Merge Logical Plan #15862 (would be great if anyone can take a look)

[skyzh]

I think #14595 is in a decent shape and could be merged :) Though it cannot unnest all queries and might produce wrong result for some lateral joins, that would be a good starting point for us to slowly add lateral join and full unnesting support to the codebase.

[alamb]

• I am also trying to help organize a join task force for planning joins / subqueries: [DISCUSSION] JOIN "task force" / project team #15885

[alamb]

I think #14595 is in a decent shape and could be merged :) Though it cannot unnest all queries and might produce wrong result for some lateral joins, that would be a good starting point for us to slowly add lateral join and full unnesting support to the codebase.

@skyzh -- I think the conversation on #5492 might be relevant to you as well

[comphead]

• I'll try to start with Add Memory Profiling Functionality #14510

[xudong963]

My lists:

• Work with @suremarc @alamb and @wiedld to make Optimized SPM cross the finish line
• Finish all my legacy PRs: https://github.com/apache/datafusion/pulls/xudong963
• Start to work on the dynamic filter for join Push Dynamic Join Predicates into Scan ("Sideways Information Passing", etc) #7955, it's helpful for my team
• Speed up logical optimizer, there is a tracking issue: Tracking: speed up the logical optimizer #15775
• Rich statistics tests
• Releases, help review code, etc

[Omega359]

My list includes:

• feat: Add ConfigOptions to ScalarFunctionArgs #13527 (finally finish this)
• Introduce Async User Defined Functions  #14837 (adopt my async UDF's to use this and validate)
• physical-expr: Nullability of Literal is not determined by surrounding context #15394
• [EPIC] Additional Date/Time related open issues #8282 (Specifically allowing changing default tz via config and UDF's/etc properly handling tz data ).
• look into adding an 'ANSI' mode that when disabled causes null to be returned instead of an error. Useful especially when parsing external date/timestamps from strings.

[Dandandan]

I am currently interested in the following subjects where I'll probably experiment with some things or help out others.

• Window Functions (profiling, implementing improvements / optimizations)

1. ** Vectorize window functions #15607
2. Possible (logical plan) optimizations - limit pushdown?

• Sorting performance ideas / helping out, examples:

1. Reuse Rows allocation in SortPreservingMergeStream / RowCursorStream #15720
2. Perf: Optimize in memory sort #15380 (review)

• Streamlining arrow-rs kernels in terms of speed, consistency, reduce use of unsafe, etc.