[Epic]: Google Summer of Code 2025 Correlated Subquery Support

[irenjj]

This ticket tracks the progress for a 2025 Google Summer of Code (GSOC) sponsored project on Correlated Subquery Support

Project Documentation

• GSOC proposal: "Supporting Correlated Subqueries in Apache DataFusion"

Is your feature request related to a problem or challenge?

DataFusion currently has limited support for correlated subqueries. This project aims to implement comprehensive support for correlated subqueries in Apache DataFusion by applying Hyper's 'Unnesting Arbitrary
Queries' framework.

Timeline:

Except from the Official GSOC Timeline:

• May 8 - June 1: Community Bonding Period | GSoC contributors get to know mentors, read documentation, get up to speed to begin working on their projects
• June 2: Coding officially begins!
• July 18: Mid term evaluation
• August 25: Final week
• Sep 8: Final evaluations due / wrap up

Work

Epics tracking technical work:

• Support for deeper correlated subqueries: Nested correlated subquery error with a depth exceeding 1 #15558
• Combine the optimization rules decorrelate, decorrelate_lateral_join, and decorrelate_predicate_subquery into one. #16073
• Blog about DataFusion correlated subquery support #16084
• [Epic] Transform Correlated Subquery Into Dependent Join #16173

Other potential future work

• Focus on practical use cases(like tpch-ds([EPIC] Support TPC-DS benchmarks #4763), duckdb subquery test...), in order to identify and list unsupported cases.
• Further break down tasks to address unsupported cases...

Related work:

• General framework to decorrelate the subqueries #5492
• [DISCUSSION] JOIN "task force" / project team #15885

Related documentation

• Improving Unnesting of Complex Queries
• 'Unnesting Arbitrary
  Queries'
• Optimizing SQL (and DataFrames) in DataFusion: Part 2 -- talks about DataFusion join optimization
• Query Optimization Technology for Correlated Subqueries (Alibaba cloud)

Newer research that might be interesting

• Debunking the Myth of Join Ordering: Toward Robust SQL Analytics / Implement optimized Robust Predicate Transfer to improve DuckDB robustness duckdb/duckdb#17326

[alamb]

@irenjj could you attach your GSOC proposal to this issue so it is public? I think you did quite a good job on that and it would be great to make it more public

[alamb]

Let's plan to use this ticket for high level project planning / coordination, and will use other tickets for specific technical work

I update the description on this ticket with some more detail and the GOSC timeline, along with some other related work I know about. If anyone else has suggested readings we can add to the list that would be great

@irenjj and I had a discussion today and covered the following items:

1. As described above, @irenjj plans to first focus on support for multi-level subqueries Nested correlated subquery error with a depth exceeding 1 #15558 (hopefully we'll be able to collaborate with @suibianwanwank and @duongcongtoai for this)
2. It is likely that supporting multiple subquery decorrelation (e.g. arbitrary levels of correlation) will require a more general subquery decorrelation framework (which we will discuss in more detail on related tickets)
3. One of the major lessons for working with DataFusion and open source is figuring out how to work with the broader community (among other things, you never know who is reading an issue or who might be interested in an update)
4. We also hope to write a blog post about correlated subquery support, as the final report: Blog about DataFusion correlated subquery support #16084