feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations by mkleen · Pull Request #19930 · apache/datafusion

mkleen · 2026-01-21T17:07:20Z

Which issue does this PR close?

Cleaned-up version of #18806 with:

Removed outer_queries_schema from PlannerContext
Planning logic only (optimizer modifications removed)
sql logic tests moved to sql_integration.rs
Closes Supporting planning (binding) Nested correlated subquery error with a depth exceeding 1 #19816

Rationale for this change

See #18806

What changes are included in this PR?

See #18806

Are these changes tested?

Yes

Are there any user-facing changes?

outer_queries_schema is removed from PlannerContext.

mkleen · 2026-01-21T17:18:11Z

datafusion/sql/tests/sql_integration.rs

+    Projection: l.a, l.b, l.c, l.d, l.e
      SubqueryAlias: l
-        Projection: lineitem.l_item_id AS a, lineitem.l_description AS b, lineitem.price AS c
+        Projection: lineitem.l_orderkey AS a, lineitem.l_item_id AS b, lineitem.l_description AS c, lineitem.l_extendedprice AS d, lineitem.price AS e


This is because the lineitem table is extended now.

mkleen · 2026-01-21T17:18:50Z

datafusion/sql/tests/sql_integration.rs

+        "o1.o_custkey".to_string(),
+        "o1.o_orderstatus".to_string(),
        "o1.customer_id".to_string(),
+        "o1.o_totalprice".to_string(),


This is because the order table is extended now.

mkleen · 2026-01-21T17:19:40Z

datafusion/sql/tests/sql_integration.rs

    assert_snapshot!(
        err.strip_backtrace(),
-        @"Error during planning: Source table contains 3 columns but only 2 names given as column alias"
+        @"Error during planning: Source table contains 5 columns but only 2 names given as column alias"


This is because the lineitem table is extended now.

mkleen · 2026-01-21T17:20:16Z

datafusion/sql/tests/common/mod.rs

            ])),
            "orders" => Ok(Schema::new(vec![
                Field::new("order_id", DataType::UInt32, false),
+                Field::new("o_orderkey", DataType::UInt32, false),


These extensions were necessary to get the tests to work.

## Which issue does this PR close? While working on #19930 , I noticed a trailing whitespace in the CROSS JOIN logical plan output. This whitespace is inconsistent with the rest of the logical plan formatting. ## Rationale for this change This change removes the unnecessary trailing whitespace in the logical plan output of a CROSS JOIN. ## What changes are included in this PR? ## Are these changes tested? Yes. ## Are there any user-facing changes? No.

mkleen · 2026-01-26T08:49:55Z

@alamb @duongcongtoai Could you review this when you have a moment so we can get it merged?

duongcongtoai · 2026-01-28T03:15:47Z

nice, i'll take a look once i have time

duongcongtoai · 2026-02-05T01:27:50Z

datafusion/sql/tests/sql_integration.rs

 fn table_with_column_alias() {
-    let sql = "SELECT a, b, c
-                   FROM lineitem l (a, b, c)";
+    let sql = "SELECT a, b, c, d, e


would be nice to have a sqllogictest for this PR

ah i see you replaced them with integration test instead

duongcongtoai · 2026-02-05T01:34:51Z

datafusion/expr/src/logical_plan/invariants.rs

                        // TODO revisit this validation logic
                        plan_err!(
-                            "Correlated scalar subquery in the GROUP BY clause must also be in the aggregate expressions"
+                            "Correlated scalar subquery in the GROUP BY clause must \


nit: this change is unrelated (it's in my original PR)

mkleen · 2026-02-05T08:00:03Z

@duongcongtoai Thank you for your feedback!

github-actions bot added sql SQL Planner logical-expr Logical plan and expressions sqllogictest SQL Logic Tests (.slt) and removed sqllogictest SQL Logic Tests (.slt) labels Jan 21, 2026

mkleen changed the title ~~Plann recursive subquery cleanup~~ feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations Cleanup Jan 21, 2026

mkleen commented Jan 21, 2026

View reviewed changes

mkleen mentioned this pull request Jan 22, 2026

Fix trailing whitespace in CROSS JOIN logical plan formatting #19936

Merged

mkleen changed the title ~~feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations Cleanup~~ feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations Jan 23, 2026

mkleen force-pushed the plann-recursive-subquery-cleanup branch from 99902bc to 3b1fa64 Compare January 23, 2026 08:51

duongcongtoai and others added 16 commits January 26, 2026 09:52

fix: allow OuterRefColumn for non-adjacent outer relation

d5a37c0

fix: accidentally pushdown filter with subquery

4ca4284

chore: clippy

edcded4

chore: rm debug details

c3e08d8

fix: breaking changes

dc1c228

fix: lateral join losing its outer ref columns

477bb3c

test: more test case for other decorrelation

112d98e

doc: better comments

fc39d8c

fix: test

00bf97b

Remove outer_query_schema from planner

7c3040b

Add new tests

08e928c

fixup! Add new tests

333bd88

fixup! fixup! Add new tests

7a968f4

Fix fmt

ce062a1

Remove test from joins.slt

8206a9b

Remove whitespace in CROSS JOIN output

cd81a28

mkleen force-pushed the plann-recursive-subquery-cleanup branch from af87772 to cd81a28 Compare January 26, 2026 08:52

Merge branch 'main' into plann-recursive-subquery-cleanup

47b421a

alamb mentioned this pull request Feb 2, 2026

Andrew Lamb Weekly-ish Open Source plan - 2026-02-01 #20108

Open

10 tasks

duongcongtoai reviewed Feb 5, 2026

View reviewed changes

duongcongtoai approved these changes Feb 5, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations #19930

feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations #19930
mkleen wants to merge 17 commits intoapache:mainfrom
mkleen:plann-recursive-subquery-cleanup

mkleen commented Jan 21, 2026 •

edited

Loading

Uh oh!

mkleen Jan 21, 2026 •

edited

Loading

Uh oh!

mkleen Jan 21, 2026

Uh oh!

mkleen Jan 21, 2026

Uh oh!

mkleen Jan 21, 2026 •

edited

Loading

Uh oh!

mkleen commented Jan 26, 2026

Uh oh!

duongcongtoai commented Jan 28, 2026

Uh oh!

duongcongtoai Feb 5, 2026

Uh oh!

duongcongtoai Feb 5, 2026

Uh oh!

duongcongtoai Feb 5, 2026

Uh oh!

mkleen commented Feb 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mkleen commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

mkleen Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mkleen Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

mkleen Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

mkleen Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mkleen commented Jan 26, 2026

Uh oh!

duongcongtoai commented Jan 28, 2026

Uh oh!

duongcongtoai Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

duongcongtoai Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

duongcongtoai Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

mkleen commented Feb 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mkleen commented Jan 21, 2026 •

edited

Loading

mkleen Jan 21, 2026 •

edited

Loading

mkleen Jan 21, 2026 •

edited

Loading