[SPARK-53733][SQL] Delay resolveColsLastResort until all previous UnresolvedAliases are resolved
#52471
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cloud-fan
reviewed
Sep 30, 2025
mihailotim-db
force-pushed
the
mihailo-timotic_data/fix_lca_correctness
branch
2 times, most recently
from
174d1f1 to
a696d0d
Compare
mihailoale-db
approved these changes
Sep 30, 2025
sql/core/src/test/resources/sql-tests/inputs/column-last-resort-resolution-precedence.sql
Show resolved
Hide resolved
mihailotim-db
force-pushed
the
mihailo-timotic_data/fix_lca_correctness
branch
from
a696d0d to
ff85dcd
Compare
| SELECT 'a' AS a, a; | ||
|
|
||
| SELECT 'a', a FROM VALUES(1) AS t(a); | ||
| SELECT 'a' AS a, a FROM VALUES(1) AS t(a); |
There was a problem hiding this comment.
Can you please use a view instead of VALUES? The lines would get shorter.
| SELECT 'a' AS a, a; | ||
|
|
||
| SELECT 'a', a FROM VALUES(1) AS t(a); | ||
| SELECT 'a' AS a, a FROM VALUES(1) AS t(a); |
There was a problem hiding this comment.
I think arguments for CREATE FUNCTION might also be affected - they are resolved after LCAs and before session variables
| SELECT 'a', a FROM VALUES(1) AS t(a); | ||
| SELECT 'a' AS a, a FROM VALUES(1) AS t(a); | ||
|
|
||
| SELECT col1 FROM VALUES(1) WHERE EXISTS (SELECT 'col1', col1); |
There was a problem hiding this comment.
We have the logic for Project, but the logic for Aggregate is either missing or non-obvious.
We need the same tests with with GROUP BY.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Delay
resolveColsLastResortuntil allUnresolvedAliases that come before the column that is being resolved, are resolvedWhy are the changes needed?
For the follwing query:
Spark incorrectly resolves the second
acolumn to the variable instead of resolving it as a lateral column alias reference to the implicit alias of literal 'a'. This is not consistent with the current intended behavior and name resolution precedence in Spark:Similarly, the fix applies to precedence of LCAs over outer references as in this query:
Does this PR introduce any user-facing change?
Yes, user now sees the correct result
How was this patch tested?
Added golden file tests for affected queries.
Was this patch authored or co-authored using generative AI tooling?
No