Skip to content

[SPARK-47527][SQL] Normalize common expression ids during canonicalization of With expression #45677

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 11 commits into from

Conversation

bersprockets
Copy link
Contributor

What changes were proposed in this pull request?

Add canonicalization logic to the With expression to normalize common expression ids in the CommonExpressionDef and CommonExpressionRef instances.

Why are the changes needed?

This change increases the chance of a cache hit for a query using the nullif, between, or count_if expressions.

For example, without the fix the following does not use the cache:

create or replace temp view v1 as
select id from range(10);

create or replace temp view q1 as
select * from v1
where id between 2 and 4;

cache table q1;

explain select * from q1;

== Physical Plan ==
*(1) Filter ((id#51L >= 2) AND (id#51L <= 4))
+- *(1) Range (0, 10, step=1, splits=8)

Does this PR introduce any user-facing change?

No.

How was this patch tested?

New tests.

Was this patch authored or co-authored using generative AI tooling?

No.

@github-actions github-actions bot added the SQL label Mar 23, 2024
@bersprockets
Copy link
Contributor Author

Looking at open PRs, this seems to be a dup of #45649. Will close for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant