[SPARK-29947][SQL][followup] ResolveRelations should return relations with fresh attribute IDs

[cloud-fan]

What changes were proposed in this pull request?

This is a followup of #26589, which caches the table relations to speed up the table lookup. However, it brings some side effects: the rule ResolveRelations may return exactly the same relations, while before it always returns relations with fresh attribute IDs.

This PR is to eliminate this side effect.

Why are the changes needed?

There is no bug report yet, but this side effect may impact things like self-join. It's better to restore the 2.4 behavior and always return refresh relations.

Does this PR introduce any user-facing change?

no

How was this patch tested?

N/A

[cloud-fan]

@wangyum

[cloud-fan]

also cc @HyukjinKwon @brkyvz

[maropu]

Looks resonable changes

[HyukjinKwon]

LGTM

[SparkQA]

Test build #123472 has finished for PR 28717 at commit 6924118.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

retest this please

[SparkQA]

Test build #123485 has finished for PR 28717 at commit 6924118.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[brkyvz]

@cloud-fan Can you please add unit test? A triple self-join should be able to reproduce this issue

[brkyvz]

Why not just:

Option(AnalysisContext.get.relationCache.getOrElseUpdate(key, loaded.orNull)).map { rel =>
  rel.transform { case multi: MultiInstanceRelation => multi.newInstance() }
} 

[cloud-fan]

sorry just saw your comment after running the merge script.

We only need to refresh the attributes if we get the relation from the cache. Otherwise, it's already a fresh relation.

[brkyvz]

that is correct, but IMO it's a little cleaner. Creating new attributes should be super cheap, and not worth having a orElses and foreachs.

[cloud-fan]

thanks, merging to master/3.0!

[cloud-fan]

@brkyvz I tried triple self-join but can't find a buggy case:

sql("select * from t t1 join t t2 join t t3 where t1.a = t3.a").show

sql("select * from t t1 join t t2 on t1.b = t2.a join t t3 on t1.a = t3.a").show

sql("select * from t t1 join t t2 on t1.b = t2.a join t t3 on t2.a = t3.a").show

Please let me know if you see some broken join queries.

[brkyvz]

what if you do something like:

1. join t with t on a
2. Do a group by to get distinct values of a and b
3. Join grouped results again with t on b
   so that the expression Ids pass through

[cloud-fan]

sql("select * from (select t1.a, t1.b from t t1 join t t2 on t1.a = t2.a group by t1.a, t1.b) j join t t3 where j.b = t3.b").show

This also works. Please let me know if I miss something, thanks!