[SPARK-21351][SQL] Remove the UpdateAttributeNullability rule from the optimizer

[maropu]

What changes were proposed in this pull request?

This pr removed UpdateAttributeNullability from the optimizer because the same logic happens in the analyzer. See SPARK-26459(#23390) for more detailed discussion.

How was this patch tested?

N/A

[SparkQA]

Test build #101017 has finished for PR 23508 at commit c0bcfae.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[maropu]

@cloud-fan @gatorsmile It seems the removal don't affect the existing tests now.

[cloud-fan]

Shall we update GetArrayItem.nullable, to special-handle the combination of CreateArray and Literal?

[maropu]

Ah, I see, we can do so. Is it ok to file a separate jira and make a pr for that in following activities?

[cloud-fan]

sure

[maropu]

Made a pr in #23566

[dongjoon-hyun]

Retest this please.

[SparkQA]

Test build #103226 has finished for PR 23508 at commit c0bcfae.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[maropu]

ping @cloud-fan @gatorsmile @dongjoon-hyun

[dongjoon-hyun]

Hi, @maropu .
Based on the our discussion history and the recent nullability fixes, we decided to remove this from the optimizer step.

As a the final piece of this removal, could you remove the following UpdateAttributeNullability optimizer description? Otherwise, someone may try to add it back.

• https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UpdateAttributeNullability.scala#L32-L34

 *
 * This rule should be executed again at the end of optimization phase, as optimizer may change
 * some expressions and their nullabilities as well. See SPARK-21351 for more details.

[maropu]

oh, yeah. I'll do that.

[SparkQA]

Test build #103285 has finished for PR 23508 at commit 9f94605.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

thanks, merging to master!