Revert "[SPARK-37670][SQL] Support predicate pushdown and column pruning for de-duped CTEs"

[dongjoon-hyun]

What changes were proposed in this pull request?

This reverts 1a35685 except transformDownWithSubqueries because SPARK-39259 starts to use it.

• [SPARK-39259][SQL] Evaluate timestamps consistently in subqueries #36654
• [SPARK-39259][SQL][FOLLOWUP] Fix source and binary incompatibilities in transformDownWithSubqueries #36765 (Rename transformDownWithSubqueries to transformDownWithSubqueriesAndPruning)

Why are the changes needed?

Currently, branch-3.2 is broken.

• https://github.com/apache/spark/commits/branch-3.2

Here is the AS-IS result generation status.

• Although [SPARK-37670][SQL] Support predicate pushdown and column pruning for de-duped CTEs #34929 generates the result with SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly *PlanStabilitySuite". It causes failure at GitHub Action in branch-3.2.

$ SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly *PlanStabilitySuite" &> /dev/null
$ git status
On branch branch-3.2
Your branch is up to date with 'apache/branch-3.2'.

nothing to commit, working tree clean

• In addition, the result is different when we run the test individually.

$ SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly *PlanStabilitySuite -- -z (tpcds-v1.4/q4)" &> /dev/null
$ SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly *PlanStabilitySuite -- -z (tpcds-v1.4/q5)" &> /dev/null
$ git diff --stat
 sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q4/explain.txt | 238 ++++++++++++++++++++++++++++++++++++++++++++++++++++----------------------------------------------------
 sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q5/explain.txt | 212 ++++++++++++++++++++++++++++++++++++++++++++++----------------------------------------------
 2 files changed, 225 insertions(+), 225 deletions(-)

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CIs.

[dongjoon-hyun]

To reviewers, this will wait for @cloud-fan 's PR to fix the PlanStabilitySuite.

• [SPARK-37670][SQL] Support predicate pushdown and column pruning for de-duped CTEs #34929 (comment)

[dongjoon-hyun]

Gentle ping, @cloud-fan . I hope to recover the branch-3.2 first.

[cloud-fan]

I have created the PR for 3.2: #36828

Can we wait for a few more hours?

[dongjoon-hyun]

Thank you. The patch looks good to me.

I close this PR in favor of #36828