[SPARK-5642] [SQL] Apply column pruning on unused aggregation fields

[adrian-wang]

select k from (select key k, max(value) v from src group by k) t

[SparkQA]

Test build #26889 has started for PR 4415 at commit ce99ea1.

• This patch merges cleanly.

[SparkQA]

Test build #26889 has finished for PR 4415 at commit ce99ea1.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26889/
Test FAILed.

[SparkQA]

Test build #26898 has started for PR 4415 at commit b6420cb.

• This patch merges cleanly.

[SparkQA]

Test build #26898 has finished for PR 4415 at commit b6420cb.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26898/
Test PASSed.

[rxin]

Is it possible to add a unit test?

[rxin]

(I understand unit test coverage for the optimizer is pretty low - but that would be great to change & increase)

[SparkQA]

Test build #27080 has started for PR 4415 at commit 8f5c86d.

• This patch merges cleanly.

[SparkQA]

Test build #27080 has finished for PR 4415 at commit 8f5c86d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27080/
Test PASSed.

[marmbrus]

I'd follow the convention from the rest of the optimizer and use longer variable names (i.e. projectList, groupExprs, aggExprs)

[SparkQA]

Test build #27163 has started for PR 4415 at commit 2cd88ff.

• This patch does not merge cleanly.

[SparkQA]

Test build #27165 has started for PR 4415 at commit 2916ec9.

• This patch merges cleanly.

[SparkQA]

Test build #27163 has finished for PR 4415 at commit 2cd88ff.

• This patch passes all tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27163/
Test PASSed.

[SparkQA]

Test build #27165 has finished for PR 4415 at commit 2916ec9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27165/
Test PASSed.

[SparkQA]

Test build #27430 has started for PR 4415 at commit 5d2d8a3.

• This patch merges cleanly.

[SparkQA]

Test build #27430 has finished for PR 4415 at commit 5d2d8a3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27430/
Test PASSed.

[adrian-wang]

@rxin @marmbrus

[marmbrus]

Thanks! I'm going to go ahead and add one more test case while merging since I wasn't sure if this case was handled properly (it is :) )

Merging to master and 1.3

[marmbrus]

  test("column pruning for group with alias") {
    val originalQuery =
      testRelation
        .groupBy('a)('a as 'c, Count('b))
        .select('c)

    val optimized = Optimize(originalQuery.analyze)
    val correctAnswer =
      testRelation
        .select('a)
        .groupBy('a)('a as 'c)
        .select('c).analyze

    comparePlans(optimized, correctAnswer)
  }