Skip to content

Commit 77cc25f

Browse files
pzzsrxin
authored andcommitted
[SPARK-7267][SQL]Push down Project when it's child is Limit
SQL ``` select key from (select key,value from t1 limit 100) t2 limit 10 ``` Optimized Logical Plan before modifying ``` == Optimized Logical Plan == Limit 10 Project key#228 Limit 100 MetastoreRelation default, t1, None ``` Optimized Logical Plan after modifying ``` == Optimized Logical Plan == Limit 10 Limit 100 Project key#228 MetastoreRelation default, t1, None ``` After this, we can combine limits Author: Zhongshuai Pei <799203320@qq.com> Author: DoingDone9 <799203320@qq.com> Closes #5797 from DoingDone9/ProjectLimit and squashes the following commits: 70d0fca [Zhongshuai Pei] Update FilterPushdownSuite.scala dc83ae9 [Zhongshuai Pei] Update FilterPushdownSuite.scala 485c61c [Zhongshuai Pei] Update Optimizer.scala f03fe7f [Zhongshuai Pei] Merge pull request #12 from apache/master f12fa50 [Zhongshuai Pei] Merge pull request #10 from apache/master f61210c [Zhongshuai Pei] Merge pull request #9 from apache/master 34b1a9a [Zhongshuai Pei] Merge pull request #8 from apache/master 802261c [DoingDone9] Merge pull request #7 from apache/master d00303b [DoingDone9] Merge pull request #6 from apache/master 98b134f [DoingDone9] Merge pull request #5 from apache/master 161cae3 [DoingDone9] Merge pull request #4 from apache/master c87e8b6 [DoingDone9] Merge pull request #3 from apache/master cb1852d [DoingDone9] Merge pull request #2 from apache/master c3f046f [DoingDone9] Merge pull request #1 from apache/master
1 parent 07a8620 commit 77cc25f

File tree

2 files changed

+19
-0
lines changed

2 files changed

+19
-0
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -170,6 +170,9 @@ object ColumnPruning extends Rule[LogicalPlan] {
170170

171171
Project(substitutedProjection, child)
172172

173+
case Project(projectList, Limit(exp, child)) =>
174+
Limit(exp, Project(projectList, child))
175+
173176
// Eliminate no-op Projects
174177
case Project(projectList, child) if child.output == projectList => child
175178
}

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,23 @@ class FilterPushdownSuite extends PlanTest {
9090

9191
comparePlans(optimized, correctAnswer)
9292
}
93+
94+
test("column pruning for Project(ne, Limit)") {
95+
val originalQuery =
96+
testRelation
97+
.select('a,'b)
98+
.limit(2)
99+
.select('a)
100+
101+
val optimized = Optimize.execute(originalQuery.analyze)
102+
val correctAnswer =
103+
testRelation
104+
.select('a)
105+
.limit(2).analyze
93106

107+
comparePlans(optimized, correctAnswer)
108+
}
109+
94110
// After this line is unimplemented.
95111
test("simple push down") {
96112
val originalQuery =

0 commit comments

Comments
 (0)