Skip to content

[SPARK-22361][SQL][TEST] Add unit test for Window Frames #20019

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed

[SPARK-22361][SQL][TEST] Add unit test for Window Frames #20019

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
87e61ce
Add unit test for Window Frames
gaborgsomogyi Dec 14, 2017
a2fa042
Review fix
gaborgsomogyi Dec 20, 2017
fa20e73
ExpressionParserSuite window related test enhanced + review fixes
gaborgsomogyi Dec 21, 2017
677a4d6
Added group comment for boundaries
gaborgsomogyi Jan 2, 2018
6dfc77e
Merge branch 'master' into SPARK-22361
gaborgsomogyi Jan 8, 2018
d1e2454
Test fix
gaborgsomogyi Jan 8, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 49 additions & 8 deletions sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,8 +249,8 @@ class ExpressionParserSuite extends PlanTest {
assertEqual("foo(*) over (partition by a, b)", windowed(Seq('a, 'b)))
assertEqual("foo(*) over (distribute by a, b)", windowed(Seq('a, 'b)))
assertEqual("foo(*) over (cluster by a, b)", windowed(Seq('a, 'b)))
assertEqual("foo(*) over (order by a desc, b asc)", windowed(Seq.empty, Seq('a.desc, 'b.asc )))
assertEqual("foo(*) over (sort by a desc, b asc)", windowed(Seq.empty, Seq('a.desc, 'b.asc )))
assertEqual("foo(*) over (order by a desc, b asc)", windowed(Seq.empty, Seq('a.desc, 'b.asc)))
assertEqual("foo(*) over (sort by a desc, b asc)", windowed(Seq.empty, Seq('a.desc, 'b.asc)))
assertEqual("foo(*) over (partition by a, b order by c)", windowed(Seq('a, 'b), Seq('c.asc)))
assertEqual("foo(*) over (distribute by a, b sort by c)", windowed(Seq('a, 'b), Seq('c.asc)))

Expand All @@ -263,21 +263,62 @@ class ExpressionParserSuite extends PlanTest {
"sum(product + 1) over (partition by ((product / 2) + 1) order by 2)",
WindowExpression('sum.function('product + 1),
WindowSpecDefinition(Seq('product / 2 + 1), Seq(Literal(2).asc), UnspecifiedFrame)))
}

test("range/rows window function expressions") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about window function expressions on various window frames?

Copy link
Contributor Author

@gaborgsomogyi gaborgsomogyi Jan 2, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just back from holiday. What do you mean exactly here? Range/rows between test with most of the combinations are covered. If you could give an example it would be helpful.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jiangxb1987 gentle ping

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry for the late response. Just update the test case name.

val func = 'foo.function(star())
def windowed(
partitioning: Seq[Expression] = Seq.empty,
ordering: Seq[SortOrder] = Seq.empty,
frame: WindowFrame = UnspecifiedFrame): Expression = {
WindowExpression(func, WindowSpecDefinition(partitioning, ordering, frame))
}

// Range/Row
val frameTypes = Seq(("rows", RowFrame), ("range", RangeFrame))
val boundaries = Seq(
("10 preceding", -Literal(10), CurrentRow),
// No between combinations
("unbounded preceding", UnboundedPreceding, CurrentRow),
("2147483648 preceding", -Literal(2147483648L), CurrentRow),
("10 preceding", -Literal(10), CurrentRow),
("3 + 1 preceding", -Add(Literal(3), Literal(1)), CurrentRow),
("0 preceding", -Literal(0), CurrentRow),
("current row", CurrentRow, CurrentRow),
("0 following", Literal(0), CurrentRow),
("3 + 1 following", Add(Literal(3), Literal(1)), CurrentRow),
("unbounded preceding", UnboundedPreceding, CurrentRow),
("10 following", Literal(10), CurrentRow),
("2147483649 following", Literal(2147483649L), CurrentRow),
("unbounded following", UnboundedFollowing, CurrentRow), // Will fail during analysis

// Between combinations
("between unbounded preceding and 5 following",
UnboundedPreceding, Literal(5)),
("between unbounded preceding and 3 + 1 following",
UnboundedPreceding, Add(Literal(3), Literal(1))),
("between unbounded preceding and 2147483649 following",
UnboundedPreceding, Literal(2147483649L)),
("between unbounded preceding and current row", UnboundedPreceding, CurrentRow),
("between unbounded preceding and unbounded following",
UnboundedPreceding, UnboundedFollowing),
("between 2147483648 preceding and current row", -Literal(2147483648L), CurrentRow),
("between 10 preceding and current row", -Literal(10), CurrentRow),
("between 3 + 1 preceding and current row", -Add(Literal(3), Literal(1)), CurrentRow),
("between 0 preceding and current row", -Literal(0), CurrentRow),
("between current row and current row", CurrentRow, CurrentRow),
("between current row and 0 following", CurrentRow, Literal(0)),
("between current row and 5 following", CurrentRow, Literal(5)),
("between 10 preceding and 5 following", -Literal(10), Literal(5))
("between current row and 3 + 1 following", CurrentRow, Add(Literal(3), Literal(1))),
("between current row and 2147483649 following", CurrentRow, Literal(2147483649L)),
("between current row and unbounded following", CurrentRow, UnboundedFollowing),
("between 2147483648 preceding and unbounded following",
-Literal(2147483648L), UnboundedFollowing),
("between 10 preceding and unbounded following",
-Literal(10), UnboundedFollowing),
("between 3 + 1 preceding and unbounded following",
-Add(Literal(3), Literal(1)), UnboundedFollowing),
("between 0 preceding and unbounded following", -Literal(0), UnboundedFollowing),

// Between partial and full range
("between 10 preceding and 5 following", -Literal(10), Literal(5)),
("between unbounded preceding and unbounded following",
UnboundedPreceding, UnboundedFollowing)
)
frameTypes.foreach {
case (frameTypeSql, frameType) =>
Expand Down
Loading