[SPARK-16355] [SPARK-16354] [SQL] Fix Bugs When LIMIT/TABLESAMPLE is Non-foldable, Zero or Negative #14034

gatorsmile · 2016-07-03T06:05:19Z

What changes were proposed in this pull request?

Issue 1: When a query containing LIMIT/TABLESAMPLE 0, the statistics could be zero. Results are correct but it could cause a huge performance regression. For example,

Seq(("one", 1), ("two", 2), ("three", 3), ("four", 4)).toDF("k", "v")
  .createOrReplaceTempView("test")
val df1 = spark.table("test")
val df2 = spark.table("test").limit(0)
val df = df1.join(df2, Seq("k"), "left")

The statistics of both df and df2 are zero. The statistics values should never be zero; otherwise sizeInBytes of BinaryNode will also be zero (product of children). This PR is to increase it to 1 when the num of rows is equal to 0.

Issue 2: When a query containing negative LIMIT/TABLESAMPLE, we should issue exceptions. Negative values could break the implementation assumption of multiple parts. For example, statistics calculation. Below is the example query.

SELECT * FROM testData TABLESAMPLE (-1 rows)
SELECT * FROM testData LIMIT -1

This PR is to issue an appropriate exception in this case.

Issue 3: Spark SQL follows the restriction of LIMIT clause in Hive. The argument to the LIMIT clause must evaluate to a constant value. It can be a numeric literal, or another kind of numeric expression involving operators, casts, and function return values. You cannot refer to a column or use a subquery. Currently, we do not detect whether the expression in LIMIT clause is foldable or not. If non-foldable, we might issue a strange error message. For example,

SELECT * FROM testData LIMIT rand() > 0.2

Then, a misleading error message is issued, like

assertion failed: No plan for GlobalLimit (_nondeterministic#203 > 0.2)
+- Project [key#11, value#12, rand(-1441968339187861415) AS _nondeterministic#203]
   +- LocalLimit (_nondeterministic#202 > 0.2)
      +- Project [key#11, value#12, rand(-1308350387169017676) AS _nondeterministic#202]
         +- LogicalRDD [key#11, value#12]

java.lang.AssertionError: assertion failed: No plan for GlobalLimit (_nondeterministic#203 > 0.2)
+- Project [key#11, value#12, rand(-1441968339187861415) AS _nondeterministic#203]
   +- LocalLimit (_nondeterministic#202 > 0.2)
      +- Project [key#11, value#12, rand(-1308350387169017676) AS _nondeterministic#202]
         +- LogicalRDD [key#11, value#12]

This PR detects it and then issues a meaningful error message.

How was this patch tested?

Added test cases.

SparkQA · 2016-07-03T07:49:12Z

Test build #61678 has finished for PR 14034 at commit bdf4e56.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2016-07-04T18:10:31Z

cc @cloud-fan @davies Thanks!

jaceklaskowski · 2016-07-04T19:54:08Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala

+                  s"number_rows:$numRows")
+            }
+
+          case l: LocalLimit =>


What do you think about merging the two cases to case l @ (_: LocalLimit | _: GlobalLimit) => to remove the duplication (or at least introduce a local method).

I do not think we can merge them, but, yeah, we can create a local function for it. Thank you!

SparkQA · 2016-07-05T00:13:58Z

Test build #61736 has finished for PR 14034 at commit 3c402d3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2016-07-05T01:11:20Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala

@@ -46,6 +46,15 @@ trait CheckAnalysis extends PredicateHelper {
    }).length > 1
  }

+  private def checkLimitClause(limitExpr: Expression): Unit = {
+    val numRows = limitExpr.eval().asInstanceOf[Int]


is the limit expression guaranteed to be literal?

Nope. Users can input an expression here. For example,

spark/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala

Line 234 in e5d703b

assertEqual(s"$sql limit cast(9 / 4 as int)", plan.limit(Cast(Literal(9) / 4, IntegerType)))

ah, but it's still foldable. Is it possible it's non-foldable?

Oracle: http://docs.oracle.com/javadb/10.5.3.0/ref/rrefsqljoffsetfetch.html

DB2 z/OS: https://www.ibm.com/support/knowledgecenter/SSEPEK_10.0.0/com.ibm.db2z10.doc.sqlref/src/tpc/db2z_sql_fetchfirstclause.html

MySQL: http://dev.mysql.com/doc/refman/5.7/en/select.html

PostgreSQL: https://www.postgresql.org/docs/8.1/static/queries-limit.html

It sounds like nobody supports it. All of the mainstream DB vendors only support integer

We do not support non-foldable limit clauses.

spark/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala

Lines 67 to 89 in d063898

object SpecialLimits extends Strategy {

override def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {

case logical.ReturnAnswer(rootPlan) => rootPlan match {

case logical.Limit(IntegerLiteral(limit), logical.Sort(order, true, child)) =>

execution.TakeOrderedAndProjectExec(limit, order, None, planLater(child)) :: Nil

case logical.Limit(

IntegerLiteral(limit),

logical.Project(projectList, logical.Sort(order, true, child))) =>

execution.TakeOrderedAndProjectExec(

limit, order, Some(projectList), planLater(child)) :: Nil

case logical.Limit(IntegerLiteral(limit), child) =>

execution.CollectLimitExec(limit, planLater(child)) :: Nil

case other => planLater(other) :: Nil

}

case logical.Limit(IntegerLiteral(limit), logical.Sort(order, true, child)) =>

execution.TakeOrderedAndProjectExec(limit, order, None, planLater(child)) :: Nil

case logical.Limit(

IntegerLiteral(limit), logical.Project(projectList, logical.Sort(order, true, child))) =>

execution.TakeOrderedAndProjectExec(

limit, order, Some(projectList), planLater(child)) :: Nil

case _ => Nil

}

}

spark/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala

Lines 398 to 401 in d063898

case logical.LocalLimit(IntegerLiteral(limit), child) =>

execution.LocalLimitExec(limit, planLater(child)) :: Nil

case logical.GlobalLimit(IntegerLiteral(limit), child) =>

execution.GlobalLimitExec(limit, planLater(child)) :: Nil

But,,, we do not issue an exception if users do it. Thus, the error we got is strange:

assertion failed: No plan for GlobalLimit (_nondeterministic#203 > 0.2) +- Project [key#11, value#12, rand(-1441968339187861415) AS _nondeterministic#203] +- LocalLimit (_nondeterministic#202 > 0.2) +- Project [key#11, value#12, rand(-1308350387169017676) AS _nondeterministic#202] +- LogicalRDD [key#11, value#12] java.lang.AssertionError: assertion failed: No plan for GlobalLimit (_nondeterministic#203 > 0.2) +- Project [key#11, value#12, rand(-1441968339187861415) AS _nondeterministic#203] +- LocalLimit (_nondeterministic#202 > 0.2) +- Project [key#11, value#12, rand(-1308350387169017676) AS _nondeterministic#202] +- LogicalRDD [key#11, value#12]

Let me do it in this PR. Thank you for your review! : )

SparkQA · 2016-07-05T08:08:08Z

Test build #61746 has finished for PR 14034 at commit 5b36fbc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jaceklaskowski · 2016-07-05T09:04:34Z

sql/core/src/test/scala/org/apache/spark/sql/StatisticsSuite.scala

+      }
+
+      assert(sizes.size === 1, s"Size wrong for:\n ${df.queryExecution}")
+      assert(sizes(0).equals(BigInt(96)),


Why do you equals? Would === not work here?

This is just following what the other test cases did. Sure, we can change all of them. Let me do it. Thanks!

SparkQA · 2016-07-05T18:42:30Z

Test build #61761 has finished for PR 14034 at commit a2a828f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2016-07-08T04:37:14Z

ping @cloud-fan : )

cloud-fan · 2016-07-08T04:48:42Z

...alyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala

@@ -660,7 +660,12 @@ case class GlobalLimit(limitExpr: Expression, child: LogicalPlan) extends UnaryN
  }
  override lazy val statistics: Statistics = {
    val limit = limitExpr.eval().asInstanceOf[Int]
-    val sizeInBytes = (limit: Long) * output.map(a => a.dataType.defaultSize).sum
+    var sizeInBytes = (limit: Long) * output.map(a => a.dataType.defaultSize).sum
+    if (sizeInBytes == 0) {


I think it's a special case for limit. How about we make it more clear? e.g.

val sizeInBytes = if (limit == 0) { 1 } else { ... }

Thanks! Let me fix it.

cloud-fan · 2016-07-08T06:21:04Z

sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala

      arrayData.collect().take(1).map(Row.fromTuple).toSeq)

    checkAnswer(
      sql("SELECT * FROM mapData LIMIT 1"),
      mapData.collect().take(1).map(Row.fromTuple).toSeq)
  }

+  test("non-foldable expressions in LIMIT") {
+    val e = intercept[AnalysisException] {
+      sql("SELECT * FROM testData LIMIT key > 3")


what will happen if the type is wrong? e.g. LIMIT true

A good question! : ) Now, the exception we issued is not good:

java.lang.Boolean cannot be cast to java.lang.Integer java.lang.ClassCastException: java.lang.Boolean cannot be cast to java.lang.Integer at scala.runtime.BoxesRunTime.unboxToInt(BoxesRunTime.java:101)

Let me fix it and throw a more reasonable exception:

number_rows in limit clause cannot be cast to integer:true;

cloud-fan · 2016-07-08T07:25:02Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala

+      case o: Int if o >= 0 => // OK
+      case o: Int => failAnalysis(
+        s"number_rows in limit clause must be equal to or greater than 0. number_rows:$o")
+      case o => failAnalysis(


should we allow other integral type? e.g. byte, long, etc.

You know, we can do it, but we need to fix the other parts. Let me try it. Thanks!

Tried to do it in different ways, but the best way I found is to introduce an Analyzer rule.

We have one rule in Optimizer and one rule in Planner for Limit. Both assume the limitExpr is an Integer type. It looks ugly to convert the type everywhere. To support different numeric types, we need to convert these foldable numeric values to int in Analyzer, I think. Let me upload a fix at first. You can judge whether this is a good way or not. Thanks!

SparkQA · 2016-07-08T08:26:21Z

Test build #61963 has finished for PR 14034 at commit f600ba4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-08T08:45:29Z

Test build #61974 has finished for PR 14034 at commit 1abdbb9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-08T09:02:42Z

Test build #61973 has finished for PR 14034 at commit 8fd72f6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-08T22:42:52Z

Test build #61997 has finished for PR 14034 at commit 3036847.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2016-07-08T22:56:14Z

retest this please

gatorsmile · 2016-07-08T23:51:12Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala

+object ResolveLimits extends Rule[LogicalPlan] {
+  def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case g @ GlobalLimit(limitExpr, _) if limitExpr.foldable && isNumeric(limitExpr.eval()) =>
+      g.copy(limitExpr = Literal(limitExpr.eval().asInstanceOf[Number].intValue(), IntegerType))


Another way is like

g.copy(limitExpr = Literal(Cast(limitExpr, IntegerType).eval(), IntegerType))

Not sure which one is better here.

SparkQA · 2016-07-09T00:57:57Z

Test build #62002 has finished for PR 14034 at commit 3036847.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-09T02:41:32Z

Test build #62006 has finished for PR 14034 at commit d135b77.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2016-07-09T12:31:24Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala

@@ -2045,6 +2047,21 @@ object EliminateUnions extends Rule[LogicalPlan] {
 }

 /**
+ * Converts foldable numeric expressions to integers in [[GlobalLimit]] and [[LocalLimit]] operators
+ */
+object ResolveLimits extends Rule[LogicalPlan] {


Thanks for trying this out! Yea looks like it's doable. Let's remove it first and do it in follow-ups, to make this PR surgical.

We can create a JIRA first and discuss with others to decide if it's useful to support all integral types in limit.

Sure, let me revert it back. Thanks!

SparkQA · 2016-07-09T16:54:01Z

Test build #62027 has finished for PR 14034 at commit 028aa79.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2016-07-10T09:35:05Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala

+      case o: Int => failAnalysis(
+        s"number_rows in limit clause must be equal to or greater than 0. number_rows:$o")
+      case o => failAnalysis(
+        s"""number_rows in limit clause cannot be cast to integer:\"$o\".""")


cannot be cast to integer -> must be integer? e.g. byte is castable to int.

cloud-fan · 2016-07-10T15:40:03Z

sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala

+      "Limit:(testdata.`key` > 3)"))
+  }
+
+  test("Limit: unable to evaluate and cast expressions in limit clauses to Int") {


We should also update the test name, we don't try to cast the limit expression to int type.

Yeah, will change it. Thanks!

cloud-fan · 2016-07-10T15:49:21Z

LGTM except some style comments, thanks for working on it!

SparkQA · 2016-07-10T17:09:43Z

Test build #62058 has finished for PR 14034 at commit 01137dc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2016-07-11T03:22:50Z

you missed one comment : https://github.com/apache/spark/pull/14034/files#r70183958 :)

gatorsmile · 2016-07-11T03:39:19Z

uh... Thanks! Let me do it now.

cloud-fan · 2016-07-11T04:50:25Z

LGTM, pending jenkins

gatorsmile · 2016-07-11T04:51:54Z

Thank you!

SparkQA · 2016-07-11T04:52:48Z

Test build #62069 has finished for PR 14034 at commit dec5ad9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-11T05:42:04Z

Test build #62073 has finished for PR 14034 at commit 2e6f8d8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2016-07-11T06:05:42Z

retest this please

SparkQA · 2016-07-11T07:42:10Z

Test build #62078 has finished for PR 14034 at commit 2e6f8d8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-11T08:13:49Z

Test build #62080 has finished for PR 14034 at commit d66870b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2016-07-11T08:25:22Z

thanks, merging to master and 2.0!

…n-foldable, Zero or Negative #### What changes were proposed in this pull request? **Issue 1:** When a query containing LIMIT/TABLESAMPLE 0, the statistics could be zero. Results are correct but it could cause a huge performance regression. For example, ```Scala Seq(("one", 1), ("two", 2), ("three", 3), ("four", 4)).toDF("k", "v") .createOrReplaceTempView("test") val df1 = spark.table("test") val df2 = spark.table("test").limit(0) val df = df1.join(df2, Seq("k"), "left") ``` The statistics of both `df` and `df2` are zero. The statistics values should never be zero; otherwise `sizeInBytes` of `BinaryNode` will also be zero (product of children). This PR is to increase it to `1` when the num of rows is equal to 0. **Issue 2:** When a query containing negative LIMIT/TABLESAMPLE, we should issue exceptions. Negative values could break the implementation assumption of multiple parts. For example, statistics calculation. Below is the example query. ```SQL SELECT * FROM testData TABLESAMPLE (-1 rows) SELECT * FROM testData LIMIT -1 ``` This PR is to issue an appropriate exception in this case. **Issue 3:** Spark SQL follows the restriction of LIMIT clause in Hive. The argument to the LIMIT clause must evaluate to a constant value. It can be a numeric literal, or another kind of numeric expression involving operators, casts, and function return values. You cannot refer to a column or use a subquery. Currently, we do not detect whether the expression in LIMIT clause is foldable or not. If non-foldable, we might issue a strange error message. For example, ```SQL SELECT * FROM testData LIMIT rand() > 0.2 ``` Then, a misleading error message is issued, like ``` assertion failed: No plan for GlobalLimit (_nondeterministic#203 > 0.2) +- Project [key#11, value#12, rand(-1441968339187861415) AS _nondeterministic#203] +- LocalLimit (_nondeterministic#202 > 0.2) +- Project [key#11, value#12, rand(-1308350387169017676) AS _nondeterministic#202] +- LogicalRDD [key#11, value#12] java.lang.AssertionError: assertion failed: No plan for GlobalLimit (_nondeterministic#203 > 0.2) +- Project [key#11, value#12, rand(-1441968339187861415) AS _nondeterministic#203] +- LocalLimit (_nondeterministic#202 > 0.2) +- Project [key#11, value#12, rand(-1308350387169017676) AS _nondeterministic#202] +- LogicalRDD [key#11, value#12] ``` This PR detects it and then issues a meaningful error message. #### How was this patch tested? Added test cases. Author: gatorsmile <gatorsmile@gmail.com> Closes #14034 from gatorsmile/limit. (cherry picked from commit e226278) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

gatorsmile added 2 commits July 2, 2016 22:46

fix

1255968

reverted

bdf4e56

gatorsmile changed the title ~~[16355] [16354] [SQL] Fix Bugs When LIMIT/TABLESAMPLE is Zero or Negative~~ [SPARK-16355] [SPARK-16354] [SQL] Fix Bugs When LIMIT/TABLESAMPLE is Zero or Negative Jul 4, 2016

jaceklaskowski reviewed Jul 4, 2016
View reviewed changes

address comments

3c402d3

cloud-fan reviewed Jul 5, 2016
View reviewed changes

Detected foldable expression in limit

5b36fbc

gatorsmile changed the title ~~[SPARK-16355] [SPARK-16354] [SQL] Fix Bugs When LIMIT/TABLESAMPLE is Zero or Negative~~ [SPARK-16355] [SPARK-16354] [SQL] Fix Bugs When LIMIT/TABLESAMPLE is Non-foldable, Zero or Negative Jul 5, 2016

jaceklaskowski reviewed Jul 5, 2016
View reviewed changes

address comments.

a2a828f

cloud-fan reviewed Jul 8, 2016
View reviewed changes

address comments.

f600ba4

cloud-fan reviewed Jul 8, 2016
View reviewed changes

gatorsmile added 2 commits July 8, 2016 00:01

address comments.

8fd72f6

address comments.

1abdbb9

cloud-fan reviewed Jul 8, 2016
View reviewed changes

address comments.

3036847

gatorsmile reviewed Jul 8, 2016
View reviewed changes

cloud-fan reviewed Jul 9, 2016
View reviewed changes

revert it back

028aa79

cloud-fan reviewed Jul 10, 2016
View reviewed changes

address comments

01137dc

cloud-fan reviewed Jul 10, 2016
View reviewed changes

gatorsmile added 2 commits July 10, 2016 19:57

Merge remote-tracking branch 'upstream/master' into limit

0ebbdfe

address comments

dec5ad9

address comments

2e6f8d8

fix test case

d66870b

asfgit closed this in e226278 Jul 11, 2016

	object SpecialLimits extends Strategy {
	override def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
	case logical.ReturnAnswer(rootPlan) => rootPlan match {
	case logical.Limit(IntegerLiteral(limit), logical.Sort(order, true, child)) =>
	execution.TakeOrderedAndProjectExec(limit, order, None, planLater(child)) :: Nil
	case logical.Limit(
	IntegerLiteral(limit),
	logical.Project(projectList, logical.Sort(order, true, child))) =>
	execution.TakeOrderedAndProjectExec(
	limit, order, Some(projectList), planLater(child)) :: Nil
	case logical.Limit(IntegerLiteral(limit), child) =>
	execution.CollectLimitExec(limit, planLater(child)) :: Nil
	case other => planLater(other) :: Nil
	}
	case logical.Limit(IntegerLiteral(limit), logical.Sort(order, true, child)) =>
	execution.TakeOrderedAndProjectExec(limit, order, None, planLater(child)) :: Nil
	case logical.Limit(
	IntegerLiteral(limit), logical.Project(projectList, logical.Sort(order, true, child))) =>
	execution.TakeOrderedAndProjectExec(
	limit, order, Some(projectList), planLater(child)) :: Nil
	case _ => Nil
	}
	}

	case logical.LocalLimit(IntegerLiteral(limit), child) =>
	execution.LocalLimitExec(limit, planLater(child)) :: Nil
	case logical.GlobalLimit(IntegerLiteral(limit), child) =>
	execution.GlobalLimitExec(limit, planLater(child)) :: Nil

[SPARK-16355] [SPARK-16354] [SQL] Fix Bugs When LIMIT/TABLESAMPLE is Non-foldable, Zero or Negative #14034

[SPARK-16355] [SPARK-16354] [SQL] Fix Bugs When LIMIT/TABLESAMPLE is Non-foldable, Zero or Negative #14034

Uh oh!

Conversation

gatorsmile commented Jul 3, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Jul 3, 2016

Uh oh!

gatorsmile commented Jul 4, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 5, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmile Jul 5, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 5, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 5, 2016

Uh oh!

gatorsmile commented Jul 8, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan Jul 8, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmile Jul 8, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 8, 2016

Uh oh!

SparkQA commented Jul 8, 2016

Uh oh!

SparkQA commented Jul 8, 2016

Uh oh!

SparkQA commented Jul 8, 2016

Uh oh!

gatorsmile commented Jul 8, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 9, 2016

Uh oh!

SparkQA commented Jul 9, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

gatorsmile commented Jul 3, 2016 •

edited

Loading

gatorsmile Jul 5, 2016 •

edited

Loading

cloud-fan Jul 8, 2016 •

edited

Loading

gatorsmile Jul 8, 2016 •

edited

Loading