[SPARK-12288] [SQL] Support UnsafeRow in Coalesce/Except/Intersect. #10285
Conversation
|
Test build #47633 has finished for PR 10285 at commit
|
| @@ -263,6 +265,10 @@ case class Except(left: SparkPlan, right: SparkPlan) extends BinaryNode { | |||
| protected override def doExecute(): RDD[InternalRow] = { | |||
| left.execute().map(_.copy()).subtract(right.execute().map(_.copy())) | |||
| } | |||
|
|
|||
| override def outputsUnsafeRows: Boolean = children.forall(_.outputsUnsafeRows) | |||
There was a problem hiding this comment.
If an operator's children produce both unsafe and safe rows, we will convert everything unsafe rows.
so here should be children.exists(_.outputsUnsafeRows)
There was a problem hiding this comment.
Thank you! I also found the same issue exists in Union I also did a correction.
|
Test build #47674 has finished for PR 10285 at commit
|
|
The failed test case is not related to my change. I will try to see if I can fix the flaky R test. |
|
retest this please |
|
Test build #47687 has finished for PR 10285 at commit
|
| val preparedPlan = sqlContext.prepareForExecution.execute(plan) | ||
| assert(getConverters(preparedPlan).size === 1) | ||
| assert(preparedPlan.outputsUnsafeRows) | ||
| } |
There was a problem hiding this comment.
should we add mixed rows test for coalesce too?
There was a problem hiding this comment.
Sorry, I am not sure if I understand your question. Coalesce only has a single child. I am not sure what mixed rows mean. Could you give me one example? Thank you! @cloud-fan
There was a problem hiding this comment.
oh sorry, I thought it was org.apache.spark.sql.catalyst.expressions.Coalesce, nvm
|
Test build #47698 has finished for PR 10285 at commit
|
|
LGTM, merging into master, thanks! |
Support UnsafeRow for the Coalesce/Except/Intersect.
Could you review if my code changes are ok? @davies Thank you!