[SPARK-41319][CONNECT][PYTHON] Implement Column.{when, otherwise} and Function when
#38935
Conversation
zhengruifeng
force-pushed
the
connect_column_case_when
branch
2 times, most recently
from
0f0b5c9 to
d210b4c
Compare
There was a problem hiding this comment.
[error] /home/runner/work/spark/spark/connector/connect/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:594:13: type mismatch;
[error] found : scala.collection.mutable.Buffer[(org.apache.spark.sql.catalyst.expressions.Expression, org.apache.spark.sql.catalyst.expressions.Expression)]
[error] required: Seq[(org.apache.spark.sql.catalyst.expressions.Expression, org.apache.spark.sql.catalyst.expressions.Expression)]
[error] .map(b => (transformExpression(b.getCondition), transformExpression(b.getValue))),
There was a problem hiding this comment.
guessing you need a .toSeq?
There was a problem hiding this comment.
Why not make this an unresolved_function? All it takes is that we add a constructor to CaseWhen.
There was a problem hiding this comment.
Alternatively you can rewrite this into a series of ifs.
There was a problem hiding this comment.
hmm for unresolved_function, how to deal with the optional else?
There was a problem hiding this comment.
If the the number of expressions is uneven, the last expression is the else expression.
There was a problem hiding this comment.
If we use this way then have to document it clear for even/odd assumption on the server side.
There was a problem hiding this comment.
Yeah we do. I am just trying to see if we can keep the number of expressions in the proto to a bare minimum. In this case you can get away with it because it sort of makes sense.
There was a problem hiding this comment.
I am not sure whether unresolved_function when can support otherwise, let me take a deeper look
There was a problem hiding this comment.
Otherwise is the elseValue in CaseWhen.
There was a problem hiding this comment.
got it, let me have a try, thanks
zhengruifeng
force-pushed
the
connect_column_case_when
branch
from
d210b4c to
110d7e7
Compare
|
close this one in favor of #38956 |
4 participants
What changes were proposed in this pull request?
Implement
Column.{when, otherwise}and FunctionwhenWhy are the changes needed?
For API coverage
Does this PR introduce any user-facing change?
yes
How was this patch tested?
added UT