[SPARK-8223][SPARK-8224][SQL] shift left and shift right

[tarekbecker]

Jira:
https://issues.apache.org/jira/browse/SPARK-8223
https://issues.apache.org/jira/browse/SPARK-8224

I am aware of #7174 and will update this pr, if it's merged. Done
I don't know if #7034 can simplify this, but we can have a look on it, if it gets merged

@rxin In the Jira ticket the function as no second argument. I added a numBits argument that allows to specify the number of bits. I guess this improves the usability. I wanted to add shiftleft(value) as well, but the selectExpr dataframe tests crashes, if I have both. I order to do this, I added the following to the functions.scala def shiftRight(e: Column): Column = ShiftRight(e.expr, lit(1).expr), but as I mentioned this doesn't pass tests like df.selectExpr("shiftRight(a)", ... (not enough arguments exception).

If we need the bitwise shift in order to be hive compatible, I suggest to add shiftLeft and something like shiftLeftX

[AmplabJenkins]

Can one of the admins verify this patch?

[tarekbecker]

@davies can you review this, if you have time?

[davies]

Cannot understand the last sentence

[tarekbecker]

Thanks for your feedback. I removed the type information from the python description and changed it for the dataframe api. I hope it's clear now.

One comment to python: The max integer value is the max long value of Java/Scala. Because of that there is no value in specifying the result type for python.

>>> type(sqlContext.createDataFrame([(sys.maxint,)], ['a']).select(shiftLeft('a', 1).alias('r')).first().asDict().get('r'))
<type 'int'>

[davies]

In python 3, there is no long type. So we always use int in Python for all integral types.

[davies]

Should we use >>> (keeping the sign bit) or >> ?

[tarekbecker]

There is a special Jira ticket for that: https://issues.apache.org/jira/browse/SPARK-8226. Someone created already a PR.

[davies]

I see.

[chenghao-intel]

Why not pattern matching the data type? instead of the value? Will that cause extra box/unbox for primtives?

[tarekbecker]

That might be a good hint. I am going to take a look on the generated code and will come back to this and create maybe a follow-up.

[tarekbecker]

@chenghao-intel I investigated it a little bit, see the gist: https://gist.github.com/tarekauel/6994983b83a51668c5dc . The interesting part is that the match on the value is even faster, did I something wrong?

[davies]

LGTM.

OK to test

[tarekbecker]

@davies I guess Jenkins didn't got it.

[davies]

ok to test

[SparkQA]

Test build #993 has started for PR 7178 at commit f3f64e6.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[SparkQA]

Test build #36365 has started for PR 7178 at commit f3f64e6.

[SparkQA]

Test build #993 has finished for PR 7178 at commit f3f64e6.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class Heartbeat(workerId: String, worker: RpcEndpointRef) extends DeployMessage
  • case class RegisteredWorker(master: RpcEndpointRef, masterWebUiUrl: String) extends DeployMessage
  • case class RegisterApplication(appDescription: ApplicationDescription, driver: RpcEndpointRef)
  • case class RegisteredApplication(appId: String, master: RpcEndpointRef) extends DeployMessage
  • case class SubmitDriverResponse(
  • case class KillDriverResponse(
  • case class MasterChanged(master: RpcEndpointRef, masterWebUiUrl: String)
  • class DCT(override val uid: String)
  • class MinMaxScaler(override val uid: String)
  • class PCA (override val uid: String) extends Estimator[PCAModel] with PCAParams
  • class StreamingLinearAlgorithm(object):
  • class StreamingLinearRegressionWithSGD(StreamingLinearAlgorithm):
  • class AnalysisException(Exception):
  • class FlumeUtils(object):
  • case class Cast(child: Expression, dataType: DataType) extends UnaryExpression with Logging
  • trait ExpectsInputTypes
  • trait AutoCastInputTypes
  • abstract class BinaryExpression extends Expression with trees.BinaryNode[Expression]
  • abstract class BinaryOperator extends BinaryExpression
  • abstract class BinaryArithmetic extends BinaryOperator
  • class SpecificOrdering extends $
  • class SpecificProjection extends $
  • final class SpecificRow extends $
  • case class ShiftLeft(left: Expression, right: Expression) extends BinaryExpression
  • case class ShiftRight(left: Expression, right: Expression) extends BinaryExpression
  • case class UnHex(child: Expression) extends UnaryExpression with Serializable
  • case class Crc32(child: Expression)
  • abstract class BinaryComparison extends BinaryOperator with Predicate
  • // compiled class file for the closure here will conflict with the one in callUDF (upper case).

[SparkQA]

Test build #36365 has finished for PR 7178 at commit f3f64e6.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Merged build finished. Test FAILed.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[SparkQA]

Test build #36373 has started for PR 7178 at commit 8023bb5.

[SparkQA]

Test build #36373 has finished for PR 7178 at commit 8023bb5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class ShiftLeft(left: Expression, right: Expression) extends BinaryExpression
  • case class ShiftRight(left: Expression, right: Expression) extends BinaryExpression

[AmplabJenkins]

Merged build finished. Test PASSed.

[davies]

Merged into master, thanks!