[SPARK-8207][SQL] Add math function bin #6721

viirya · 2015-06-09T10:53:46Z

JIRA: https://issues.apache.org/jira/browse/SPARK-8207

SparkQA · 2015-06-09T13:24:36Z

Test build #34500 has finished for PR 6721 at commit 50e0c3b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class AbstractUnaryMathExpression[T, U](name: String)
- abstract class UnaryMathExpression(f: Double => Double, name: String)
- case class Bin(child: Expression)

SparkQA · 2015-06-09T17:41:33Z

Test build #34507 has finished for PR 6721 at commit c0c3197.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class AbstractUnaryMathExpression[T, U](name: String)
- abstract class UnaryMathExpression(f: Double => Double, name: String)
- case class Bin(child: Expression)

rxin · 2015-06-09T22:57:21Z

sql/core/src/main/scala/org/apache/spark/sql/functions.scala

@@ -1299,6 +1299,21 @@ object functions {
   */
  def toRadians(columnName: String): Column = toRadians(Column(columnName))

+  /**
+   * Computes the binary format of the given value.


An expression that returns the string representation of the binary value of the given long column. For example, bin("123") returns "1100".

SparkQA · 2015-06-10T10:16:51Z

Test build #34583 has finished for PR 6721 at commit 4cb764d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class OverrideFunctionRegistry(underlying: FunctionRegistry) extends FunctionRegistry
- abstract class AbstractUnaryMathExpression[T, U](name: String)
- abstract class UnaryMathExpression(f: Double => Double, name: String)
- case class Bin(child: Expression)

rxin · 2015-06-10T16:48:28Z

sql/core/src/main/scala/org/apache/spark/sql/functions.scala

+   * @group math_funcs
+   * @since 1.4.0
+   */
+  def bin(e: Column): Column = Bin(e.expr)


the math functions are sorted. can we put this in order as well?

Conflicts: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MathFunctionsSuite.scala sql/core/src/main/scala/org/apache/spark/sql/functions.scala sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala

SparkQA · 2015-06-11T08:09:39Z

Test build #34676 has finished for PR 6721 at commit a0a2d0f.

This patch fails Python style tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
- abstract class AbstractUnaryMathExpression[T, U](name: String)
- abstract class UnaryMathExpression(f: Double => Double, name: String)
- case class Bin(child: Expression)

SparkQA · 2015-06-11T09:58:59Z

Test build #34677 has finished for PR 6721 at commit 7a0196f.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class LeafMathExpression(c: Double, name: String)
- abstract class AbstractUnaryMathExpression[T, U](name: String)
- abstract class UnaryMathExpression(f: Double => Double, name: String)
- case class EulerNumber() extends LeafMathExpression(math.E, "E")
- case class Pi() extends LeafMathExpression(math.Pi, "PI")
- case class Log2(child: Expression)
- case class Bin(child: Expression)
- case class StringLength(child: Expression) extends UnaryExpression with ExpectsInputTypes

viirya · 2015-06-11T12:10:01Z

retest this please.

SparkQA · 2015-06-11T15:19:25Z

Test build #34688 has finished for PR 6721 at commit 7a0196f.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class AbstractUnaryMathExpression[T, U](name: String)
- abstract class UnaryMathExpression(f: Double => Double, name: String)
- case class Bin(child: Expression)

SparkQA · 2015-06-11T19:38:41Z

Test build #34698 has finished for PR 6721 at commit d4f4774.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class AbstractUnaryMathExpression[T, U](name: String)
- abstract class UnaryMathExpression(f: Double => Double, name: String)
- case class Bin(child: Expression)

Conflicts: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/math.scala sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala

SparkQA · 2015-06-13T17:40:12Z

Test build #34823 has finished for PR 6721 at commit dea9c12.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class AbstractUnaryMathExpression[T, U](name: String)
- abstract class UnaryMathExpression(f: Double => Double, name: String)
- case class Bin(child: Expression)

Conflicts: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/math.scala

SparkQA · 2015-06-15T08:58:33Z

Test build #34914 has finished for PR 6721 at commit 0cf20f2.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class AbstractUnaryMathExpression[T, U](name: String)
- abstract class UnaryMathExpression(f: Double => Double, name: String)
- case class Bin(child: Expression)

davies · 2015-06-17T17:23:50Z

python/pyspark/sql/functions.py

@@ -230,6 +231,19 @@ def approxCountDistinct(col, rsd=None):
    return Column(jc)


+@ignore_unicode_prefix
+@since(1.4)


This should be 1.5.

SparkQA · 2015-06-18T07:26:00Z

Test build #35098 has finished for PR 6721 at commit 0677f1a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class ElementwiseProduct(VectorTransformer):
- class MatrixUDT(UserDefinedType):
- abstract class AbstractUnaryMathExpression[T, U](name: String)
- abstract class UnaryMathExpression(f: Double => Double, name: String)
- case class Bin(child: Expression)
- // Add jar to isolated hive (metadataHive) class loader.

SparkQA · 2015-06-18T09:55:40Z

Test build #35105 has finished for PR 6721 at commit 07e1c8f.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Bin(child: Expression)

davies · 2015-06-19T17:06:32Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/math.scala

+  override def expectedChildTypes: Seq[DataType] = Seq(LongType)
+  override def dataType: DataType = StringType
+
+  def funcName: String = name.toLowerCase


This is not needed, I think.

davies · 2015-06-19T17:08:57Z

LGTM, merging into master!

SparkQA · 2015-06-19T18:07:47Z

Test build #35287 has finished for PR 6721 at commit cde33b5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Bin(child: Expression)

viirya · 2015-06-19T18:16:42Z

Looks like an unrelated failure. Do we need to retest this?

Add math function bin(a: long): string.

50e0c3b

viirya added 2 commits June 9, 2015 23:17

Merge remote-tracking branch 'upstream/master' into expr_bin

824f761

Add bin to FunctionRegistry.

c0c3197

rxin reviewed Jun 9, 2015
View reviewed changes

viirya added 2 commits June 10, 2015 15:49

Merge remote-tracking branch 'upstream/master' into expr_bin

0f78682

For comments.

4cb764d

rxin reviewed Jun 10, 2015
View reviewed changes

viirya added 2 commits June 11, 2015 16:10

Merge remote-tracking branch 'upstream/master' into expr_bin

ac2bacd

Fix python style.

7a0196f

Add @ignore_unicode_prefix.

d4f4774

Merge remote-tracking branch 'upstream/master' into expr_bin

dea9c12

Conflicts: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/math.scala sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala

Merge remote-tracking branch 'upstream/master' into expr_bin

0cf20f2

Conflicts: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/math.scala

davies reviewed Jun 17, 2015
View reviewed changes

viirya added 3 commits June 18, 2015 14:30

Merge remote-tracking branch 'upstream/master' into expr_bin

cf62b95

For comments.

0677f1a

Remove AbstractUnaryMathExpression and let BIN inherit UnaryExpression.

07e1c8f

davies reviewed Jun 19, 2015
View reviewed changes

viirya added 2 commits June 20, 2015 01:10

Merge remote-tracking branch 'upstream/master' into expr_bin

05ebe63

Remove unused funcName.

cde33b5

asfgit closed this in 2c59d5c Jun 19, 2015

viirya deleted the expr_bin branch December 27, 2023 18:17

[SPARK-8207][SQL] Add math function bin #6721

[SPARK-8207][SQL] Add math function bin #6721

Uh oh!

Conversation

viirya commented Jun 9, 2015

Uh oh!

SparkQA commented Jun 9, 2015

Uh oh!

SparkQA commented Jun 9, 2015

Uh oh!

rxin Jun 9, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 10, 2015

Uh oh!

rxin Jun 10, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 11, 2015

Uh oh!

SparkQA commented Jun 11, 2015

Uh oh!

viirya commented Jun 11, 2015

Uh oh!

SparkQA commented Jun 11, 2015

Uh oh!

SparkQA commented Jun 11, 2015

Uh oh!

SparkQA commented Jun 13, 2015

Uh oh!

SparkQA commented Jun 15, 2015

Uh oh!

davies Jun 17, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 18, 2015

Uh oh!

SparkQA commented Jun 18, 2015

Uh oh!

davies Jun 19, 2015

Choose a reason for hiding this comment

Uh oh!

davies commented Jun 19, 2015

Uh oh!

SparkQA commented Jun 19, 2015

Uh oh!

viirya commented Jun 19, 2015

Uh oh!

Uh oh!