Keras-like API Advanced Activations, dropout and noise layers #2222

Quincy2014 · 2018-01-24T05:35:48Z

Add Keras-like API for some layers.

layers:
ELU,
LeakyReLU,
ThresholdedReLU,
SReLU,
Masking,
GaussianDropout,
GaussianNoise,
SpatialDropout1D,
SpatialDropout2D,
SpatialDropout3D.

test:
Each layer has a corresponding unit test in xxxSpec.scala.

zhichao-li · 2018-01-24T06:14:24Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/ELU.scala

+import scala.reflect.ClassTag
+
+
+@SerialVersionUID( - 6274543584907751212L)


have you re-generated the UID?

hkvision · 2018-01-24T06:19:20Z

spark/dl/src/test/scala/com/intel/analytics/bigdl/keras/nn/ELUSpec.scala

+    val seq = KSequential[Float]()
+    val elu = ELU[Float](1.0, inputShape = Shape(3))
+    seq.add(elu)
+    def weightConverter(in: Array[Tensor[Float]]): Array[Tensor[Float]] = Array(in(0).t(), in(1))


Why does ELU need a weightconverter? I don't think ELU has weights.

hkvision · 2018-01-24T06:25:59Z

spark/dl/src/test/scala/com/intel/analytics/bigdl/keras/nn/ELUSpec.scala

+      kerasCode, weightConverter)
+
+  }
+}


Add a new line at the end of a file

hkvision · 2018-01-24T07:15:38Z

spark/dl/src/test/scala/com/intel/analytics/bigdl/keras/nn/ELUSpec.scala

+      """.stripMargin
+    val seq = KSequential[Float]()
+    val elu = ELU[Float](2.7, inputShape = Shape(3, 24, 24))
+


Remove the blank line.

Quincy2014 · 2018-01-24T08:28:22Z

Jenkins passed. http://172.168.2.101:8080/view/PR-V/job/BigDL-PR-V/836

hkvision · 2018-01-24T09:03:17Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/LeakyReLU.scala

+
+
+@SerialVersionUID( - 1470253389268877486L)
+class LeakyReLU[T: ClassTag](private val alpha: Double = 0.01,


Why this alpha is private? cc @zhichao-li

Pls confirm this with the original author. we can open it if no objections.

In the original nn/LeakyReLU, the negval is private so I set the alpha as private.

cc @psyyz10 @qiuxin2012 Any comments on this?

hkvision · 2018-01-24T09:03:49Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/LeakyReLU.scala

+  }
+}
+
+


Remove useless empty lines

hkvision · 2018-01-24T09:04:46Z

spark/dl/src/test/scala/com/intel/analytics/bigdl/keras/nn/LeakyReLUSpec.scala

+  }
+}
+
+


Remove useless empty lines

hkvision · 2018-01-25T05:31:03Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/GaussianDropout.scala

+
+package com.intel.analytics.bigdl.nn.keras
+
+import com.intel.analytics.bigdl._


Remove this. This is unnecessary.

hkvision · 2018-01-25T05:31:41Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/GaussianDropout.scala

+
+import scala.reflect.ClassTag
+
+@SerialVersionUID( 5198738230229027831L)


No need for serialUID for layers cc @yiheng #2224

hkvision · 2018-01-25T05:32:06Z

spark/dl/src/test/scala/com/intel/analytics/bigdl/keras/nn/GaussianDropoutSpec.scala

+  * Created by intel on 2018/1/25.
+  */
+class GaussianDropoutSpec {
+


hkvision · 2018-01-25T05:32:15Z

spark/dl/src/test/scala/com/intel/analytics/bigdl/keras/nn/GaussianDropoutSpec.scala

@@ -0,0 +1,8 @@
+package com.intel.analytics.bigdl.keras.nn
+
+/**


hkvision · 2018-01-25T05:33:49Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/ELU.scala

+
+package com.intel.analytics.bigdl.nn.keras
+
+import com.intel.analytics.bigdl._


unnecessary.

hkvision · 2018-01-25T05:34:07Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/LeakyReLU.scala

+
+package com.intel.analytics.bigdl.nn.keras
+
+import com.intel.analytics.bigdl._


unnecessary

hkvision · 2018-01-25T06:13:33Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/GaussianNoise.scala

+
+import scala.reflect.ClassTag
+
+@SerialVersionUID( - 2224693793797534699L)


hkvision · 2018-01-25T06:14:54Z

spark/dl/src/test/scala/com/intel/analytics/bigdl/keras/nn/GaussianDropoutSpec.scala

+  "GaussianDropout forward and backward" should "work properly" in {
+    val seq = KSequential[Float]()
+    val input = Tensor[Float](Array(2, 28, 28, 1)).rand()
+    val gaussiandropout = GaussianDropout[Float](0.6, inputShape = Shape(3))


Input tensor is of shape (2, 28, 28, 1) but in the layer you specify inputShape=Shape(3)??

hkvision · 2018-01-25T06:15:27Z

spark/dl/src/test/scala/com/intel/analytics/bigdl/keras/nn/GaussianNoiseSpec.scala

+  "GaussianNoise forward and backward" should "work properly" in {
+    val seq = KSequential[Float]()
+    val input = Tensor[Float](Array(2, 28, 28, 1)).rand()
+    val gaussiannoise = GaussianNoise[Float](0.6, inputShape = Shape(3))


Same problem as gaussiandropout.

zhichao-li · 2018-01-26T02:32:03Z

Please enrich the description for the layers you added.

hkvision · 2018-01-26T06:25:37Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/LeakyReLU.scala

+import scala.reflect.ClassTag
+
+
+class LeakyReLU[T: ClassTag](private val alpha: Double = 0.01,


Seems can delete private cc @psyyz10

hkvision · 2018-01-26T11:26:09Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/LeakyReLU.scala

@@ -15,8 +15,10 @@
 */
 package com.intel.analytics.bigdl.nn

-import com.intel.analytics.bigdl.nn.abstractnn.TensorModule
+


Remove the empty line.

hkvision · 2018-01-26T11:26:20Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/LeakyReLU.scala

 import com.intel.analytics.bigdl.tensor._
+


Remove the empty line.

hkvision · 2018-01-26T11:26:25Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/LeakyReLU.scala

@@ -32,12 +34,15 @@ import scala.reflect.ClassTag
 *                using extra state memory
 */

+


Remove the empty line.

hkvision · 2018-01-26T11:26:45Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/LeakyReLU.scala

  import LeakyReLU._
+  private val negVal = ev.fromType[Double](negval)


Why do you add this?

Someone else add this. There's a conflict so I fix the conflict.

Can you have a look in the latest code of this class? I don't think this line is still there.

hkvision · 2018-01-26T11:27:56Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/Threshold.scala

@@ -34,10 +34,9 @@ import scala.reflect.ClassTag
 * @param ip inplace mode
 */

-@SerialVersionUID(3953292249027271493L)


Keep the UID in old layers unchanged. Others may propose a PR to remove all of them.
For new layers, we simply don't add UID.

hkvision · 2018-01-26T11:28:43Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/PReLU.scala

+  }
+}
+
+


Remove the empty lines.

hkvision · 2018-01-26T11:30:16Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/ParametricSoftPlus.scala

+  extends KerasLayer[Tensor[T], Tensor[T], T](KerasLayer.addBatch(inputShape)) {
+
+  override def doBuild(inputShape: Shape): AbstractModule[Tensor[T], Tensor[T], T] = {
+    if (round(alpha_init * beta_init) == 1.0) {


We only support alpha_init * beta_init == 1.0, so in this case, we just need one parameter, not two.

In converter.py, there are also alpha and beta two parameters.

In converter, we have to convert Keras layer into BigDL layer, so we should follow the parameters in Keras. But in new API, if we don't support every parameter in Keras, we need to make modifications instead of throwing exceptions.

hkvision · 2018-01-26T11:30:52Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/ParametricSoftPlus.scala

+      layer.asInstanceOf[AbstractModule[Tensor[T], Tensor[T], T]]
+    }
+    else {
+      throw new Exception("Only alpha_init = 1/beta_init is supported for now")


Don't throw exception here. If the arguments we support is less than Keras, we can just eliminate the unsupported arguments.

hkvision · 2018-02-08T05:42:30Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/PReLU.scala

+import com.intel.analytics.bigdl.tensor.TensorNumericMath.TensorNumeric
+import com.intel.analytics.bigdl.utils.Shape
+
+import scala.reflect.ClassTag


Remove this file for now. Remove its tests. I will fix and add it afterwards.

hkvision · 2018-02-08T05:42:43Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/ParametricSoftPlus.scala

+
+import scala.reflect.ClassTag
+
+/**


Remove this layer.

hkvision · 2018-02-08T05:43:06Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/SReLU.scala

+ * `f(x) = x for t^r > x > t^l`,
+ * `f(x) = t^l + a^l(x - t^l) for x <= t^l`.
+ *
+ * @param SharedAxes the axes along which to share learnable parameters for the activation function.


Capitalize & Align

Add Array of int.

hkvision · 2018-02-08T05:43:32Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/SReLU.scala

+ *                    with output shape `(batch, height, width, channels)`,
+ *                    and you wish to share parameters across space
+ *                    so that each filter only has one set of parameters,
+ *                    set `shared_axes=[1, 2]`.


Array(1, 2)

hkvision · 2018-02-08T05:44:33Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/SReLU.scala

+    val shape = inputShape.toSingle().toArray
+    if (SharedAxes == null) {
+      val layer = com.intel.analytics.bigdl.nn.SReLU(shape.slice(1, shape.length))
+      layer.asInstanceOf[AbstractModule[Tensor[T], Tensor[T], T]]


These if else seems can be integrated?

hkvision · 2018-02-08T05:44:51Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/SpatialDropout1D.scala

+ *  decrease. In this case, SpatialDropout1D will help promote independence
+ *  between feature maps and should be used instead.
+ *
+ * @param p float between 0 and 1. Fraction of the input units to drop.


hkvision · 2018-02-08T05:45:27Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/SpatialDropout2D.scala

+ *  between feature maps and should be used instead.
+ *
+ * @param p float between 0 and 1. Fraction of the input units to drop.
+ * @param format  'NCHW' or 'NHWC'.


Modify this according to Convolution2D. Change to dimOrdering.

hkvision · 2018-02-08T05:45:37Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/SpatialDropout3D.scala

+ *  between feature maps and should be used instead.
+ *
+ * @param p float between 0 and 1. Fraction of the input units to drop.
+ * @param format  'NCHW' or 'NHWC'.


hkvision · 2018-02-08T05:45:42Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/SpatialDropout3D.scala

+ *  decrease. In this case, SpatialDropout3D will help promote independence
+ *  between feature maps and should be used instead.
+ *
+ * @param p float between 0 and 1. Fraction of the input units to drop.


hkvision · 2018-02-08T05:46:10Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/SpatialDropout2D.scala

+ * @param format  'NCHW' or 'NHWC'.
+ *                 In 'NCHW' mode, the channels dimension (the depth)
+ *                 is at index 1, in 'NHWC' mode is it at index 4.
+ * @tparam T The numeric type in the criterion, usually which are [[Float]] or [[Double]]


hkvision · 2018-02-08T05:46:24Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/SpatialDropout3D.scala

+    s"$format is not supported")
+
+  override def doBuild(inputShape: Shape): AbstractModule[Tensor[T], Tensor[T], T] = {
+


remove empty lines.

hkvision · 2018-02-08T05:46:39Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/ThresholdedReLU.scala

+ * `f(x) = x for x > theta`,
+ * `f(x) = 0 otherwise`.
+ *
+ * @param theta float >= 0. Threshold location of activation.


hkvision · 2018-02-08T05:48:13Z

spark/dl/src/test/scala/com/intel/analytics/bigdl/keras/nn/PReLUSpec.scala

@@ -0,0 +1,57 @@
+/*


remove this file.

hkvision · 2018-02-08T05:48:22Z

spark/dl/src/test/scala/com/intel/analytics/bigdl/keras/nn/ParametricSoftPlusSpec.scala

@@ -0,0 +1,57 @@
+/*


remove this file.

hkvision · 2018-02-08T05:48:39Z

spark/dl/src/test/scala/com/intel/analytics/bigdl/keras/nn/SReLUSpec.scala

+      """
+        |input_tensor = Input(shape=[2, 3])
+        |input = np.random.uniform(-1, 1, [1, 2, 3])
+        |# input = np.array([[[0.1, 0.2, 0.3], [0.1, 0.2, 0.3]]])


remove comments....

hkvision · 2018-02-08T05:49:56Z

spark/dl/src/test/scala/com/intel/analytics/bigdl/keras/nn/SReLUSpec.scala

+        |input_tensor = Input(shape=[2, 3])
+        |input = np.random.uniform(-1, 1, [1, 2, 3])
+        |# input = np.array([[[0.1, 0.2, 0.3], [0.1, 0.2, 0.3]]])
+        |output_tensor = SReLU(a_left_init='one', t_right_init='one')(input_tensor)


remove a_left_init and t_right_init

hkvision · 2018-02-08T05:50:02Z

spark/dl/src/test/scala/com/intel/analytics/bigdl/keras/nn/SReLUSpec.scala

+      kerasCode)
+  }
+
+  "SReLU 3D" should "be the same as Keras" in {


Both ut are 3d input. This is OK. Change name of ut to SReLU with shared axes

hkvision · 2018-02-08T07:43:48Z

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/SpatialDropout2D.scala

+ */
+class SpatialDropout2D[T: ClassTag](
+   val p: Double = 0.5,
+   val format: DataFormat = DataFormat.NCHW,


dimOrdering

hkvision

LGTM

hkvision · 2018-02-08T08:41:06Z

jenkins passed http://172.168.2.101:8080/view/PR-V(new)-Pipeline/job/BigDL-PR-Validation-v3/173/

…analytics#2222) * Keras API for ELU * fix style check error * fix weightConverter * add one more unit test for ELU * remove blank line * Keras API for LeakyReLU * remove useless empty lines in LeakyReLU * add GaussianDropout * add GaussianNoise * remove UID and unnecessary import * fix two Gaussian unit test * add layer Masking * add layer SpatialDropout1D * change 3D to 4D * Revert "change 3D to 4D" This reverts commit 9efdb0a. * change unit test from 4D to 3D * add layer SpatialDropout2D * add layer PReLU. Unit test success without weight * add 3D unit test for PReLU * add layer ParametricSoftPlus. Unit test success without weight * add layer SpatialDropout3D * add layer ThresholdedReLU * fix the above problems * fix problems * add format lowercase to support both uppercase and lowercase * fix format problem * SReLU * add documentation and serializer * remove a blank in documentation and change inputshape from var to val * delete four files * update * modify * modify problem * modify * update * modify style

hkvision requested review from zhichao-li and hkvision January 24, 2018 06:05

zhichao-li reviewed Jan 24, 2018

View reviewed changes

hkvision reviewed Jan 24, 2018

View reviewed changes

hkvision changed the title ~~[WIP] Keras API for ELU~~ [WIP] More layers for Keras-like API Jan 24, 2018

hkvision reviewed Jan 24, 2018

View reviewed changes

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/LeakyReLU.scala Outdated

}

}

Copy link

Contributor

hkvision Jan 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove useless empty lines

hkvision reviewed Jan 24, 2018

View reviewed changes

spark/dl/src/test/scala/com/intel/analytics/bigdl/keras/nn/LeakyReLUSpec.scala Outdated

}

}

Copy link

Contributor

hkvision Jan 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove useless empty lines

hkvision reviewed Jan 25, 2018

View reviewed changes

hkvision reviewed Jan 26, 2018

View reviewed changes

spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/PReLU.scala Outdated

}

}

Copy link

Contributor

hkvision Jan 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove the empty lines.

hkvision reviewed Jan 26, 2018

View reviewed changes

hkvision reviewed Feb 8, 2018

View reviewed changes

spark/dl/src/test/scala/com/intel/analytics/bigdl/keras/nn/PReLUSpec.scala Outdated

@@ -0,0 +1,57 @@

/*

Copy link

Contributor

hkvision Feb 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this file.

hkvision reviewed Feb 8, 2018

View reviewed changes

Quincy2014 force-pushed the master branch from c114b15 to 4392d45 Compare February 8, 2018 07:30

Quincy2014 added 3 commits February 8, 2018 15:31

delete four files

4392d45

update

5a75157

modify

32ff46e

hkvision reviewed Feb 8, 2018

View reviewed changes

Quincy2014 added 2 commits February 8, 2018 15:51

modify problem

9c596f2

modify

f8beee3

hkvision approved these changes Feb 8, 2018

View reviewed changes

Quincy2014 added 2 commits February 8, 2018 15:57

update

6fabd80

modify style

1ada8b3

hkvision merged commit 99c6410 into intel-analytics:master Feb 8, 2018

		import scala.reflect.ClassTag


		@SerialVersionUID( - 6274543584907751212L)



		@SerialVersionUID( - 1470253389268877486L)
		class LeakyReLU[T: ClassTag](private val alpha: Double = 0.01,


		package com.intel.analytics.bigdl.nn.keras

		import com.intel.analytics.bigdl._


		import scala.reflect.ClassTag

		@SerialVersionUID( 5198738230229027831L)

		@@ -0,0 +1,8 @@
		package com.intel.analytics.bigdl.keras.nn

		/**


		import scala.reflect.ClassTag

		@SerialVersionUID( - 2224693793797534699L)

		@@ -32,12 +34,15 @@ import scala.reflect.ClassTag
		* using extra state memory
		*/

		import LeakyReLU._
		private val negVal = ev.fromType[Double](negval)

		s"$format is not supported")

		override def doBuild(inputShape: Shape): AbstractModule[Tensor[T], Tensor[T], T] = {

Keras-like API Advanced Activations, dropout and noise layers #2222

Keras-like API Advanced Activations, dropout and noise layers #2222

Conversation

Quincy2014 commented Jan 24, 2018 • edited by hkvision Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Quincy2014 commented Jan 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhichao-li commented Jan 26, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hkvision left a comment

Choose a reason for hiding this comment

hkvision commented Feb 8, 2018

Quincy2014 commented Jan 24, 2018 •

edited by hkvision

Loading