-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Keras-like API Advanced Activations, dropout and noise layers (#2222)
* Keras API for ELU * fix style check error * fix weightConverter * add one more unit test for ELU * remove blank line * Keras API for LeakyReLU * remove useless empty lines in LeakyReLU * add GaussianDropout * add GaussianNoise * remove UID and unnecessary import * fix two Gaussian unit test * add layer Masking * add layer SpatialDropout1D * change 3D to 4D * Revert "change 3D to 4D" This reverts commit 9efdb0a. * change unit test from 4D to 3D * add layer SpatialDropout2D * add layer PReLU. Unit test success without weight * add 3D unit test for PReLU * add layer ParametricSoftPlus. Unit test success without weight * add layer SpatialDropout3D * add layer ThresholdedReLU * fix the above problems * fix problems * add format lowercase to support both uppercase and lowercase * fix format problem * SReLU * add documentation and serializer * remove a blank in documentation and change inputshape from var to val * delete four files * update * modify * modify problem * modify * update * modify style
- Loading branch information
1 parent
d884178
commit 99c6410
Showing
32 changed files
with
1,192 additions
and
22 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
57 changes: 57 additions & 0 deletions
57
spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/ELU.scala
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| /* | ||
| * Copyright 2016 The BigDL Authors. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| package com.intel.analytics.bigdl.nn.keras | ||
|
|
||
| import com.intel.analytics.bigdl.nn.abstractnn._ | ||
| import com.intel.analytics.bigdl.tensor.Tensor | ||
| import com.intel.analytics.bigdl.tensor.TensorNumericMath.TensorNumeric | ||
| import com.intel.analytics.bigdl.utils.Shape | ||
|
|
||
| import scala.reflect.ClassTag | ||
|
|
||
| /** | ||
| * Exponential Linear Unit. | ||
| * It follows: | ||
| * `f(x) = alpha * (exp(x) - 1.) for x < 0`, | ||
| * `f(x) = x for x >= 0`. | ||
| * | ||
| * When you use this layer as the first layer of a model, you need to provide the argument | ||
| * inputShape (a Single Shape, does not include the batch dimension). | ||
| * | ||
| * @param alpha Double, scale for the negative factor. | ||
| * @tparam T Numeric type of parameter(e.g. weight, bias). Only support float/double now. | ||
| */ | ||
| class ELU[T: ClassTag]( | ||
| val alpha: Double = 1.0, | ||
| val inputShape: Shape = null)(implicit ev: TensorNumeric[T]) | ||
| extends KerasLayer[Tensor[T], Tensor[T], T](KerasLayer.addBatch(inputShape)) { | ||
|
|
||
| override def doBuild(inputShape: Shape): AbstractModule[Tensor[T], Tensor[T], T] = { | ||
| val layer = com.intel.analytics.bigdl.nn.ELU( | ||
| alpha = alpha, | ||
| inplace = false) | ||
| layer.asInstanceOf[AbstractModule[Tensor[T], Tensor[T], T]] | ||
| } | ||
| } | ||
|
|
||
| object ELU { | ||
| def apply[@specialized(Float, Double) T: ClassTag]( | ||
| alpha: Double = 1.0, | ||
| inputShape: Shape = null)(implicit ev: TensorNumeric[T]): ELU[T] = { | ||
| new ELU[T](alpha, inputShape) | ||
| } | ||
| } |
53 changes: 53 additions & 0 deletions
53
spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/GaussianDropout.scala
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| /* | ||
| * Copyright 2016 The BigDL Authors. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| package com.intel.analytics.bigdl.nn.keras | ||
|
|
||
| import com.intel.analytics.bigdl.nn.abstractnn._ | ||
| import com.intel.analytics.bigdl.tensor.Tensor | ||
| import com.intel.analytics.bigdl.tensor.TensorNumericMath.TensorNumeric | ||
| import com.intel.analytics.bigdl.utils.Shape | ||
|
|
||
| import scala.reflect.ClassTag | ||
|
|
||
| /** | ||
| * Apply multiplicative 1-centered Gaussian noise. | ||
| * As it is a regularization layer, it is only active at training time. | ||
| * When you use this layer as the first layer of a model, you need to provide the argument | ||
| * inputShape (a Single Shape, does not include the batch dimension). | ||
| * | ||
| * @param p Double, drop probability (as with `Dropout`). | ||
| * The multiplicative noise will have standard deviation `sqrt(p / (1 - p))`. | ||
| * @tparam T Numeric type of parameter(e.g. weight, bias). Only support float/double now. | ||
| */ | ||
| class GaussianDropout[T: ClassTag]( | ||
| val p: Double, | ||
| val inputShape: Shape = null)(implicit ev: TensorNumeric[T]) | ||
| extends KerasLayer[Tensor[T], Tensor[T], T](KerasLayer.addBatch(inputShape)) { | ||
|
|
||
| override def doBuild(inputShape: Shape): AbstractModule[Tensor[T], Tensor[T], T] = { | ||
| val layer = com.intel.analytics.bigdl.nn.GaussianDropout(rate = p) | ||
| layer.asInstanceOf[AbstractModule[Tensor[T], Tensor[T], T]] | ||
| } | ||
| } | ||
|
|
||
| object GaussianDropout { | ||
| def apply[@specialized(Float, Double) T: ClassTag]( | ||
| p: Double, | ||
| inputShape: Shape = null)(implicit ev: TensorNumeric[T]): GaussianDropout[T] = { | ||
| new GaussianDropout[T](p, inputShape) | ||
| } | ||
| } |
56 changes: 56 additions & 0 deletions
56
spark/dl/src/main/scala/com/intel/analytics/bigdl/nn/keras/GaussianNoise.scala
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| /* | ||
| * Copyright 2016 The BigDL Authors. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| package com.intel.analytics.bigdl.nn.keras | ||
|
|
||
|
|
||
| import com.intel.analytics.bigdl.nn.abstractnn._ | ||
| import com.intel.analytics.bigdl.tensor.Tensor | ||
| import com.intel.analytics.bigdl.tensor.TensorNumericMath.TensorNumeric | ||
| import com.intel.analytics.bigdl.utils.Shape | ||
|
|
||
| import scala.reflect.ClassTag | ||
|
|
||
| /** | ||
| * Apply additive zero-centered Gaussian noise. | ||
| * This is useful to mitigate overfitting (you could see it as a form of random data augmentation). | ||
| * Gaussian Noise (GS) is a natural choice as corruption process for real valued inputs. | ||
| * As it is a regularization layer, it is only active at training time. | ||
| * | ||
| * When you use this layer as the first layer of a model, you need to provide the argument | ||
| * inputShape (a Single Shape, does not include the batch dimension). | ||
| * | ||
| * @param sigma Double, standard deviation of the noise distribution. | ||
| * @tparam T Numeric type of parameter(e.g. weight, bias). Only support float/double now. | ||
| */ | ||
| class GaussianNoise[T: ClassTag]( | ||
| val sigma: Double, | ||
| val inputShape: Shape = null)(implicit ev: TensorNumeric[T]) | ||
| extends KerasLayer[Tensor[T], Tensor[T], T](KerasLayer.addBatch(inputShape)) { | ||
|
|
||
| override def doBuild(inputShape: Shape): AbstractModule[Tensor[T], Tensor[T], T] = { | ||
| val layer = com.intel.analytics.bigdl.nn.GaussianNoise(stddev = sigma) | ||
| layer.asInstanceOf[AbstractModule[Tensor[T], Tensor[T], T]] | ||
| } | ||
| } | ||
|
|
||
| object GaussianNoise { | ||
| def apply[@specialized(Float, Double) T: ClassTag]( | ||
| sigma: Double, | ||
| inputShape: Shape = null)(implicit ev: TensorNumeric[T]): GaussianNoise[T] = { | ||
| new GaussianNoise[T](sigma, inputShape) | ||
| } | ||
| } |
Oops, something went wrong.