Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Layer definitions and declarations re-organization and documentation #376

Merged
merged 7 commits into from
May 19, 2014
Merged

Layer definitions and declarations re-organization and documentation #376

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
2779ce9
layer definition reorganization and documentation
sergeyk Apr 29, 2014
48a8a64
Split all loss layers into own .cpp files
sergeyk Apr 29, 2014
4051cef
Cosmetic change in prep for data layer work
sergeyk Apr 29, 2014
d01c6ed
fwd/back math docs for neuron layers
sergeyk Apr 29, 2014
efeaf17
Lil’ more docstring, and cosmetic change in EuclideanLossLayer
sergeyk May 19, 2014
1b735b5
Cosmetic change in ConcatLayer
sergeyk May 19, 2014
c5db25b
Incorporated Evan’s comments for neuron layers
sergeyk May 19, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Incorporated Evan’s comments for neuron layers
  • Loading branch information
@sergeyk
sergeyk committed May 19, 2014
commit c5db25b2033fbe44712d5a709e5af7fbabc4308d
16 changes: 11 additions & 5 deletions include/caffe/neuron_layers.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,11 +124,13 @@ class PowerLayer : public NeuronLayer<Dtype> {
};

/* ReLULayer
Rectified Linear Unit non-linearity: fast and stable.
Rectified Linear Unit non-linearity.
The simple max is fast to compute, and the function does not saturate.

y = max(0, x).

y' = x > 0
y' = 0 if x < 0
y' = 1 if x > 0
*/
template <typename Dtype>
class ReLULayer : public NeuronLayer<Dtype> {
Expand All @@ -149,10 +151,14 @@ class ReLULayer : public NeuronLayer<Dtype> {
};

/* SigmoidLayer
Sigmoid function non-linearity: a classic.
Sigmoid function non-linearity, a classic choice in neural networks.
Note that the gradient vanishes as the values move away from 0.
The ReLULayer is often a better choice for this reason.

y = 1. / (1 + exp(-x))

y ' = exp(x) / (1 + exp(x))^2
or
y' = y * (1 - y)
*/
template <typename Dtype>
Expand All @@ -173,11 +179,11 @@ class SigmoidLayer : public NeuronLayer<Dtype> {
};

/* TanHLayer
Hyperbolic tangent non-linearity.
Hyperbolic tangent non-linearity, popular in auto-encoders.

y = 1. * (exp(2x) - 1) / (exp(2x) + 1)

y' = 1 - [(exp(2x) - 1) / (exp(2x) + 1)] ^ 2
y' = 1 - ( (exp(2x) - 1) / (exp(2x) + 1) ) ^ 2
*/
template <typename Dtype>
class TanHLayer : public NeuronLayer<Dtype> {
Expand Down