You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: doc/design/auto_gradient_check.md
+36-36Lines changed: 36 additions & 36 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,17 +1,17 @@
1
1
## Auto Gradient Checker Design
2
2
3
3
## Backgraound：
4
-
-Operator forward computing is easy to check if the result is right because it has a clear definition. **But** backpropagation is a notoriously difficult algorithm to debug and get right:
5
-
-1. you should get the right backpropagation formula according to the forward computation.
6
-
-2. you should implement it right in CPP.
7
-
-3. it's difficult to prepare test data.
4
+
-Generally, it is easy to check whether the forward computation of an Operator is correct or not. However, backpropagation is a notoriously difficult algorithm to debug and get right:
5
+
1. you should get the right backpropagation formula according to the forward computation.
6
+
2. you should implement it right in CPP.
7
+
3. it's difficult to prepare test data.
8
8
9
-
- Auto gradient check gets a numeric gradient by forward Operator and use it as a reference of the backward Operator's result. It has several advantages:
10
-
-1.numeric gradient checker only need forward operator.
11
-
-2. user only need to prepare the input data for forward Operator.
9
+
- Auto gradient checking gets a numerical gradient by forward Operator and use it as a reference of the backward Operator's result. It has several advantages:
10
+
1.numerical gradient checker only need forward operator.
11
+
2. user only need to prepare the input data for forward Operator.
12
12
13
13
## Mathematical Theory
14
-
The following two document from stanford has a detailed explanation of how to get numeric gradient and why it's useful.
14
+
The following two document from Stanford has a detailed explanation of how to get numerical gradient and why it's useful.
15
15
16
16
-[Gradient checking and advanced optimization(en)](http://deeplearning.stanford.edu/wiki/index.php/Gradient_checking_and_advanced_optimization)
17
17
-[Gradient checking and advanced optimization(cn)](http://ufldl.stanford.edu/wiki/index.php/%E6%A2%AF%E5%BA%A6%E6%A3%80%E9%AA%8C%E4%B8%8E%E9%AB%98%E7%BA%A7%E4%BC%98%E5%8C%96)
@@ -20,7 +20,7 @@ The following two document from stanford has a detailed explanation of how to ge
20
20
## Numeric Gradient Implementation
21
21
### Python Interface
22
22
```python
23
-
defget_numeric_gradient(op,
23
+
defget_numerical_gradient(op,
24
24
input_values,
25
25
output_name,
26
26
input_to_check,
@@ -30,13 +30,13 @@ def get_numeric_gradient(op,
30
30
Get Numeric Gradient for an operator's input.
31
31
32
32
:param op: C++ operator instance, could be an network
33
-
:param input_values: The input variables. Should be an dictionary, key is
34
-
variable name. Value is numpy array.
33
+
:param input_values: The input variables. Should be an dictionary, whose key is
34
+
variable name, and value is numpy array.
35
35
:param output_name: The final output variable name.
36
-
:param input_to_check: The input variable need to get gradient.
36
+
:param input_to_check: The input variable with respect to which to compute the gradient.
37
37
:param delta: The perturbation value for numeric gradient method. The
38
38
smaller delta is, the more accurate result will get. But if that delta is
39
-
too small, it could occur numerical stability problem.
39
+
too small, it will suffer from numerical stability problem.
40
40
:param local_scope: The local scope used for get_numeric_gradient.
41
41
:return: The gradient array in numpy format.
42
42
"""
@@ -45,28 +45,28 @@ def get_numeric_gradient(op,
45
45
### Explaination:
46
46
47
47
- Why need `output_name`
48
-
-One Operator may have multiple Output, you can get independent gradient from each Output. So user should set one output to calculate.
48
+
-An Operator may have multiple Output, one can get independent gradient from each Output. So caller should specify the name of the output variable.
49
49
50
50
- Why need `input_to_check`
51
-
- One operator may have multiple inputs. Gradient Op can calculate the gradient of these Inputs at the same time. But Numeric Gradient needs to calculate them one by one. So `get_numeric_gradient` is designed to calculate the gradient for one input. If you need to compute multiple inputs, you can call `get_numeric_gradient` multiple times.
51
+
- One operator may have multiple inputs. Gradient Op can calculate the gradient of these inputs at the same time. But Numeric Gradient needs to calculate them one by one. So `get_numeric_gradient` is designed to calculate the gradient for one input. If you need to compute multiple inputs, you can call `get_numeric_gradient` multiple times.
52
52
53
53
54
54
### Core Algorithm Implementation
55
55
56
56
57
57
```python
58
-
# we only compute gradient of one element each time.
59
-
# we use a for loop to compute the gradient of every element.
58
+
# we only compute gradient of one element a time.
59
+
# we use a for loop to compute the gradient of each element.
60
60
for i inxrange(tensor_size):
61
-
# get one input element throw it's index i.
61
+
# get one input element by its index i.
62
62
origin = tensor_to_check.get_float_element(i)
63
63
64
-
# add delta to it, run op and then get the sum of the result tensor.
64
+
# add delta to it, run op and then get the new value of the result tensor.
65
65
x_pos = origin + delta
66
66
tensor_to_check.set_float_element(i, x_pos)
67
67
y_pos = get_output()
68
68
69
-
# plus delta to this element, run op and get the sum of the result tensor.
69
+
# plus delta to this element, run op and get the new value of the result tensor.
70
70
x_neg = origin - delta
71
71
tensor_to_check.set_float_element(i, x_neg)
72
72
y_neg = get_output()
@@ -85,15 +85,15 @@ def get_numeric_gradient(op,
85
85
86
86
Each Operator Kernel has three kinds of Gradient:
87
87
88
-
-1.Numeric Gradient
89
-
-2. CPU Operator Gradient
90
-
-3. GPU Operator Gradient(if supported)
88
+
1.Numerical gradient
89
+
2. CPU kernel gradient
90
+
3. GPU kernel gradient (if supported)
91
91
92
-
Numeric Gradient Only relies on forward Operator. So we use Numeric Gradient as the reference value.
92
+
The numerical gradient only relies on forward Operator. So we use the numerical gradient as the reference value. And the gradient checking is performed in the following three steps:
93
93
94
-
-1. calculate the numeric gradient.
95
-
-2. calculate CPU kernel Gradient with the backward Operator and compare it with the numeric gradient.
96
-
-3. calculate GPU kernel Gradient with the backward Operator and compare it with the numeric gradient.(if support GPU)
94
+
1. calculate the numerical gradient
95
+
2. calculate CPU kernel gradient with the backward Operator and compare it with the numerical gradient
96
+
3. calculate GPU kernel gradient with the backward Operator and compare it with the numeric gradient(if supported)
97
97
98
98
#### Python Interface
99
99
@@ -110,8 +110,8 @@ Numeric Gradient Only relies on forward Operator. So we use Numeric Gradient as
110
110
:param forward_op: used to create backward_op
111
111
:param input_vars: numpy value of input variable. The following
112
112
computation will use these variables.
113
-
:param inputs_to_check: inputs var names that should check gradient.
114
-
:param output_name: output name that used to
113
+
:param inputs_to_check: the input variable with respect to which to compute the gradient.
114
+
:param output_name: The final output variable name.
115
115
:param max_relative_error: The relative tolerance parameter.
116
116
:param no_grad_set: used when create backward ops
117
117
:param only_cpu: only compute and check gradient on cpu kernel.
@@ -120,24 +120,24 @@ Numeric Gradient Only relies on forward Operator. So we use Numeric Gradient as
120
120
```
121
121
122
122
### How to check if two numpy array is close enough?
123
-
if `abs_numeric_grad` is nearly zero, then use abs error for numeric_grad, not relative
123
+
if `abs_numerical_grad` is nearly zero, then use abs error for numerical_grad
Copy file name to clipboardExpand all lines: doc/design/graph.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -56,7 +56,7 @@ For each parameter, like W and b created by `layer.fc`, marked as double circles
56
56
57
57
## Block and Graph
58
58
59
-
The word block and graph are interchangable in the desgin of PaddlePaddle. A [Block[(https://github.com/PaddlePaddle/Paddle/pull/3708) is a metaphore of the code and local variables in a pair of curly braces in programming languages, where operators are like statements or instructions. A graph of operators and variables is a representation of the block.
59
+
The word block and graph are interchangable in the desgin of PaddlePaddle. A [Block](https://github.com/PaddlePaddle/Paddle/pull/3708) is a metaphore of the code and local variables in a pair of curly braces in programming languages, where operators are like statements or instructions. A graph of operators and variables is a representation of the block.
60
60
61
61
A Block keeps operators in an array `BlockDesc::ops`
62
62
@@ -67,4 +67,4 @@ message BlockDesc {
67
67
}
68
68
```
69
69
70
-
in the order that there appear in user programs, like the Python program at the beginning of this article. We can imagine that in `ops`, we have some forward operators, followed by some gradient operators, and then some optimization operators.
70
+
in the order that they appear in user programs, like the Python program at the beginning of this article. We can imagine that in `ops`, we have some forward operators, followed by some gradient operators, and then some optimization operators.
Copy file name to clipboardExpand all lines: doc/design/parameters_in_cpp.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,19 +1,19 @@
1
1
# Design Doc: The C++ Class `Parameters`
2
2
3
-
`Parameters` is a concept we designed in Paddle V2 API. `Parameters` is a container of parameters, and make Paddle can shared parameter between topologies. We described usages of `Parameter` in [api.md](./api.md).
3
+
`Parameters` is a concept we designed in PaddlePaddle V2 API. `Parameters` is a container of parameters, which makes PaddlePaddle capable of sharing parameter between topologies. We described usages of `Parameter` in [api.md](./api.md).
4
4
5
-
We used Python to implement Parameters when designing V2 API before. There are several defects for current implementation:
5
+
We used Python to implement Parameters when designing V2 API before. There are several defects for the current implementation:
6
6
* We just use `memcpy` to share Parameters between topologies, but this is very inefficient.
7
-
* We did not implement share Parameters while training. We just trigger `memcpy` when start training.
7
+
* We did not support sharing Parameters while training. We just trigger `memcpy` when start training.
8
8
9
-
It is necessary that we implement Parameters in CPP side. However, it could be a code refactoring for Paddle, because Paddle was designed for training only one topology before, i.e., each GradientMachine contains its Parameter as a data member. In current Paddle implementation, there are three concepts associated with `Parameters`:
9
+
It is necessary that we implement Parameters in CPP side. However, it could result a code refactoring for PaddlePaddle, because PaddlePaddle was designed for training only one topology before, i.e., each GradientMachine contains its Parameter as a data member. In current PaddlePaddle implementation, there are three concepts associated with `Parameters`:
10
10
11
11
1.`paddle::Parameter`. A `Parameters` is a container for `paddle::Parameter`.
12
12
It is evident that we should use `paddle::Parameter` when developing `Parameters`.
13
13
However, the `Parameter` class contains many functions and does not have a clear interface.
14
14
It contains `create/store Parameter`, `serialize/deserialize`, `optimize(i.e SGD)`, `randomize/zero`.
15
15
When we developing `Parameters`, we only use `create/store Parameter` functionality.
16
-
We should extract functionalities of Parameter into many classes to clean Paddle CPP implementation.
16
+
We should extract functionalities of Parameter into many classes to clean PaddlePaddle CPP implementation.
17
17
18
18
2.`paddle::GradientMachine` and its sub-classes, e.g., `paddle::MultiGradientMachine`, `paddle::NeuralNetwork`.
19
19
We should pass `Parameters` to `paddle::GradientMachine` when `forward/backward` to avoid `memcpy` between topologies.
@@ -24,7 +24,7 @@ Also, we should handle multi-GPU/CPU training, because `forward` and `backward`
24
24
So `Parameters` should be used by `paddle::ParameterUpdater`, and `paddle::ParameterUpdater` should optimize `Parameters` (by SGD).
25
25
26
26
27
-
The step by step approach for implementation Parameters in Paddle C++ core is listed below. Each step should be a PR and could be merged into Paddle one by one.
27
+
The step by step approach for implementation Parameters in PaddlePaddle C++ core is listed below. Each step should be a PR and could be merged into PaddlePaddle one by one.
28
28
29
29
1. Clean `paddle::Parameter` interface. Extract the functionalities of `paddle::Parameter` to prepare for the implementation of Parameters.
0 commit comments