【AMP OP&Test】instance_norm fp16 and bf16 support. #52241
Conversation
|
你的PR提交成功，感谢你对开源项目的贡献! |
qizhaoaoe
force-pushed
the
instance_norm_amp
branch
from
4027894 to
134fbcc
Compare
| const T *x, | ||
| const BatchNormParamType<T> *variance, | ||
| const BatchNormParamType<AccT> *variance, |
There was a problem hiding this comment.
可以看下BatchNormParamType的定义，这里不需要再使用AccT，直接用T就可以,BatchNormParamType（T==fp16orbf16）就是float了
| self.bias = np.random.random([self.shape[1]]).astype(np.float32) | ||
|
|
||
| def set_err_thre(self): | ||
| self.atol = 1e-3 |
There was a problem hiding this comment.
试了1e-4不通过，就还是保持1e-3吧？
| self.max_relative_error = 1e-2 | ||
| self.inputs = { | ||
| 'X': convert_float_to_uint16(self.value), | ||
| 'Scale': convert_float_to_uint16(self.scale), |
There was a problem hiding this comment.
bf16这里的scale和bias应该也不需要convert吧
| return x_norm, mean.reshape(N * C), std.reshape(N * C) | ||
|
|
||
| def test_check_output(self): | ||
| place = core.CUDAPlace(0) |
There was a problem hiding this comment.
这里需要判断下是否编译的是GPU版本，或者直接调用check_output，然后下面bf16的重写一下
|
|
||
| def set_err_thre(self): | ||
| self.atol = 0.03125 | ||
| self.max_relative_error = 8e-3 |
| self.init_dtype() | ||
| self.init_shape() | ||
| self.init_value() | ||
| self.atol = 1e-2 |
| @@ -22,6 +22,11 @@ | |||
| #include "paddle/phi/kernels/funcs/norm_utils.h" | |||
| #include "paddle/phi/kernels/gpu/instance_norm_utils.h" | |||
|
|
|||
| #include "paddle/phi/common/bfloat16.h" | |||
| #include "paddle/phi/common/data_type.h" | |||
There was a problem hiding this comment.
删除无用头文件，data_type.h应该包含了bfloat16和float16，device_context.h也需要确认下是否需要
There was a problem hiding this comment.
删除了引用 data_type.h and device_context.h.
| @@ -22,8 +22,13 @@ | |||
| #include "paddle/phi/kernels/funcs/norm_utils.h" | |||
| #include "paddle/phi/kernels/gpu/instance_norm_utils.h" | |||
|
|
|||
| #include "paddle/phi/common/bfloat16.h" | |||
|
|
||
| def init_value(self): | ||
| np.random.seed(0) | ||
| self.value = np.random.random(self.shape).astype(self.dtype) |
There was a problem hiding this comment.
这里对于bf16应该是初始化为fp32，然后convert
| self.python_api = instance_norm_warpper | ||
| self.eps = 1e-5 | ||
| self.data_format = "NCHW" | ||
| self.init_dtype() |
There was a problem hiding this comment.
bf16的单测并没有对init_dtype重写，所以当前还是dtype还是float32
PR types
Others
PR changes
OPs
Describe