Dev where scalar #5797

simonJJJ · 2021-08-08T12:23:32Z

No description provided.

MARD1NO · 2021-08-11T02:30:43Z

oneflow/user/kernels/where_kernel.cpp

-                                                  out->mut_dptr<T>());
+    if (!(x->shape() == y->shape() && y->shape() == cond->shape())) {
+      size_t num_axes = out->shape().NumAxes();
+      int64_t elem_cnt = out->shape().elem_cnt();


torch那边的where支持0-d tensor吗，比如（torch.randn(2, 0, 2)) 这样他其实是空的（因为2x0x2 = 0）

如果支持的话，需要对elem_cnt == 0情况特判下

好！我看看

感觉你这里是不是还没对 0-d tensor的情况做考虑

…dev_where_scalar

MARD1NO · 2021-08-17T02:06:40Z

oneflow/core/autograd/gradient_funcs/where.cpp


 namespace oneflow {
 namespace one {

+namespace {


能不能简单解释下为什么这里这么处理

我感觉你是想在reducesumlike的基础前面，加入一个broadcast操作

如果是这样，那我感觉还不如放到functional那里，并且名字改为BroadcastReduceSumLike，放在gradient_funcs是不太合适的个人认为，并且有可能以后别的地方会用到这个函数

可以，好像有个其他的functor也用到了

oneflow/core/autograd/gradient_funcs/where.cpp

+    const std::shared_ptr<oneflow::one::Tensor>& y = ctx->SavedTensors().at(1);
+
+    std::shared_ptr<oneflow::one::Tensor> zero_out = JUST(functional::ZerosLike(y));
+    in_grads->resize(2);


oneflow/core/functional/impl/array_functor.cpp

+    } else {
+      UNIMPLEMENTED_THEN_RETURN() << "The scalar in Where shoule be float or int.";
+    }
+    return OpInterpUtil::Dispatch<Tensor>(*op_, {condition, y}, attrs);


MARD1NO · 2021-08-17T02:14:18Z

oneflow/user/kernels/where_kernel.cpp

-                                                  out->mut_dptr<T>());
+    if (!(x->shape() == y->shape() && y->shape() == cond->shape())) {
+      size_t num_axes = out->shape().NumAxes();
+      int64_t elem_cnt = out->shape().elem_cnt();


感觉你这里是不是还没对 0-d tensor的情况做考虑

oneflow/user/kernels/where_kernel.cpp

+};
+
+template<DeviceType device_type, typename T, typename CondT>
+class WhereScalarXKernel final : public user_op::OpKernel {


MARD1NO · 2021-08-17T02:15:14Z

oneflow/user/kernels/where_kernel.cpp

+      x_scalar_operand = static_cast<T>(ctx->Attr<double>("x_float_operand"));
+      y_scalar_operand = static_cast<T>(ctx->Attr<double>("y_float_operand"));
+    } else {
+      UNIMPLEMENTED();


加点输出提示

oneflow/user/kernels/where_kernel_util.cpp

+                           const T x_scalar, const T* rhs, T* out) {
+    FOR_RANGE(int64_t, i, 0, elem_cnt) { out[i] = static_cast<bool>(cond[i]) ? x_scalar : rhs[i]; }
+  }
+  static void WhereYScalar(DeviceCtx* ctx, const int64_t elem_cnt, const CondT* cond, const T* lhs,


oneflow/user/kernels/where_kernel_util.cu

@@ -27,6 +27,35 @@ struct WhereFunctor {
  }
 };

+template<typename T, typename CondT>
+struct WhereScalarXFunctor {


MARD1NO · 2021-08-17T02:39:59Z

这个PR的修改，可以让masked_fill变得很简单，这部分也请修改掉构造tensor的部分
https://github.com/Oneflow-Inc/oneflow/blob/master/python/oneflow/nn/modules/masked_fill.py#L59-L62

oneflow/core/functional/impl/array_functor.cpp

…dev_where_scalar

github-actions · 2021-08-17T11:31:13Z

Speed stats:

GPU Name: GeForce GTX 1080 

PyTorch resnet50 time: 139.4ms (= 6969.1ms / 50, input_shape=[16, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 127.8ms (= 6389.2ms / 50, input_shape=[16, 3, 224, 224], backward is enabled)
Relative speed: 1.09 (= 139.4ms / 127.8ms)

PyTorch resnet50 time: 84.9ms (= 4244.6ms / 50, input_shape=[8, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 74.2ms (= 3712.0ms / 50, input_shape=[8, 3, 224, 224], backward is enabled)
Relative speed: 1.14 (= 84.9ms / 74.2ms)

PyTorch resnet50 time: 60.1ms (= 3003.9ms / 50, input_shape=[4, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 47.6ms (= 2380.8ms / 50, input_shape=[4, 3, 224, 224], backward is enabled)
Relative speed: 1.26 (= 60.1ms / 47.6ms)

PyTorch resnet50 time: 49.8ms (= 2488.6ms / 50, input_shape=[2, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 42.9ms (= 2142.8ms / 50, input_shape=[2, 3, 224, 224], backward is enabled)
Relative speed: 1.16 (= 49.8ms / 42.9ms)

PyTorch resnet50 time: 40.0ms (= 1999.7ms / 50, input_shape=[1, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 37.7ms (= 1886.0ms / 50, input_shape=[1, 3, 224, 224], backward is enabled)
Relative speed: 1.06 (= 40.0ms / 37.7ms)

simonJJJ added 5 commits August 8, 2021 20:19

where broadcast forward

01450e2

delete py broadcast code

400fcd8

delete op_expr

36c4deb

Merge branch 'master' into dev_where_scalar

ffcde99

add where backward broadcast gradient

2a1cda0

simonJJJ added eager enhancement op labels Aug 9, 2021

scalar x op

1803a7b

MARD1NO reviewed Aug 11, 2021

View reviewed changes

simonJJJ added 7 commits August 16, 2021 10:52

Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow into …

2ea1ecc

…dev_where_scalar

add where scalar_x, scalar_y, scalar_xy ops & kernels

d1e37c7

add where scalar_x, scalar_y, scalar_xy functors

4ffcdf4

fix bug

f4f2f87

add where backward

669197c

add pybind api

43014b9

add testcase

a0e9c83

MARD1NO reviewed Aug 17, 2021

View reviewed changes

oneflow/core/functional/impl/array_functor.cpp Show resolved Hide resolved

simonJJJ added 3 commits August 17, 2021 15:45

fix

4cf74e4

Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow into …

891daba

…dev_where_scalar

fix

414ec12

MARD1NO approved these changes Aug 17, 2021

View reviewed changes

simonJJJ added the automerge label Aug 17, 2021

simonJJJ requested a review from oneflow-ci-bot August 17, 2021 08:14

Merge branch 'master' into dev_where_scalar

edf7dc8

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot August 17, 2021 08:38

Merge branch 'master' into dev_where_scalar

03f963d

oneflow-ci-bot self-requested a review August 17, 2021 10:12

oneflow-ci-bot merged commit 572951d into master Aug 17, 2021

oneflow-ci-bot deleted the dev_where_scalar branch August 17, 2021 11:31

kevinzhangcode mentioned this pull request Aug 20, 2021

自动测试重构剩下的任务列表。 #5905

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dev where scalar #5797

Dev where scalar #5797

simonJJJ commented Aug 8, 2021

MARD1NO Aug 11, 2021

simonJJJ Aug 11, 2021

MARD1NO Aug 17, 2021

MARD1NO Aug 17, 2021

simonJJJ Aug 17, 2021

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

MARD1NO Aug 17, 2021

This comment was marked as resolved.

MARD1NO Aug 17, 2021

This comment was marked as resolved.

This comment was marked as resolved.

MARD1NO commented Aug 17, 2021

github-actions bot commented Aug 17, 2021

Dev where scalar #5797

Dev where scalar #5797

Conversation

simonJJJ commented Aug 8, 2021

MARD1NO Aug 11, 2021

Choose a reason for hiding this comment

simonJJJ Aug 11, 2021

Choose a reason for hiding this comment

MARD1NO Aug 17, 2021

Choose a reason for hiding this comment

MARD1NO Aug 17, 2021

Choose a reason for hiding this comment

simonJJJ Aug 17, 2021

Choose a reason for hiding this comment

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

MARD1NO Aug 17, 2021

Choose a reason for hiding this comment

This comment was marked as resolved.

MARD1NO Aug 17, 2021

Choose a reason for hiding this comment

This comment was marked as resolved.

This comment was marked as resolved.

MARD1NO commented Aug 17, 2021

github-actions bot commented Aug 17, 2021