Modify reduce ops #8085

zhongshsh · 2022-04-24T05:17:09Z

reduce_ops.py 里的max、min、sum等操作改成Functor直接导出。同时解决 Oneflow-Inc/oneflow-documentation#480

prod 原代码对齐参数 *, dtype=None，sum 和 mean 没有对齐这个参数，但是 oneflow 并没有对齐 PyTorch * 后的参数（如典型的 out=None），对齐意义也不大，因此没有补全。

BBuf · 2022-04-25T06:45:51Z

oneflow/core/functional/functional_api.yaml

-  signature: "Tensor (Tensor x, Int32List axis, Bool keepdims=False) => ReduceAll"
+  signature: [
+    "Tensor (Tensor x, Int32List[1] dim, Bool keepdim=False) => ReduceAll",
+    "Tensor (Tensor x) => ReduceAllAll"


这里这个后缀有点鬼畜，要不重新想个后缀名吧例如ReducexxxFlatten？

已改成 ReducexxxWhole

BBuf · 2022-04-25T06:46:01Z

oneflow/core/functional/impl/math_functor.cpp

+        one::OpBuilder("reduce_sum").Input("input_tensor").Output("output_tensor").Build());
+  }
+  Maybe<Tensor> operator()(const std::shared_ptr<one::Tensor>& x) const {
+    // const DataType dtype = x->dtype()->data_type();


Suggested change

// const DataType dtype = x->dtype()->data_type();

oneflow/core/functional/impl/math_functor.cpp

oneflow/core/functional/functional_api.yaml

oneflow/core/functional/impl/math_functor.cpp

oneflow/core/functional/functional_api.yaml

BBuf · 2022-04-26T01:34:39Z

oneflow/core/functional/impl/math_functor.cpp

    MutableAttrMap attrs;
-    if (axis.empty()) {
-      std::vector<int32_t> reduce_axis(x->shape()->NumAxes());
+    const int32_t naxis = x->shape()->NumAxes();


just a suggestion!

Suggested change

const int32_t naxis = x->shape()->NumAxes();

const int32_t ndim = x->ndim();

👍 赞同

由于算子属性名就是 axis，所以还是保持原来的写法更一致些。

BBuf · 2022-04-26T01:37:00Z

oneflow/core/functional/impl/math_functor.cpp

+
+      std::vector<int32_t> reduce_axis(axis.size());
+      for (int i = 0; i < axis.size(); i++) {
+        CHECK_GE_OR_RETURN(naxis, axis[i])


规范exception的写法，并添加对应测试：参考#8080

oneflow/core/functional/impl/math_functor.cpp

BBuf · 2022-04-26T01:38:48Z

oneflow/core/functional/impl/math_functor.cpp

      std::iota(reduce_axis.begin(), reduce_axis.end(), 0);
      JUST(attrs.SetAttr<std::vector<int32_t>>("axis", reduce_axis));
    } else {
-      JUST(attrs.SetAttr<std::vector<int32_t>>("axis", axis));
+      CHECK_GE_OR_RETURN(naxis, axis.size())


同上的exception的意见

BBuf · 2022-04-27T01:56:46Z

python/oneflow/framework/tensor.py

@@ -703,11 +703,11 @@ def _unbind(self, dim=0):
    return flow._C.unbind(self, dim)


-def _all(self, dim=None, keepdim=False):
+def _all(self, dim=[], keepdim=False):


这里感觉不用改？

以如下接口为例：

signature: [ "Tensor (Tensor x, Int32List[1] dim, Bool keepdim=False) => ReduceAll", "Tensor (Tensor x) => ReduceAllWhole" ]

传None进不了Int32List，而另一个接口不传入dim参数，所以用None会报错，应该传入空数组。

BBuf · 2022-04-27T01:57:45Z

oneflow/core/functional/impl/math_functor.cpp

@@ -38,6 +38,22 @@ namespace oneflow {
 namespace one {
 namespace functional {

+namespace {
+std::string exception_check(int32_t base, int32_t value, bool check_ge = true,


这个为什么要加呢

还没改好

github-actions · 2022-05-02T17:30:08Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/8085/

github-actions · 2022-05-02T17:32:34Z

CI failed when running job: cpu-misc. PR label automerge has been removed

github-actions · 2022-05-02T17:34:09Z

Speed stats:

github-actions · 2022-05-03T15:16:56Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/8085/

github-actions · 2022-05-03T15:26:02Z

Speed stats:

GPU Name: NVIDIA GeForce GTX 1080 

❌ OneFlow resnet50 time: 129.3ms (= 12925.9ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 142.8ms (= 14279.7ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.10 (= 142.8ms / 129.3ms)

OneFlow resnet50 time: 83.8ms (= 8377.9ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 83.6ms (= 8363.2ms / 100, input_shape=[8, 3, 224, 224])
❌ Relative speed: 1.00 (= 83.6ms / 83.8ms)

OneFlow resnet50 time: 54.3ms (= 10858.2ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 55.1ms (= 11025.3ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.02 (= 55.1ms / 54.3ms)

OneFlow resnet50 time: 42.6ms (= 8525.0ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 48.3ms (= 9662.9ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.13 (= 48.3ms / 42.6ms)

OneFlow resnet50 time: 38.1ms (= 7629.5ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 37.8ms (= 7559.0ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 0.99 (= 37.8ms / 38.1ms)

OneFlow swin dataloader time: 0.256s (= 51.126s / 200, num_workers=1)
PyTorch swin dataloader time: 0.150s (= 30.057s / 200, num_workers=1)
Relative speed: 0.588 (= 0.150s / 0.256s)

OneFlow swin dataloader time: 0.067s (= 13.370s / 200, num_workers=4)
PyTorch swin dataloader time: 0.043s (= 8.551s / 200, num_workers=4)
Relative speed: 0.640 (= 0.043s / 0.067s)

OneFlow swin dataloader time: 0.036s (= 7.218s / 200, num_workers=8)
PyTorch swin dataloader time: 0.022s (= 4.395s / 200, num_workers=8)
Relative speed: 0.609 (= 0.022s / 0.036s)

❌ OneFlow resnet50 time: 145.5ms (= 14549.3ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 167.8ms (= 16776.4ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.15 (= 167.8ms / 145.5ms)

OneFlow resnet50 time: 97.5ms (= 9747.4ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 122.9ms (= 12286.2ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.26 (= 122.9ms / 97.5ms)

OneFlow resnet50 time: 75.3ms (= 15052.3ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 87.9ms (= 17577.8ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
❌ Relative speed: 1.17 (= 87.9ms / 75.3ms)

OneFlow resnet50 time: 64.8ms (= 12969.6ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 85.4ms (= 17075.2ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.32 (= 85.4ms / 64.8ms)

OneFlow resnet50 time: 55.3ms (= 11065.8ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 75.4ms (= 15077.8ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.36 (= 75.4ms / 55.3ms)

github-actions · 2022-05-03T15:41:15Z

CI failed when running job: cuda-speed-test. PR label automerge has been removed

…to modify_reduce_ops

github-actions · 2022-05-04T12:38:11Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/8085/

github-actions · 2022-05-04T13:02:38Z

CI failed when running job: cuda-benchmark. PR label automerge has been removed

github-actions · 2022-05-05T05:23:52Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/8085/

github-actions · 2022-05-05T05:30:31Z

Speed stats:

GPU Name: NVIDIA GeForce GTX 1080 

❌ OneFlow resnet50 time: 129.4ms (= 12943.0ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 140.7ms (= 14074.9ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.09 (= 140.7ms / 129.4ms)

OneFlow resnet50 time: 83.5ms (= 8346.1ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 85.2ms (= 8517.4ms / 100, input_shape=[8, 3, 224, 224])
❌ Relative speed: 1.02 (= 85.2ms / 83.5ms)

OneFlow resnet50 time: 51.8ms (= 10367.6ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 54.0ms (= 10790.8ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.04 (= 54.0ms / 51.8ms)

OneFlow resnet50 time: 41.6ms (= 8313.7ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 41.6ms (= 8316.8ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.00 (= 41.6ms / 41.6ms)

OneFlow resnet50 time: 37.0ms (= 7405.1ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 37.8ms (= 7553.1ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.02 (= 37.8ms / 37.0ms)

OneFlow swin dataloader time: 0.253s (= 50.523s / 200, num_workers=1)
PyTorch swin dataloader time: 0.151s (= 30.230s / 200, num_workers=1)
Relative speed: 0.598 (= 0.151s / 0.253s)

OneFlow swin dataloader time: 0.068s (= 13.692s / 200, num_workers=4)
PyTorch swin dataloader time: 0.044s (= 8.825s / 200, num_workers=4)
Relative speed: 0.644 (= 0.044s / 0.068s)

OneFlow swin dataloader time: 0.038s (= 7.522s / 200, num_workers=8)
PyTorch swin dataloader time: 0.022s (= 4.392s / 200, num_workers=8)
Relative speed: 0.584 (= 0.022s / 0.038s)

❌ OneFlow resnet50 time: 146.5ms (= 14645.7ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 169.2ms (= 16924.0ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.16 (= 169.2ms / 146.5ms)

OneFlow resnet50 time: 99.4ms (= 9941.8ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 112.3ms (= 11234.3ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.13 (= 112.3ms / 99.4ms)

OneFlow resnet50 time: 77.8ms (= 15557.6ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 88.3ms (= 17667.2ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
❌ Relative speed: 1.14 (= 88.3ms / 77.8ms)

OneFlow resnet50 time: 65.3ms (= 13064.5ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 77.1ms (= 15417.2ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.18 (= 77.1ms / 65.3ms)

OneFlow resnet50 time: 55.8ms (= 11166.6ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 72.0ms (= 14406.0ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.29 (= 72.0ms / 55.8ms)

update

2226f1e

zhongshsh requested review from BBuf, daquexian and jackalcooper as code owners April 24, 2022 05:17

update

9b561c5

zhongshsh requested a review from hjchen2 as a code owner April 24, 2022 13:23

zhongshsh and others added 5 commits April 25, 2022 09:32

Merge branch 'master' into modify_reduce_ops

8521c55

update

42ef156

update

68f7886

update

0dbb606

update

3adebc9

zhongshsh requested a review from doombeaker as a code owner April 25, 2022 02:29

zhongshsh added 6 commits April 25, 2022 13:48

update

bc32675

update

4f8563f

update

b9b8643

update

fc24cc2

update

3cf3313

update

3312aa5

zhongshsh added enhancement automerge eager labels Apr 25, 2022

BBuf reviewed Apr 25, 2022

View reviewed changes

zhongshsh added 2 commits April 25, 2022 15:45

update

76e3a09

update

65cb2a5

zhongshsh added the op label Apr 25, 2022

EsdeathYZH reviewed Apr 25, 2022

View reviewed changes

oneflow/core/functional/functional_api.yaml Show resolved Hide resolved

oneflow/core/functional/impl/math_functor.cpp Outdated Show resolved Hide resolved

BBuf reviewed Apr 26, 2022

View reviewed changes

update

4c2b184

BBuf reviewed Apr 27, 2022

View reviewed changes

update

0dfdb67

github-actions bot removed the automerge label May 2, 2022

zhongshsh added the automerge label May 3, 2022

zhongshsh requested review from oneflow-ci-bot and removed request for oneflow-ci-bot May 3, 2022 11:48

zhongshsh and others added 2 commits May 3, 2022 19:48

Merge branch 'master' into modify_reduce_ops

a509865

Merge branch 'master' into modify_reduce_ops

71a2b39

github-actions bot removed the automerge label May 3, 2022

zhongshsh removed the request for review from oneflow-ci-bot May 4, 2022 07:07

zhongshsh added the automerge label May 4, 2022

mergify bot and others added 4 commits May 4, 2022 07:08

Merge branch 'master' into modify_reduce_ops

3c68ba6

fix CheckAxis

b1c932b

Merge branch 'modify_reduce_ops' of github.com:Oneflow-Inc/oneflow in…

b78f4c4

…to modify_reduce_ops

Merge branch 'master' into modify_reduce_ops

2b5fdb2

zhongshsh requested a review from oneflow-ci-bot May 4, 2022 11:48

github-actions bot removed the automerge label May 4, 2022

zhongshsh added the automerge label May 5, 2022

Merge branch 'master' into modify_reduce_ops

f8f7ca0

zhongshsh merged commit f86de84 into master May 5, 2022

zhongshsh deleted the modify_reduce_ops branch May 5, 2022 06:18

zhongshsh mentioned this pull request May 6, 2022

OneFlow 算子对齐 PyTorch 完备计划推进表 #4936

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Modify reduce ops #8085

Modify reduce ops #8085

zhongshsh commented Apr 24, 2022 •

edited

Loading

BBuf Apr 25, 2022

zhongshsh Apr 25, 2022 •

edited

Loading

BBuf Apr 25, 2022

zhongshsh Apr 25, 2022

BBuf Apr 26, 2022

zhongshsh Apr 26, 2022

zhongshsh Apr 27, 2022

BBuf Apr 26, 2022

zhongshsh Apr 27, 2022

BBuf Apr 26, 2022

zhongshsh Apr 27, 2022

BBuf Apr 27, 2022

zhongshsh Apr 27, 2022

BBuf Apr 27, 2022

zhongshsh Apr 27, 2022

github-actions bot commented May 2, 2022

github-actions bot commented May 2, 2022

github-actions bot commented May 2, 2022

github-actions bot commented May 3, 2022

github-actions bot commented May 3, 2022

github-actions bot commented May 3, 2022

github-actions bot commented May 4, 2022

github-actions bot commented May 4, 2022

github-actions bot commented May 5, 2022

github-actions bot commented May 5, 2022

	const int32_t naxis = x->shape()->NumAxes();
	const int32_t ndim = x->ndim();

Modify reduce ops #8085

Modify reduce ops #8085

Conversation

zhongshsh commented Apr 24, 2022 • edited Loading

Choose a reason for hiding this comment

zhongshsh Apr 25, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented May 2, 2022

github-actions bot commented May 2, 2022

github-actions bot commented May 2, 2022

github-actions bot commented May 3, 2022

github-actions bot commented May 3, 2022

github-actions bot commented May 3, 2022

github-actions bot commented May 4, 2022

github-actions bot commented May 4, 2022

github-actions bot commented May 5, 2022

github-actions bot commented May 5, 2022

zhongshsh commented Apr 24, 2022 •

edited

Loading

zhongshsh Apr 25, 2022 •

edited

Loading