support consistent_tensor.to(copy=True) #6122

lixinqi · 2021-09-01T03:05:08Z

No description provided.

CLAassistant · 2021-09-01T03:05:10Z

All committers have signed the CLA.

chengtbf · 2021-09-01T04:41:58Z

python/oneflow/nn/modules/to.py

@@ -185,7 +185,7 @@ def to_op(input, *args, **kwargs):
            )

        if copy is True:
-            raise TypeError("A consistent tensor do not support to(copy=True)")
+            return input.detach().clone()


这里直接这样改不对？还是需要传到 _consistent_tensor_to 里，因为上面的 device dtype 可能有值

这里应该把 copy 传到 _consistent_tensor_to 里面去，然后在 55 行做下改动

if device_type == input.placement.device_type and dtype == input.dtype: if copy: return input.detach().clone() else: return input

是不是应该直接把copy的逻辑放到_consistent_tensor_to里面去，和_tensor_to接口对齐。

目前这个to的逻辑在python里面有点重，后续找人挪到C++吧。

看上去其实还不能有detach。因为原本有可能requires_grad就是为True。

github-actions · 2021-09-01T12:08:37Z

CI failed, removing label automerge

strint · 2021-09-01T14:46:41Z

python/oneflow/nn/modules/to.py

@@ -53,7 +53,7 @@ def _consistent_tensor_to(input, device_type, dtype):
    assert isinstance(dtype, flow.dtype)

    if device_type == input.placement.device_type and dtype == input.dtype:
-        return input
+        return input if not copy else input.clone()


revert这行，失败的那个test_graph_asymmetric_io.py测试就可以过

…alue()

lixinqi · 2021-09-01T16:45:54Z

oneflow/core/framework/tensor_impl.cpp

+    cur_rank_phy_tensor = std::make_shared<MirroredTensor>(cur_rank_phy_tensor_impl);
+  } else {
+    const auto& dtype_symbol = JUST(DType::Get(dtype));
+    const auto& empty = JUST(functional::Empty(*cur_rank_phy_shape, dtype_symbol, device));


虽然cur_shape_phy_shape为全0 shape。但如果不执行Empty op，将导致eager_blob_object的blob对象为空。

…onsistent_to

lixinqi · 2021-09-02T02:04:40Z

oneflow/core/vm/vm_util.cpp

  BlockingCounter bc(1);
  JUST(PhysicalRun([&bc](InstructionsBuilder* builder) -> Maybe<void> {
-    JUST(builder->ComputeGlobalFrontSeqBarrier());


NNGraph同步vm的时候不应该操心其他rank的vm。

lixinqi · 2021-09-02T02:05:21Z

oneflow/core/vm/vm_util.cpp

@@ -40,9 +40,11 @@ Maybe<void> Run(vm::InstructionMsgList* instr_msg_list) {
  return Maybe<void>::Ok();
 }

-Maybe<void> SingleClientSync() {
+Maybe<void> ClusterSync() {


重名的原因是这里兼职了旧版的SingleClientSync和MultiClientSync

github-actions · 2021-09-02T05:14:11Z

Speed stats:

GPU Name: GeForce GTX 1080 

OneFlow resnet50 time: 128.2ms (= 6409.3ms / 50, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 140.7ms (= 7034.5ms / 50, input_shape=[16, 3, 224, 224])
Relative speed: 1.10 (= 140.7ms / 128.2ms)

OneFlow resnet50 time: 74.3ms (= 3714.7ms / 50, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 84.3ms (= 4216.7ms / 50, input_shape=[8, 3, 224, 224])
Relative speed: 1.14 (= 84.3ms / 74.3ms)

OneFlow resnet50 time: 47.6ms (= 2381.1ms / 50, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 58.9ms (= 2946.2ms / 50, input_shape=[4, 3, 224, 224])
Relative speed: 1.24 (= 58.9ms / 47.6ms)

OneFlow resnet50 time: 42.8ms (= 2142.0ms / 50, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 51.2ms (= 2561.6ms / 50, input_shape=[2, 3, 224, 224])
Relative speed: 1.20 (= 51.2ms / 42.8ms)

OneFlow resnet50 time: 40.4ms (= 2018.3ms / 50, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 42.4ms (= 2121.3ms / 50, input_shape=[1, 3, 224, 224])
Relative speed: 1.05 (= 42.4ms / 40.4ms)

OneFlow resnet50 time: 141.5ms (= 7075.2ms / 50, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 157.0ms (= 7848.8ms / 50, input_shape=[16, 3, 224, 224], ddp, world size=2)
Relative speed: 1.11 (= 157.0ms / 141.5ms)

OneFlow resnet50 time: 92.9ms (= 4646.2ms / 50, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 100.8ms (= 5037.7ms / 50, input_shape=[8, 3, 224, 224], ddp, world size=2)
Relative speed: 1.08 (= 100.8ms / 92.9ms)

OneFlow resnet50 time: 68.8ms (= 3437.6ms / 50, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 74.8ms (= 3737.9ms / 50, input_shape=[4, 3, 224, 224], ddp, world size=2)
Relative speed: 1.09 (= 74.8ms / 68.8ms)

OneFlow resnet50 time: 64.4ms (= 3222.5ms / 50, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 70.6ms (= 3528.6ms / 50, input_shape=[2, 3, 224, 224], ddp, world size=2)
Relative speed: 1.10 (= 70.6ms / 64.4ms)

OneFlow resnet50 time: 63.3ms (= 3163.1ms / 50, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 72.6ms (= 3627.5ms / 50, input_shape=[1, 3, 224, 224], ddp, world size=2)
Relative speed: 1.15 (= 72.6ms / 63.3ms)

lixinqi added enhancement automerge eager labels Sep 1, 2021

lixinqi requested review from jackalcooper, chengtbf and oneflow-ci-bot September 1, 2021 03:05

support consistent_tensor.to(copy=True)

c361d60

lixinqi force-pushed the consistent_to branch from 0cb7244 to c361d60 Compare September 1, 2021 03:09

chengtbf reviewed Sep 1, 2021

View reviewed changes

chengtbf requested review from leaves-zwx and chengtbf September 1, 2021 04:42

oneflow-ci-bot and others added 2 commits September 1, 2021 04:43

auto format by CI

2d8584b

refactor _consistent_tensor_to

065a403

hjchen2 approved these changes Sep 1, 2021

View reviewed changes

leaves-zwx approved these changes Sep 1, 2021

View reviewed changes

Merge branch 'master' into consistent_to

df2474f

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot September 1, 2021 06:43

Merge branch 'master' into consistent_to

d952a20

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot September 1, 2021 07:13

Merge branch 'master' into consistent_to

6a2ba93

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot September 1, 2021 08:07

chengtbf approved these changes Sep 1, 2021

View reviewed changes

Merge branch 'master' into consistent_to

c7d2737

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot September 1, 2021 09:44

Merge branch 'master' into consistent_to

415696a

oneflow-ci-bot self-requested a review September 1, 2021 11:01

github-actions bot removed the automerge label Sep 1, 2021

oneflow-ci-bot removed their request for review September 1, 2021 12:11

strint reviewed Sep 1, 2021

View reviewed changes

lixinqi and others added 2 commits September 2, 2021 00:42

let cur_rank_phy_tensor be function::Empty(...) if !parallel_id.has_v…

c7f78e9

…alue()

Merge branch 'master' into consistent_to

fbf10c6

lixinqi requested a review from oneflow-ci-bot September 1, 2021 16:44

lixinqi added the automerge label Sep 1, 2021

auto format by CI

46d0307

lixinqi commented Sep 1, 2021

View reviewed changes

Merge branch 'master' into consistent_to

690f2b6

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot September 1, 2021 17:02

lixinqi requested a review from oneflow-ci-bot September 1, 2021 23:32

oneflow-ci-bot removed their request for review September 2, 2021 01:04

lixinqi added 2 commits September 2, 2021 09:55

no OF_ENV_BARRIER when sync vm

acd7965

Merge branch 'consistent_to' of github.com:Oneflow-Inc/oneflow into c…

bd9e83e

…onsistent_to

lixinqi commented Sep 2, 2021

View reviewed changes

Merge branch 'master' into consistent_to

88cb196

oneflow-ci-bot self-requested a review September 2, 2021 02:16

auto format by CI

bd38a9a

chengtbf approved these changes Sep 2, 2021

View reviewed changes

Merge branch 'master' into consistent_to

abc51d1

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot September 2, 2021 04:16

oneflow-ci-bot merged commit 5987caf into master Sep 2, 2021

oneflow-ci-bot deleted the consistent_to branch September 2, 2021 05:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support consistent_tensor.to(copy=True) #6122

support consistent_tensor.to(copy=True) #6122

lixinqi commented Sep 1, 2021

CLAassistant commented Sep 1, 2021 •

edited

Loading

chengtbf Sep 1, 2021

leaves-zwx Sep 1, 2021

hjchen2 Sep 1, 2021

lixinqi Sep 1, 2021

github-actions bot commented Sep 1, 2021

strint Sep 1, 2021 •

edited

Loading

lixinqi Sep 1, 2021

lixinqi Sep 2, 2021

lixinqi Sep 2, 2021

github-actions bot commented Sep 2, 2021

support consistent_tensor.to(copy=True) #6122

support consistent_tensor.to(copy=True) #6122

Conversation

lixinqi commented Sep 1, 2021

CLAassistant commented Sep 1, 2021 • edited Loading

chengtbf Sep 1, 2021

Choose a reason for hiding this comment

leaves-zwx Sep 1, 2021

Choose a reason for hiding this comment

hjchen2 Sep 1, 2021

Choose a reason for hiding this comment

lixinqi Sep 1, 2021

Choose a reason for hiding this comment

github-actions bot commented Sep 1, 2021

strint Sep 1, 2021 • edited Loading

Choose a reason for hiding this comment

lixinqi Sep 1, 2021

Choose a reason for hiding this comment

lixinqi Sep 2, 2021

Choose a reason for hiding this comment

lixinqi Sep 2, 2021

Choose a reason for hiding this comment

github-actions bot commented Sep 2, 2021

CLAassistant commented Sep 1, 2021 •

edited

Loading

strint Sep 1, 2021 •

edited

Loading