source op support s and fixed generator bug #7571

grybd · 2022-02-23T03:32:19Z

这个PR的目的

randperm op支持 S0、添加单测
处理 random op 支持s 过程中，各个rank间local tensor 总是相等的bug 相关背景记录在randperm and arange op support S0 #7434 (comment)

random op 支持 global tensor 一致性

在处理 randint op 和 rand op 支持B/S 保持global tensor 的一致性所采取的方案是利用 GetOpKernelRandomSeed(ctx)这个工具函数进行设计，当op 支持 S时不同rank 间调用GetOpKernelRandomSeed(ctx) 返回一个不同的seed，再通过generator->set_current_seed(ctx->Attr<int64_t>("seed") + GetOpKernelRandomSeed(ctx)) 就可以为每个rank 设计不同的seed,这样能保证uniform 类的kernel 经过S 生成同分布不同数值的local tensor ，当op支持B时每个rank 上 kernel 调用GetOpKernelRandomSeed(ctx) 时会生成相同的seed ,再通过generator->set_current_seed(ctx->Attr<int64_t>("seed") + GetOpKernelRandomSeed(ctx)) 就保证了每个rank 都拿到了相同的seed,这样就可以保持global tensor 的一致性
在处理 randperm op 和 arange op 支持 B/S 时保持 global tensor 的一致性，目前打算采用的处理方案是让多个rank 公用seed 然后在先在每个rank上生成完整的tensor再根据 infer physic shape信息利GetTensorSliceView4ParallelId(parallel_hierarchy, nd_sbp, logical_shape, parallel_id) 这个工具函数，获得本rank_id 和 physic shape所对应的tensor 上的索引信息，再把对应的位置的数据拷贝到本rank 的local tensor 上

以上方案是通过与xiaoyu，yinggang开会总结出来的

fixed: https://github.com/Oneflow-Inc/OneTeam/issues/1167

oneflow/user/kernels/distributions/uniform_int_kernel.h

oneflow/user/kernels/distributions/uniform_kernel.h

oneflow/user/kernels/randperm_kernel.cpp

oneflow/user/kernels/randperm_kernel.cu

oneflow/user/ops/randperm_op.cpp

oneflow/user/kernels/distributions/uniform_int_kernel.h

python/oneflow/test/modules/test_arange.py

grybd · 2022-02-23T18:17:43Z

testcase 还没跑通, 我明天再看看~，麻烦有空也帮忙review一下哈 @strint @wyg1997

grybd · 2022-02-24T07:31:03Z

跑./test_consistent_arange.py会报错，报错信息：
File "/home/fengdaochao/fdz/oneflow/oneflow/core/job/job_build_and_infer_ctx.cpp", line 630, in AddAndInferOp
CheckOpBlobSplitability(op, parallel_desc.parallel_num())
File "/home/fengdaochao/fdz/oneflow/oneflow/core/job/job_build_and_infer_ctx.cpp", line 335, in CheckOpBlobSplitability
Check failed: (current_shape.At(axis) % parallel_hierarchy->At(i)) == (0) (1 vs 0) op_name: arange-0 blob_name: out_0 cannot split blob by parallel_hierarchy: 2

跑./test_consistent_randperm.py会报错，报错信息：
F20220224 15:16:32.818838 986938 user_kernel.cpp:171] Check failed: parallel_desc_.hierarchy()->NumAxes() == 1 (2 vs. 1)
*** Check failure stack trace: ***
F20220224 15:16:32.819939 986955 user_kernel.cpp:171] Check failed: parallel_desc_.hierarchy()->NumAxes() == 1 (2 vs. 1)
*** Check failure stack trace: ***
@ 0x7f1cd9c17f97 google::LogMessage::Fail()
@ 0x7f1266515f97 google::LogMessage::Fail()
@ 0x7f1cd9c17ed8 google::LogMessage::SendToLog()
@ 0x7f1266515ed8 google::LogMessage::SendToLog()
@ 0x7f1cd9c177bb google::LogMessage::Flush()
@ 0x7f12665157bb google::LogMessage::Flush()
@ 0x7f1cd9c1b31c google::LogMessageFatal::~LogMessageFatal()
@ 0x7f126651931c google::LogMessageFatal::~LogMessageFatal()
@ 0x7f1cd3066957 oneflow::UserKernelInitAndCacheContext::SbpParallel4ArgNameAndIndex()
@ 0x7f125f964957 oneflow::UserKernelInitAndCacheContext::SbpParallel4ArgNameAndIndex()
@ 0x7f1cd5c83fd5 oneflow::GetOpKernelSeed()
@ 0x7f1cd5c85e81 oneflow::CpuRandPermKernel::CreateOpKernelState()
@ 0x7f1262581fd5 oneflow::GetOpKernelSeed()
@ 0x7f1262583e81 oneflow::CpuRandPermKernel::CreateOpKernelState()
@ 0x7f1cd3060655 oneflow::UserKernel::CreateOpKernelState()
@ 0x7f1cd3060ad8 oneflow::UserKernel::VirtualKernelInit()
@ 0x7f125f95e655 oneflow::UserKernel::CreateOpKernelState()
@ 0x7f1cd2fea86d oneflow::Kernel::Init()
@ 0x7f125f95ead8 oneflow::UserKernel::VirtualKernelInit()
@ 0x7f1cd2feacef oneflow::ConstructKernel()
@ 0x7f1cd30a6e95 oneflow::Actor::Init()
@ 0x7f125f8e886d oneflow::Kernel::Init()
@ 0x7f1cd30bdfa9 oneflow::NewActor()
@ 0x7f125f8e8cef oneflow::ConstructKernel()
@ 0x7f1cd44799c9 oneflow::Thread::ConstructActor()
@ 0x7f1cd4479411 oneflow::Thread::PollMsgChannel()
@ 0x7f125f9a4e95 oneflow::Actor::Init()
@ 0x7f1cd4478ae7 _ZZN7oneflow6ThreadC4ERKNS_8StreamIdEENKUlvE_clEv
@ 0x7f1cd447a4e3 ZSt13__invoke_implIvZN7oneflow6ThreadC4ERKNS0_8StreamIdEEUlvE_JEET_St14__invoke_otherOT0_DpOT1
@ 0x7f1cd447a484 ZSt8__invokeIZN7oneflow6ThreadC4ERKNS0_8StreamIdEEUlvE_JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS7_DpOS8
@ 0x7f125f9bbfa9 oneflow::NewActor()
@ 0x7f1cd447a422 _ZNSt6thread8_InvokerISt5tupleIJZN7oneflow6ThreadC4ERKNS2_8StreamIdEEUlvE_EEE9_M_invokeIJLm0EEEEvSt12_Index_tupleIJXspT_EEE
@ 0x7f1cd447a3e3 _ZNSt6thread8_InvokerISt5tupleIJZN7oneflow6ThreadC4ERKNS2_8StreamIdEEUlvE_EEEclEv
@ 0x7f1cd447a3b8 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN7oneflow6ThreadC4ERKNS3_8StreamIdEEUlvE_EEEEE6_M_runEv
@ 0x7f1c7f0d8de4 (unknown)
@ 0x7f1d2c76e609 start_thread
@ 0x7f1d2c8aa293 clone
@ 0x7f1260d779c9 oneflow::Thread::ConstructActor()
@ 0x7f1260d77411 oneflow::Thread::PollMsgChannel()
@ 0x7f1260d76ae7 _ZZN7oneflow6ThreadC4ERKNS_8StreamIdEENKUlvE_clEv
@ 0x7f1260d784e3 ZSt13__invoke_implIvZN7oneflow6ThreadC4ERKNS0_8StreamIdEEUlvE_JEET_St14__invoke_otherOT0_DpOT1
@ 0x7f1260d78484 ZSt8__invokeIZN7oneflow6ThreadC4ERKNS0_8StreamIdEEUlvE_JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS7_DpOS8
@ 0x7f1260d78422 _ZNSt6thread8_InvokerISt5tupleIJZN7oneflow6ThreadC4ERKNS2_8StreamIdEEUlvE_EEE9_M_invokeIJLm0EEEEvSt12_Index_tupleIJXspT_EEE
@ 0x7f1260d783e3 _ZNSt6thread8_InvokerISt5tupleIJZN7oneflow6ThreadC4ERKNS2_8StreamIdEEUlvE_EEEclEv
@ 0x7f1260d783b8 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN7oneflow6ThreadC4ERKNS3_8StreamIdEEUlvE_EEEEE6_M_runEv
@ 0x7f120b9d6de4 (unknown)
@ 0x7f12b906c609 start_thread
@ 0x7f12b91a8293 clone
Killing subprocess 986513
Killing subprocess 986514

补充

torch.arange 文档链接： https://pytorch.org/docs/stable/generated/torch.arange.html
torch.randperm 文档 https://pytorch.org/docs/stable/generated/torch.randperm.html?highlight=randperm

grybd · 2022-02-24T15:25:49Z

合并 fix-all_Sbp4ArgNameAndIndex_bug 分支后跑 test_consistent_randperm.py 还会报错，报错信息：
F20220224 23:03:03.118664 1706926 user_kernel.cpp:171] Check failed: parallel_desc_.hierarchy()->NumAxes() == 1 (2 vs. 1)
*** Check failure stack trace: ***
F20220224 23:03:03.120785 1706914 user_kernel.cpp:171] Check failed: parallel_desc_.hierarchy()->NumAxes() == 1 (2 vs. 1)

跑 test_consistent_arange.py 报错，报错信息
File "/home/fengdaochao/fdz/oneflow/oneflow/core/job/job_build_and_infer_ctx.cpp", line 335, in CheckOpBlobSplitability
Check failed: (current_shape.At(axis) % parallel_hierarchy->At(i)) == (0) (1 vs 0) op_name: arange-0 blob_name: out_0 cannot split blob by parallel_hierarchy: 2

@strint @wyg1997 问题应该是比较明确的么，目前还不知道怎么去改，有时间麻烦帮忙看看~

wyg1997 · 2022-02-24T15:43:23Z

合并 fix-all_Sbp4ArgNameAndIndex_bug 分支后跑 test_consistent_randperm.py 还会报错，报错信息： F20220224 23:03:03.118664 1706926 user_kernel.cpp:171] Check failed: parallel_desc_.hierarchy()->NumAxes() == 1 (2 vs. 1) *** Check failure stack trace: *** F20220224 23:03:03.120785 1706914 user_kernel.cpp:171] Check failed: parallel_desc_.hierarchy()->NumAxes() == 1 (2 vs. 1)

跑 test_consistent_arange.py 报错，报错信息 File "/home/fengdaochao/fdz/oneflow/oneflow/core/job/job_build_and_infer_ctx.cpp", line 335, in CheckOpBlobSplitability Check failed: (current_shape.At(axis) % parallel_hierarchy->At(i)) == (0) (1 vs 0) op_name: arange-0 blob_name: out_0 cannot split blob by parallel_hierarchy: 2

@strint @wyg1997 问题应该是比较明确的么，目前还不知道怎么去改，有时间麻烦帮忙看看~

第二个看起来是不能切分导致的；第一个似乎是在哪里又调用了 SbpParallel4ArgNameAndIndex 接口，定位一下应该不难解决。

…p_support_S

github-actions · 2022-04-30T02:19:44Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/7571/

github-actions · 2022-04-30T02:51:29Z

CI failed when running job: cpu-module. PR label automerge has been removed

github-actions · 2022-05-02T12:27:11Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/7571/

github-actions · 2022-05-02T12:35:09Z

Speed stats:

GPU Name: NVIDIA GeForce GTX 1080 

❌ OneFlow resnet50 time: 129.5ms (= 12948.8ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 142.8ms (= 14280.6ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.10 (= 142.8ms / 129.5ms)

OneFlow resnet50 time: 78.4ms (= 7842.6ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 84.7ms (= 8467.5ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.08 (= 84.7ms / 78.4ms)

OneFlow resnet50 time: 52.7ms (= 10537.4ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 61.8ms (= 12357.3ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.17 (= 61.8ms / 52.7ms)

OneFlow resnet50 time: 40.4ms (= 8075.3ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 44.8ms (= 8965.6ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.11 (= 44.8ms / 40.4ms)

OneFlow resnet50 time: 39.9ms (= 7973.6ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 36.0ms (= 7201.4ms / 200, input_shape=[1, 3, 224, 224])
❌ Relative speed: 0.90 (= 36.0ms / 39.9ms)

OneFlow swin dataloader time: 0.253s (= 50.674s / 200, num_workers=1)
PyTorch swin dataloader time: 0.152s (= 30.311s / 200, num_workers=1)
Relative speed: 0.598 (= 0.152s / 0.253s)

OneFlow swin dataloader time: 0.066s (= 13.210s / 200, num_workers=4)
PyTorch swin dataloader time: 0.043s (= 8.575s / 200, num_workers=4)
Relative speed: 0.649 (= 0.043s / 0.066s)

OneFlow swin dataloader time: 0.036s (= 7.299s / 200, num_workers=8)
PyTorch swin dataloader time: 0.022s (= 4.395s / 200, num_workers=8)
Relative speed: 0.602 (= 0.022s / 0.036s)

❌ OneFlow resnet50 time: 145.4ms (= 14543.1ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 169.7ms (= 16973.0ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.17 (= 169.7ms / 145.4ms)

OneFlow resnet50 time: 97.7ms (= 9765.8ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 111.1ms (= 11114.5ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.14 (= 111.1ms / 97.7ms)

OneFlow resnet50 time: 80.4ms (= 16079.3ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 88.5ms (= 17691.8ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
❌ Relative speed: 1.10 (= 88.5ms / 80.4ms)

OneFlow resnet50 time: 63.7ms (= 12741.1ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 86.7ms (= 17340.3ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.36 (= 86.7ms / 63.7ms)

OneFlow resnet50 time: 56.3ms (= 11263.3ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 68.7ms (= 13732.4ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.22 (= 68.7ms / 56.3ms)

github-actions · 2022-05-02T12:52:49Z

CI failed when running job: cuda-benchmark. PR label automerge has been removed

github-actions · 2022-05-02T13:20:36Z

CI failed when running job: cuda-module. PR label automerge has been removed

grybd · 2022-05-02T13:55:51Z

FAILED python/oneflow/test/modules/test_one_embedding_ftrl.py::TestOptimizers::test_ftrl
!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
==== 1 failed, 854 passed, 180 skipped, 996 warnings in 3232.69s (0:53:52) =====

grybd · 2022-05-02T13:56:17Z

本地跑没有问题

github-actions · 2022-05-02T14:06:53Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/7571/

github-actions · 2022-05-02T14:33:38Z

CI failed when running job: cuda-benchmark. PR label automerge has been removed

source op support s and fixed generator bug

a6e44f2

grybd requested a review from strint February 23, 2022 03:32

grybd requested review from daquexian, jackalcooper, liujuncheng, guo-ran and MARD1NO as code owners February 23, 2022 03:32

fixed random op generator bug

546ff4a

strint reviewed Feb 23, 2022

View reviewed changes

oneflow/user/ops/randperm_op.cpp Outdated Show resolved Hide resolved

grybd added 3 commits February 23, 2022 16:57

Fix SbpParallel4ArgNameAndIndex bug

e43739d

add notes

57d3955

format code

3737e97

wyg1997 reviewed Feb 23, 2022

View reviewed changes

oneflow/user/ops/randperm_op.cpp Outdated Show resolved Hide resolved

wyg1997 reviewed Feb 23, 2022

View reviewed changes

oneflow/user/kernels/distributions/uniform_int_kernel.h Outdated Show resolved Hide resolved

strint reviewed Feb 23, 2022

View reviewed changes

oneflow/user/kernels/distributions/uniform_int_kernel.h Outdated Show resolved Hide resolved

strint reviewed Feb 23, 2022

View reviewed changes

python/oneflow/test/modules/test_arange.py Outdated Show resolved Hide resolved

grybd added 3 commits February 24, 2022 01:56

fixed generator bug and add consistent testcase

1f2f135

modified randperm op

04e4c91

format code

fb01e9d

fixed add GetOpKernelSeed functon

7b09e4f

wyg1997 mentioned this pull request Feb 28, 2022

fix arange and add global unittest #7576

Merged

grybd added 3 commits March 1, 2022 01:07

merge into master

5906226

Merge branch 'master' of github.com:Oneflow-Inc/oneflow into source_o…

1034869

…p_support_S

fixed bug with some checks

c3a0ee5

grybd requested a review from chengtbf as a code owner March 1, 2022 04:02

randperm op support global tensor

0978452

grybd requested review from oneflow-ci-bot and removed request for oneflow-ci-bot April 30, 2022 01:13

grybd added 2 commits April 30, 2022 09:15

Merge branch 'master' into source_op_support_S

c48f851

randperm op support global tensor

55961ff

github-actions bot removed the automerge label Apr 30, 2022

grybd added 2 commits May 2, 2022 19:03

randperm op support global tensor

fd369e1

randperm op support global tensor

5f8bb79

grybd added the automerge label May 2, 2022

Merge branch 'master' into source_op_support_S

e94aaf4

github-actions bot removed the automerge label May 2, 2022

grybd added the automerge label May 2, 2022

grybd requested review from oneflow-ci-bot and removed request for oneflow-ci-bot May 2, 2022 13:58

github-actions bot removed the automerge label May 2, 2022

grybd added the automerge label May 2, 2022

grybd requested review from oneflow-ci-bot and removed request for oneflow-ci-bot May 2, 2022 14:43

mergify bot merged commit eb628c7 into master May 2, 2022

mergify bot deleted the source_op_support_S branch May 2, 2022 16:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

source op support s and fixed generator bug #7571

source op support s and fixed generator bug #7571

grybd commented Feb 23, 2022 •

edited by chengtbf

Loading

grybd commented Feb 23, 2022

grybd commented Feb 24, 2022

grybd commented Feb 24, 2022

wyg1997 commented Feb 24, 2022

github-actions bot commented Apr 30, 2022

github-actions bot commented Apr 30, 2022

github-actions bot commented May 2, 2022

github-actions bot commented May 2, 2022

github-actions bot commented May 2, 2022

github-actions bot commented May 2, 2022

grybd commented May 2, 2022

grybd commented May 2, 2022

github-actions bot commented May 2, 2022

github-actions bot commented May 2, 2022

source op support s and fixed generator bug #7571

source op support s and fixed generator bug #7571

Conversation

grybd commented Feb 23, 2022 • edited by chengtbf Loading

这个PR的目的

random op 支持 global tensor 一致性

grybd commented Feb 23, 2022

grybd commented Feb 24, 2022

补充

grybd commented Feb 24, 2022

wyg1997 commented Feb 24, 2022

github-actions bot commented Apr 30, 2022

github-actions bot commented Apr 30, 2022

github-actions bot commented May 2, 2022

github-actions bot commented May 2, 2022

github-actions bot commented May 2, 2022

github-actions bot commented May 2, 2022

grybd commented May 2, 2022

grybd commented May 2, 2022

github-actions bot commented May 2, 2022

github-actions bot commented May 2, 2022

grybd commented Feb 23, 2022 •

edited by chengtbf

Loading