Use onednn to optimize permute #7872

luqiang-guo · 2022-03-23T03:13:18Z

详细记录在此链接中 https://github.com/Oneflow-Inc/OneTeam/issues/1170#issuecomment-1075888383

测试结果

测试环境 21机器，CPU Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz
单线程模式

	oneflow 优化前	oneflow onednn实现
(224,224,3 -> 3,224,224，permute 2 0 1) 访存命中	500us	120us
(224,224,3 -> 3,224,224，permute 2 0 1) 访存未命中	903us	500us
(224,224,3 -> 224,3,224，permute 0 2 1) 访存命中	1074us	95us
(224,224,3 -> 224,3,224，permute 0 2 1) 访存未命中	1363us	484us
(224,224,3 -> 224,3,224， permute 1 0 2) 访存命中	1072us	101us
(224,224,3 -> 224,3,224， permute 1 0 2) 访存未命中	1389us	488us

oneflow/core/ep/cpu/primitive/permute.cpp

liujuncheng · 2022-04-02T08:40:39Z

oneflow/core/ep/cpu/primitive/permute.cpp

@@ -64,6 +66,74 @@ class PermuteImpl : public Permute {
  }
 };

+#ifdef WITH_ONEDNN
+// Onednn only supports up to 32bit type of data
+constexpr size_t kMaxOneDNNMovementSize = 4;


Onednn/OneDnn/OneDNN 这三种写法都出现了

github-actions · 2022-04-16T06:02:58Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/7872/

github-actions · 2022-04-16T06:10:20Z

Speed stats:

GPU Name: GeForce GTX 1080 

✔️ OneFlow resnet50 time: 128.7ms (= 12865.6ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 139.5ms (= 13951.8ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.08 (= 139.5ms / 128.7ms)

OneFlow resnet50 time: 78.8ms (= 7876.3ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 84.0ms (= 8400.2ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.07 (= 84.0ms / 78.8ms)

OneFlow resnet50 time: 52.1ms (= 10410.6ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 56.4ms (= 11281.9ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.08 (= 56.4ms / 52.1ms)

OneFlow resnet50 time: 43.3ms (= 8665.3ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 47.6ms (= 9518.6ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.10 (= 47.6ms / 43.3ms)

OneFlow resnet50 time: 38.2ms (= 7634.7ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 37.8ms (= 7559.9ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 0.99 (= 37.8ms / 38.2ms)

OneFlow swin dataloader time: 0.250s (= 49.993s / 200, num_workers=1)
PyTorch swin dataloader time: 0.252s (= 50.315s / 200, num_workers=1)
✔️ Relative speed: 1.006 (= 0.252s / 0.250s)

OneFlow swin dataloader time: 0.066s (= 13.105s / 200, num_workers=4)
PyTorch swin dataloader time: 0.068s (= 13.559s / 200, num_workers=4)
✔️ Relative speed: 1.035 (= 0.068s / 0.066s)

OneFlow swin dataloader time: 0.036s (= 7.177s / 200, num_workers=8)
PyTorch swin dataloader time: 0.036s (= 7.296s / 200, num_workers=8)
✔️ Relative speed: 1.017 (= 0.036s / 0.036s)

✔️ OneFlow resnet50 time: 135.6ms (= 13565.0ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 157.3ms (= 15730.6ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.16 (= 157.3ms / 135.6ms)

OneFlow resnet50 time: 87.9ms (= 8787.0ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 105.8ms (= 10579.1ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.20 (= 105.8ms / 87.9ms)

OneFlow resnet50 time: 62.5ms (= 12490.1ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 85.8ms (= 17151.8ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.37 (= 85.8ms / 62.5ms)

OneFlow resnet50 time: 52.2ms (= 10431.9ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 66.5ms (= 13294.1ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.27 (= 66.5ms / 52.2ms)

OneFlow resnet50 time: 48.2ms (= 9639.5ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 65.8ms (= 13163.2ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.37 (= 65.8ms / 48.2ms)

github-actions · 2022-04-16T08:26:27Z

CI failed when running job: cuda-benchmark. PR label automerge has been removed

github-actions · 2022-04-16T09:13:24Z

CI failed when running job: cuda-module. PR label automerge has been removed

github-actions · 2022-04-16T15:32:56Z

Static analysis with clang failed. PR label automerge has been removed

github-actions · 2022-04-16T17:17:42Z

Speed stats:

GPU Name: GeForce GTX 1080 

✔️ OneFlow resnet50 time: 128.6ms (= 12856.2ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 138.6ms (= 13861.7ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.08 (= 138.6ms / 128.6ms)

OneFlow resnet50 time: 79.1ms (= 7909.4ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 84.3ms (= 8432.6ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.07 (= 84.3ms / 79.1ms)

OneFlow resnet50 time: 53.2ms (= 10643.8ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 62.7ms (= 12541.5ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.18 (= 62.7ms / 53.2ms)

OneFlow resnet50 time: 42.9ms (= 8587.7ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 47.7ms (= 9549.0ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.11 (= 47.7ms / 42.9ms)

OneFlow resnet50 time: 39.2ms (= 7844.3ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 38.1ms (= 7622.4ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 0.97 (= 38.1ms / 39.2ms)

OneFlow swin dataloader time: 0.251s (= 50.249s / 200, num_workers=1)
PyTorch swin dataloader time: 0.250s (= 49.930s / 200, num_workers=1)
✔️ Relative speed: 0.994 (= 0.250s / 0.251s)

OneFlow swin dataloader time: 0.066s (= 13.284s / 200, num_workers=4)
PyTorch swin dataloader time: 0.068s (= 13.591s / 200, num_workers=4)
✔️ Relative speed: 1.023 (= 0.068s / 0.066s)

OneFlow swin dataloader time: 0.037s (= 7.355s / 200, num_workers=8)
PyTorch swin dataloader time: 0.036s (= 7.261s / 200, num_workers=8)
✔️ Relative speed: 0.987 (= 0.036s / 0.037s)

✔️ OneFlow resnet50 time: 135.4ms (= 13539.0ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 159.6ms (= 15956.5ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.18 (= 159.6ms / 135.4ms)

OneFlow resnet50 time: 88.3ms (= 8830.2ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 98.8ms (= 9881.4ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.12 (= 98.8ms / 88.3ms)

OneFlow resnet50 time: 61.4ms (= 12289.0ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 75.7ms (= 15148.8ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.23 (= 75.7ms / 61.4ms)

OneFlow resnet50 time: 52.7ms (= 10547.1ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 67.1ms (= 13418.2ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.27 (= 67.1ms / 52.7ms)

OneFlow resnet50 time: 51.7ms (= 10333.7ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 62.4ms (= 12489.4ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.21 (= 62.4ms / 51.7ms)

github-actions · 2022-04-16T17:46:04Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/7872/

github-actions · 2022-04-16T22:57:55Z

Speed stats:

GPU Name: GeForce GTX 1080 

✔️ OneFlow resnet50 time: 128.3ms (= 12825.9ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 140.0ms (= 14000.0ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.09 (= 140.0ms / 128.3ms)

OneFlow resnet50 time: 78.4ms (= 7838.5ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 83.3ms (= 8333.1ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.06 (= 83.3ms / 78.4ms)

OneFlow resnet50 time: 52.1ms (= 10413.2ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 61.3ms (= 12268.8ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.18 (= 61.3ms / 52.1ms)

OneFlow resnet50 time: 43.6ms (= 8718.1ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 47.7ms (= 9537.7ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.09 (= 47.7ms / 43.6ms)

OneFlow resnet50 time: 38.4ms (= 7673.8ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 38.5ms (= 7699.2ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.00 (= 38.5ms / 38.4ms)

OneFlow swin dataloader time: 0.254s (= 50.765s / 200, num_workers=1)
PyTorch swin dataloader time: 0.251s (= 50.163s / 200, num_workers=1)
✔️ Relative speed: 0.988 (= 0.251s / 0.254s)

OneFlow swin dataloader time: 0.068s (= 13.569s / 200, num_workers=4)
PyTorch swin dataloader time: 0.068s (= 13.568s / 200, num_workers=4)
✔️ Relative speed: 1.000 (= 0.068s / 0.068s)

OneFlow swin dataloader time: 0.038s (= 7.604s / 200, num_workers=8)
PyTorch swin dataloader time: 0.037s (= 7.414s / 200, num_workers=8)
✔️ Relative speed: 0.975 (= 0.037s / 0.038s)

✔️ OneFlow resnet50 time: 135.4ms (= 13535.9ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 156.8ms (= 15684.8ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.16 (= 156.8ms / 135.4ms)

OneFlow resnet50 time: 87.5ms (= 8746.6ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 99.6ms (= 9956.8ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.14 (= 99.6ms / 87.5ms)

OneFlow resnet50 time: 61.3ms (= 12264.9ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 75.7ms (= 15139.6ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.23 (= 75.7ms / 61.3ms)

OneFlow resnet50 time: 53.2ms (= 10632.6ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 76.9ms (= 15386.5ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.45 (= 76.9ms / 53.2ms)

OneFlow resnet50 time: 49.1ms (= 9824.0ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 69.8ms (= 13966.1ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.42 (= 69.8ms / 49.1ms)

github-actions · 2022-04-16T23:16:02Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/7872/

github-actions · 2022-04-16T23:29:30Z

CI failed when running job: cuda-benchmark. PR label automerge has been removed

github-actions · 2022-04-17T18:38:35Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/7872/

github-actions · 2022-04-17T18:48:29Z

Speed stats:

GPU Name: GeForce GTX 1080 

✔️ OneFlow resnet50 time: 128.4ms (= 12836.2ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 138.0ms (= 13803.5ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.08 (= 138.0ms / 128.4ms)

OneFlow resnet50 time: 79.1ms (= 7908.4ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 84.4ms (= 8435.8ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.07 (= 84.4ms / 79.1ms)

OneFlow resnet50 time: 51.5ms (= 10295.7ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 55.5ms (= 11102.2ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.08 (= 55.5ms / 51.5ms)

OneFlow resnet50 time: 42.4ms (= 8475.2ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 47.0ms (= 9396.3ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.11 (= 47.0ms / 42.4ms)

OneFlow resnet50 time: 38.7ms (= 7743.9ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 38.7ms (= 7737.4ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.00 (= 38.7ms / 38.7ms)

OneFlow swin dataloader time: 0.258s (= 51.602s / 200, num_workers=1)
PyTorch swin dataloader time: 0.253s (= 50.593s / 200, num_workers=1)
✔️ Relative speed: 0.980 (= 0.253s / 0.258s)

OneFlow swin dataloader time: 0.067s (= 13.407s / 200, num_workers=4)
PyTorch swin dataloader time: 0.069s (= 13.723s / 200, num_workers=4)
✔️ Relative speed: 1.024 (= 0.069s / 0.067s)

OneFlow swin dataloader time: 0.037s (= 7.341s / 200, num_workers=8)
PyTorch swin dataloader time: 0.038s (= 7.634s / 200, num_workers=8)
✔️ Relative speed: 1.040 (= 0.038s / 0.037s)

✔️ OneFlow resnet50 time: 135.4ms (= 13541.7ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 155.1ms (= 15513.0ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.15 (= 155.1ms / 135.4ms)

OneFlow resnet50 time: 82.4ms (= 8237.8ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 99.2ms (= 9918.7ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.20 (= 99.2ms / 82.4ms)

OneFlow resnet50 time: 61.2ms (= 12235.1ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 74.9ms (= 14988.0ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.23 (= 74.9ms / 61.2ms)

OneFlow resnet50 time: 53.5ms (= 10690.3ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 65.9ms (= 13181.0ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.23 (= 65.9ms / 53.5ms)

OneFlow resnet50 time: 48.9ms (= 9789.5ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 62.2ms (= 12436.9ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.27 (= 62.2ms / 48.9ms)

luqiang-guo added 2 commits March 22, 2022 17:43

test onednn stride

e8361da

add onednn permute

e35ade9

luqiang-guo requested a review from liujuncheng as a code owner March 23, 2022 03:13

luqiang-guo added enhancement primitive labels Mar 23, 2022

luqiang-guo added 8 commits March 23, 2022 18:33

add special case

7e4ac4e

fix special case error

963352f

add rgb case

ae20853

add case

4386035

delete case

34f3b0b

fix bug

e5fc4b7

delete special case

163d153

delete line

9b38cdc

luqiang-guo added the op label Mar 24, 2022

luqiang-guo requested a review from oneflow-ci-bot March 25, 2022 03:30

fix static analysis used NOLINT

0a226f9

luqiang-guo requested review from oneflow-ci-bot and removed request for oneflow-ci-bot March 25, 2022 09:25

add comment

5cb5cfa

luqiang-guo requested review from oneflow-ci-bot and removed request for oneflow-ci-bot March 25, 2022 09:44

Merge branch 'master' into dev_add_onednn_permute

c5192f6

luqiang-guo requested review from oneflow-ci-bot and removed request for oneflow-ci-bot March 25, 2022 10:00

Merge branch 'master' into dev_add_onednn_permute

bfc58a0

daquexian reviewed Mar 31, 2022

View reviewed changes

oneflow/core/ep/cpu/primitive/permute.cpp Outdated Show resolved Hide resolved

oneflow/core/ep/cpu/primitive/permute.cpp Show resolved Hide resolved

oneflow/core/ep/cpu/primitive/permute.cpp Show resolved Hide resolved

liujuncheng reviewed Apr 2, 2022

View reviewed changes

comment by dalaoshi

4327499

Merge branch 'master' into dev_add_onednn_permute

f2a431e

github-actions bot removed the automerge label Apr 16, 2022

daquexian added the automerge label Apr 16, 2022

github-actions bot removed the automerge label Apr 16, 2022

daquexian added the automerge label Apr 16, 2022

luqiang-guo requested review from oneflow-ci-bot and removed request for oneflow-ci-bot April 16, 2022 15:04

Merge branch 'master' into dev_add_onednn_permute

a0aead4

github-actions bot removed the automerge label Apr 16, 2022

daquexian added the automerge label Apr 16, 2022

Merge branch 'master' into dev_add_onednn_permute

6ccfca8

github-actions bot removed the automerge label Apr 16, 2022

Merge branch 'master' into dev_add_onednn_permute

f295437

luqiang-guo added the automerge label Apr 17, 2022

mergify bot added 2 commits April 17, 2022 15:10

Merge branch 'master' into dev_add_onednn_permute

297fb02

Merge branch 'master' into dev_add_onednn_permute

7e0d9ed

daquexian merged commit c3deae1 into master Apr 17, 2022

daquexian deleted the dev_add_onednn_permute branch April 17, 2022 20:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use onednn to optimize permute #7872

Use onednn to optimize permute #7872

luqiang-guo commented Mar 23, 2022 •

edited

Loading

liujuncheng Apr 2, 2022

luqiang-guo Apr 2, 2022 •

edited

Loading

github-actions bot commented Apr 16, 2022

github-actions bot commented Apr 16, 2022

github-actions bot commented Apr 16, 2022

github-actions bot commented Apr 16, 2022

github-actions bot commented Apr 16, 2022

github-actions bot commented Apr 16, 2022

github-actions bot commented Apr 16, 2022

github-actions bot commented Apr 16, 2022

github-actions bot commented Apr 16, 2022

github-actions bot commented Apr 16, 2022

github-actions bot commented Apr 17, 2022

github-actions bot commented Apr 17, 2022

Use onednn to optimize permute #7872

Use onednn to optimize permute #7872

Conversation

luqiang-guo commented Mar 23, 2022 • edited Loading

测试结果

liujuncheng Apr 2, 2022

Choose a reason for hiding this comment

luqiang-guo Apr 2, 2022 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Apr 16, 2022

github-actions bot commented Apr 16, 2022

github-actions bot commented Apr 16, 2022

github-actions bot commented Apr 16, 2022

github-actions bot commented Apr 16, 2022

github-actions bot commented Apr 16, 2022

github-actions bot commented Apr 16, 2022

github-actions bot commented Apr 16, 2022

github-actions bot commented Apr 16, 2022

github-actions bot commented Apr 16, 2022

github-actions bot commented Apr 17, 2022

github-actions bot commented Apr 17, 2022

luqiang-guo commented Mar 23, 2022 •

edited

Loading

luqiang-guo Apr 2, 2022 •

edited

Loading