Fix linalg vector norm and clip grad bug #8007

BBuf · 2022-04-12T08:15:11Z

fix linalg vector_norm op bug when ord equals zero!
align clip_grad op nan/inf check with PyTorch.

linked issue: https://github.com/Oneflow-Inc/OneTeam/issues/984

python/oneflow/test/modules/test_norm.py

python/oneflow/nn/utils/clip_grad.py

github-actions · 2022-04-13T19:13:30Z

Static analysis with clang failed. PR label automerge has been removed

* has multi definition bug * fix bug * fix commnet

github-actions · 2022-04-17T16:50:28Z

Speed stats:

GPU Name: GeForce GTX 1080 

✔️ OneFlow resnet50 time: 128.4ms (= 12844.4ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 141.2ms (= 14115.5ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.10 (= 141.2ms / 128.4ms)

OneFlow resnet50 time: 78.8ms (= 7882.4ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 85.0ms (= 8504.4ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.08 (= 85.0ms / 78.8ms)

OneFlow resnet50 time: 53.7ms (= 10743.5ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 62.2ms (= 12432.3ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.16 (= 62.2ms / 53.7ms)

OneFlow resnet50 time: 44.3ms (= 8869.4ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 46.7ms (= 9347.9ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.05 (= 46.7ms / 44.3ms)

OneFlow resnet50 time: 38.5ms (= 7709.3ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 38.4ms (= 7688.9ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.00 (= 38.4ms / 38.5ms)

OneFlow swin dataloader time: 0.263s (= 52.504s / 200, num_workers=1)
PyTorch swin dataloader time: 0.261s (= 52.284s / 200, num_workers=1)
✔️ Relative speed: 0.996 (= 0.261s / 0.263s)

OneFlow swin dataloader time: 0.064s (= 12.840s / 200, num_workers=4)
PyTorch swin dataloader time: 0.067s (= 13.441s / 200, num_workers=4)
✔️ Relative speed: 1.047 (= 0.067s / 0.064s)

OneFlow swin dataloader time: 0.036s (= 7.218s / 200, num_workers=8)
PyTorch swin dataloader time: 0.037s (= 7.490s / 200, num_workers=8)
✔️ Relative speed: 1.038 (= 0.037s / 0.036s)

✔️ OneFlow resnet50 time: 135.2ms (= 13525.0ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 157.1ms (= 15710.9ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.16 (= 157.1ms / 135.2ms)

OneFlow resnet50 time: 85.1ms (= 8505.1ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 99.2ms (= 9922.1ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.17 (= 99.2ms / 85.1ms)

OneFlow resnet50 time: 60.9ms (= 12174.8ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 76.0ms (= 15207.0ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.25 (= 76.0ms / 60.9ms)

OneFlow resnet50 time: 53.2ms (= 10644.2ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 66.8ms (= 13369.3ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.26 (= 66.8ms / 53.2ms)

OneFlow resnet50 time: 47.9ms (= 9571.3ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 62.6ms (= 12518.2ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.31 (= 62.6ms / 47.9ms)

github-actions · 2022-04-17T20:03:54Z

Speed stats:

GPU Name: GeForce GTX 1080 

✔️ OneFlow resnet50 time: 128.7ms (= 12872.8ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 139.2ms (= 13922.5ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.08 (= 139.2ms / 128.7ms)

OneFlow resnet50 time: 78.8ms (= 7878.7ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 85.4ms (= 8536.9ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.08 (= 85.4ms / 78.8ms)

OneFlow resnet50 time: 53.2ms (= 10648.2ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 58.0ms (= 11591.5ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.09 (= 58.0ms / 53.2ms)

OneFlow resnet50 time: 41.0ms (= 8192.4ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 45.6ms (= 9127.2ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.11 (= 45.6ms / 41.0ms)

OneFlow resnet50 time: 40.2ms (= 8049.3ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 39.1ms (= 7828.3ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 0.97 (= 39.1ms / 40.2ms)

OneFlow swin dataloader time: 0.259s (= 51.874s / 200, num_workers=1)
PyTorch swin dataloader time: 0.251s (= 50.237s / 200, num_workers=1)
✔️ Relative speed: 0.968 (= 0.251s / 0.259s)

OneFlow swin dataloader time: 0.065s (= 12.969s / 200, num_workers=4)
PyTorch swin dataloader time: 0.068s (= 13.540s / 200, num_workers=4)
✔️ Relative speed: 1.044 (= 0.068s / 0.065s)

OneFlow swin dataloader time: 0.036s (= 7.249s / 200, num_workers=8)
PyTorch swin dataloader time: 0.041s (= 8.157s / 200, num_workers=8)
✔️ Relative speed: 1.125 (= 0.041s / 0.036s)

✔️ OneFlow resnet50 time: 135.1ms (= 13512.4ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 155.8ms (= 15583.7ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.15 (= 155.8ms / 135.1ms)

OneFlow resnet50 time: 88.2ms (= 8817.4ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 98.9ms (= 9894.1ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.12 (= 98.9ms / 88.2ms)

OneFlow resnet50 time: 61.4ms (= 12285.8ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 75.8ms (= 15162.7ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.23 (= 75.8ms / 61.4ms)

OneFlow resnet50 time: 53.6ms (= 10710.2ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 66.0ms (= 13196.1ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.23 (= 66.0ms / 53.6ms)

OneFlow resnet50 time: 48.8ms (= 9755.5ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 61.7ms (= 12346.6ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.27 (= 61.7ms / 48.8ms)

github-actions · 2022-04-17T22:45:11Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/8007/

github-actions · 2022-04-17T22:55:03Z

Speed stats:

GPU Name: GeForce GTX 1080 

✔️ OneFlow resnet50 time: 128.3ms (= 12832.1ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 139.7ms (= 13972.8ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.09 (= 139.7ms / 128.3ms)

OneFlow resnet50 time: 77.6ms (= 7759.5ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 83.4ms (= 8336.2ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.07 (= 83.4ms / 77.6ms)

OneFlow resnet50 time: 51.8ms (= 10351.1ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 59.5ms (= 11896.5ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.15 (= 59.5ms / 51.8ms)

OneFlow resnet50 time: 43.3ms (= 8659.0ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 48.4ms (= 9687.4ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.12 (= 48.4ms / 43.3ms)

OneFlow resnet50 time: 39.3ms (= 7856.8ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 42.4ms (= 8480.0ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.08 (= 42.4ms / 39.3ms)

OneFlow swin dataloader time: 0.249s (= 49.851s / 200, num_workers=1)
PyTorch swin dataloader time: 0.247s (= 49.460s / 200, num_workers=1)
✔️ Relative speed: 0.992 (= 0.247s / 0.249s)

OneFlow swin dataloader time: 0.066s (= 13.264s / 200, num_workers=4)
PyTorch swin dataloader time: 0.071s (= 14.163s / 200, num_workers=4)
✔️ Relative speed: 1.068 (= 0.071s / 0.066s)

OneFlow swin dataloader time: 0.036s (= 7.271s / 200, num_workers=8)
PyTorch swin dataloader time: 0.036s (= 7.271s / 200, num_workers=8)
✔️ Relative speed: 1.000 (= 0.036s / 0.036s)

✔️ OneFlow resnet50 time: 135.2ms (= 13520.2ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 156.2ms (= 15622.1ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.16 (= 156.2ms / 135.2ms)

OneFlow resnet50 time: 88.9ms (= 8886.0ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 99.1ms (= 9912.4ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.12 (= 99.1ms / 88.9ms)

OneFlow resnet50 time: 60.0ms (= 11999.6ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 86.6ms (= 17320.8ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.44 (= 86.6ms / 60.0ms)

OneFlow resnet50 time: 50.6ms (= 10123.1ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 66.0ms (= 13199.1ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.30 (= 66.0ms / 50.6ms)

OneFlow resnet50 time: 46.2ms (= 9238.0ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 62.5ms (= 12508.2ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.35 (= 62.5ms / 46.2ms)

github-actions · 2022-04-17T22:58:22Z

CI failed when running job: cuda-speed-test. PR label automerge has been removed

github-actions · 2022-04-18T02:36:10Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/8007/

github-actions · 2022-04-18T02:36:34Z

Speed stats:

github-actions · 2022-04-18T05:32:01Z

Speed stats:

GPU Name: GeForce GTX 1080 

✔️ OneFlow resnet50 time: 128.8ms (= 12876.5ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 140.5ms (= 14049.5ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.09 (= 140.5ms / 128.8ms)

OneFlow resnet50 time: 79.2ms (= 7915.8ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 86.4ms (= 8637.3ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.09 (= 86.4ms / 79.2ms)

OneFlow resnet50 time: 52.2ms (= 10434.8ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 56.6ms (= 11325.8ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.09 (= 56.6ms / 52.2ms)

OneFlow resnet50 time: 43.9ms (= 8775.8ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 48.7ms (= 9733.1ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.11 (= 48.7ms / 43.9ms)

OneFlow resnet50 time: 39.4ms (= 7878.9ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 38.2ms (= 7635.8ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 0.97 (= 38.2ms / 39.4ms)

OneFlow swin dataloader time: 0.249s (= 49.842s / 200, num_workers=1)
PyTorch swin dataloader time: 0.249s (= 49.879s / 200, num_workers=1)
✔️ Relative speed: 1.001 (= 0.249s / 0.249s)

OneFlow swin dataloader time: 0.067s (= 13.329s / 200, num_workers=4)
PyTorch swin dataloader time: 0.070s (= 13.968s / 200, num_workers=4)
✔️ Relative speed: 1.048 (= 0.070s / 0.067s)

OneFlow swin dataloader time: 0.037s (= 7.430s / 200, num_workers=8)
PyTorch swin dataloader time: 0.038s (= 7.553s / 200, num_workers=8)
✔️ Relative speed: 1.017 (= 0.038s / 0.037s)

✔️ OneFlow resnet50 time: 135.1ms (= 13508.7ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 161.8ms (= 16176.9ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.20 (= 161.8ms / 135.1ms)

OneFlow resnet50 time: 85.6ms (= 8562.3ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 100.1ms (= 10013.2ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.17 (= 100.1ms / 85.6ms)

OneFlow resnet50 time: 60.6ms (= 12118.7ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 76.4ms (= 15284.0ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.26 (= 76.4ms / 60.6ms)

OneFlow resnet50 time: 51.0ms (= 10209.2ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 68.9ms (= 13785.1ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.35 (= 68.9ms / 51.0ms)

OneFlow resnet50 time: 48.1ms (= 9627.1ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 60.8ms (= 12151.6ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.26 (= 60.8ms / 48.1ms)

github-actions · 2022-04-18T05:36:55Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/8007/

BBuf added 24 commits March 22, 2022 19:15

fix reduce_sum scalar check bug

e6e6973

Merge branch 'master' of github.com:Oneflow-Inc/oneflow

a0abdd5

Merge branch 'master' of github.com:Oneflow-Inc/oneflow

00522df

Merge branch 'master' of github.com:Oneflow-Inc/oneflow

68e0e08

Merge branch 'master' of github.com:Oneflow-Inc/oneflow

0b90f9b

Merge branch 'master' of github.com:Oneflow-Inc/oneflow

d81aa80

Merge branch 'master' of github.com:Oneflow-Inc/oneflow

81954d7

Merge branch 'master' of github.com:Oneflow-Inc/oneflow

c910fbe

Merge branch 'master' of github.com:Oneflow-Inc/oneflow

32702cf

Merge branch 'master' of github.com:Oneflow-Inc/oneflow

9fa0513

Merge branch 'master' of github.com:Oneflow-Inc/oneflow

8402ce3

Merge branch 'master' of github.com:Oneflow-Inc/oneflow

e3bf835

Merge branch 'master' of github.com:Oneflow-Inc/oneflow

7a46669

Merge branch 'master' of github.com:Oneflow-Inc/oneflow

5786ed5

Merge branch 'master' of github.com:Oneflow-Inc/oneflow

1f66c27

Merge branch 'master' of github.com:Oneflow-Inc/oneflow

265c15a

Merge branch 'master' of github.com:Oneflow-Inc/oneflow

68f1baf

Merge branch 'master' of github.com:Oneflow-Inc/oneflow

8f4ce10

Merge branch 'master' of github.com:Oneflow-Inc/oneflow

df08558

Merge branch 'master' of github.com:Oneflow-Inc/oneflow

e811e5b

Merge branch 'master' of github.com:Oneflow-Inc/oneflow

f7aba4b

Merge branch 'master' of github.com:Oneflow-Inc/oneflow

af091f1

Merge branch 'master' of github.com:Oneflow-Inc/oneflow

f8adac3

fix linalg vector norm and clip grad bug

cb574ff

BBuf requested review from hjchen2, daquexian and jackalcooper as code owners April 12, 2022 08:15

Flowingsun007 approved these changes Apr 12, 2022

View reviewed changes

Flowingsun007 reviewed Apr 12, 2022

View reviewed changes

python/oneflow/test/modules/test_norm.py Outdated Show resolved Hide resolved

liufengwei0103 reviewed Apr 12, 2022

View reviewed changes

python/oneflow/nn/utils/clip_grad.py Outdated Show resolved Hide resolved

oneflow-ci-bot self-requested a review April 13, 2022 19:02

github-actions bot removed the automerge label Apr 13, 2022

BBuf and others added 3 commits April 14, 2022 09:41

Fix linalg vector norm backward bug (#8015)

a2f3499

* has multi definition bug * fix bug * fix commnet

fix bug

023c91a

Merge branch 'master' into fix_linalg_vector_norm_and_clip_grad_bug

98ce227

BBuf added the automerge label Apr 17, 2022

Merge branch 'master' into fix_linalg_vector_norm_and_clip_grad_bug

85b7a9d

BBuf requested review from oneflow-ci-bot and removed request for oneflow-ci-bot April 17, 2022 13:59

Merge branch 'master' into fix_linalg_vector_norm_and_clip_grad_bug

b54c16f

Merge branch 'master' into fix_linalg_vector_norm_and_clip_grad_bug

88079b7

Merge branch 'master' into fix_linalg_vector_norm_and_clip_grad_bug

ee5650a

github-actions bot removed the automerge label Apr 17, 2022

BBuf added the automerge label Apr 18, 2022

BBuf requested review from oneflow-ci-bot and removed request for oneflow-ci-bot April 18, 2022 01:59

Merge branch 'master' into fix_linalg_vector_norm_and_clip_grad_bug

9fd0771

mergify bot merged commit 888ad73 into master Apr 18, 2022

mergify bot deleted the fix_linalg_vector_norm_and_clip_grad_bug branch April 18, 2022 06:54

BBuf mentioned this pull request Apr 22, 2022

OneFlow 算子对齐 PyTorch 完备计划推进表 #4936

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix linalg vector norm and clip grad bug #8007

Fix linalg vector norm and clip grad bug #8007

BBuf commented Apr 12, 2022

github-actions bot commented Apr 13, 2022

github-actions bot commented Apr 17, 2022

github-actions bot commented Apr 17, 2022

github-actions bot commented Apr 17, 2022

github-actions bot commented Apr 17, 2022

github-actions bot commented Apr 17, 2022

github-actions bot commented Apr 18, 2022

github-actions bot commented Apr 18, 2022

github-actions bot commented Apr 18, 2022

github-actions bot commented Apr 18, 2022

Fix linalg vector norm and clip grad bug #8007

Fix linalg vector norm and clip grad bug #8007

Conversation

BBuf commented Apr 12, 2022

github-actions bot commented Apr 13, 2022

github-actions bot commented Apr 17, 2022

github-actions bot commented Apr 17, 2022

github-actions bot commented Apr 17, 2022

github-actions bot commented Apr 17, 2022

github-actions bot commented Apr 17, 2022

github-actions bot commented Apr 18, 2022

github-actions bot commented Apr 18, 2022

github-actions bot commented Apr 18, 2022

github-actions bot commented Apr 18, 2022