Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

update speed test threshold #7871

Merged
merged 5 commits into from
Mar 27, 2022
Merged

update speed test threshold #7871

merged 5 commits into from
Mar 27, 2022

Conversation

daquexian
Copy link
Contributor

更新 speed test 阈值

等 新版 benchmark 上线之后，这个 speed test 就可以退役，不需要手动调阈值了

@daquexian
update speed test threshold
694b3b9 
Signed-off-by: daquexian <daquexian566@gmail.com>
@daquexian daquexian requested a review from jackalcooper as a code owner March 23, 2022 03:02
daquexian
daquexian commented Mar 23, 2022
View reviewed changes
ci/test/test_speed_multi_client.sh
python3 scripts/compare_speed_with_pytorch.py Vision/classification/image/resnet50/models/resnet50.py resnet50 2x3x224x224 --no-show-memory --times 200 | check_relative_speed 0.99 | write_to_file_and_print
python3 scripts/compare_speed_with_pytorch.py Vision/classification/image/resnet50/models/resnet50.py resnet50 1x3x224x224 --no-show-memory --times 200 | check_relative_speed 0.92 | write_to_file_and_print
python3 scripts/compare_speed_with_pytorch.py Vision/classification/image/resnet50/models/resnet50.py resnet50 16x3x224x224 --no-show-memory --times 100 | check_relative_speed 1.05 | check_millisecond_time 129.0 | write_to_file_and_print
python3 scripts/compare_speed_with_pytorch.py Vision/classification/image/resnet50/models/resnet50.py resnet50 8x3x224x224 --no-show-memory --times 100 | check_relative_speed 1.05 | check_millisecond_time 80.0 | write_to_file_and_print
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

速度越大越好，时间越小越好。这里的时间从 78.8 放松到了 80.0 是因为 78.8 经常有误报

@howin98 howin98 closed this Mar 24, 2022
@daquexian daquexian reopened this Mar 27, 2022
@daquexian
Merge branch 'master' into update_speed_test_threshold
5f8843b
@daquexian
relax swin dataloader num_workers=4 threshold
8de3b35 
Signed-off-by: daquexian <daquexian566@gmail.com>
@daquexian
relax resnet50 batch size=4 threshold
025ceae 
Signed-off-by: daquexian <daquexian566@gmail.com>
@daquexian daquexian added enhancement automerge ci need-highest-priority Only add this when you really need it!!! Will block all other PRs. labels Mar 27, 2022
@jackalcooper jackalcooper enabled auto-merge (squash) March 27, 2022 09:34
@github-actions
Copy link
Contributor

Speed stats: 
GPU Name: GeForce GTX 1080 

✔️ OneFlow resnet50 time: 128.3ms (= 12827.2ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 139.9ms (= 13989.3ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.09 (= 139.9ms / 128.3ms)

✔️ OneFlow resnet50 time: 77.6ms (= 7756.1ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 81.7ms (= 8169.5ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.05 (= 81.7ms / 77.6ms)

OneFlow resnet50 time: 52.0ms (= 10391.9ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 62.5ms (= 12501.7ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.20 (= 62.5ms / 52.0ms)

OneFlow resnet50 time: 42.1ms (= 8416.9ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 46.7ms (= 9337.0ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.11 (= 46.7ms / 42.1ms)

OneFlow resnet50 time: 38.9ms (= 7776.3ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 44.4ms (= 8881.4ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.14 (= 44.4ms / 38.9ms)

OneFlow swin dataloader time: 0.249s (= 49.820s / 200, num_workers=1)
PyTorch swin dataloader time: 0.249s (= 49.813s / 200, num_workers=1)
✔️ Relative speed: 1.000 (= 0.249s / 0.249s)

OneFlow swin dataloader time: 0.069s (= 13.886s / 200, num_workers=4)
PyTorch swin dataloader time: 0.066s (= 13.162s / 200, num_workers=4)
✔️ Relative speed: 0.948 (= 0.066s / 0.069s)

OneFlow swin dataloader time: 0.037s (= 7.310s / 200, num_workers=8)
PyTorch swin dataloader time: 0.039s (= 7.714s / 200, num_workers=8)
✔️ Relative speed: 1.055 (= 0.039s / 0.037s)

✔️ OneFlow resnet50 time: 135.6ms (= 13557.4ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 157.7ms (= 15774.2ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.16 (= 157.7ms / 135.6ms)

OneFlow resnet50 time: 87.1ms (= 8710.3ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 100.4ms (= 10043.4ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.15 (= 100.4ms / 87.1ms)

OneFlow resnet50 time: 61.5ms (= 12292.0ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 76.6ms (= 15313.8ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.25 (= 76.6ms / 61.5ms)

OneFlow resnet50 time: 53.3ms (= 10667.8ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 66.2ms (= 13234.7ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.24 (= 66.2ms / 53.3ms)

OneFlow resnet50 time: 47.9ms (= 9580.0ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 61.6ms (= 12315.9ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.29 (= 61.6ms / 47.9ms)

@jackalcooper jackalcooper merged commit 91c61db into master Mar 27, 2022
@jackalcooper jackalcooper deleted the update_speed_test_threshold branch March 27, 2022 10:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lixinqi lixinqi lixinqi approved these changes

@jackalcooper jackalcooper jackalcooper approved these changes

@oneflow-ci-bot oneflow-ci-bot Awaiting requested review from oneflow-ci-bot

Assignees
No one assigned
Labels
automerge ci enhancement need-highest-priority Only add this when you really need it!!! Will block all other PRs.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@daquexian @lixinqi @jackalcooper @howin98