Clear independent thread when graph destroy #7862

chengtbf · 2022-03-22T07:10:02Z

fix: https://github.com/Oneflow-Inc/OneTeam/issues/1219

仅回收 independent thread ，用于遏制 CI 中线程无限增长的问题。

…hread resume

erge branch 'dev_cc_thread_leak' of https://github.com/Oneflow-Inc/oneflow into dev_cc_thread_leak

…troy order with session

Signed-off-by: daquexian <daquexian566@gmail.com>

github-actions · 2022-04-19T19:34:14Z

Speed stats:

GPU Name: GeForce GTX 1080 

✔️ OneFlow resnet50 time: 128.6ms (= 12858.0ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 143.2ms (= 14322.4ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.11 (= 143.2ms / 128.6ms)

OneFlow resnet50 time: 81.0ms (= 8100.0ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 87.1ms (= 8706.3ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.07 (= 87.1ms / 81.0ms)

OneFlow resnet50 time: 55.2ms (= 11044.8ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 59.1ms (= 11820.4ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.07 (= 59.1ms / 55.2ms)

OneFlow resnet50 time: 45.6ms (= 9110.6ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 42.7ms (= 8536.9ms / 200, input_shape=[2, 3, 224, 224])
❌ Relative speed: 0.94 (= 42.7ms / 45.6ms)

OneFlow resnet50 time: 38.9ms (= 7771.5ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 38.5ms (= 7697.7ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 0.99 (= 38.5ms / 38.9ms)

OneFlow swin dataloader time: 0.261s (= 52.171s / 200, num_workers=1)
PyTorch swin dataloader time: 0.150s (= 30.031s / 200, num_workers=1)
Relative speed: 0.576 (= 0.150s / 0.261s)

OneFlow swin dataloader time: 0.070s (= 14.032s / 200, num_workers=4)
PyTorch swin dataloader time: 0.041s (= 8.278s / 200, num_workers=4)
Relative speed: 0.590 (= 0.041s / 0.070s)

OneFlow swin dataloader time: 0.038s (= 7.640s / 200, num_workers=8)
PyTorch swin dataloader time: 0.023s (= 4.507s / 200, num_workers=8)
Relative speed: 0.590 (= 0.023s / 0.038s)

✔️ OneFlow resnet50 time: 135.7ms (= 13566.3ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 160.6ms (= 16061.8ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.18 (= 160.6ms / 135.7ms)

OneFlow resnet50 time: 88.9ms (= 8886.9ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 102.4ms (= 10239.0ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.15 (= 102.4ms / 88.9ms)

OneFlow resnet50 time: 63.4ms (= 12676.6ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 77.4ms (= 15470.4ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.22 (= 77.4ms / 63.4ms)

OneFlow resnet50 time: 55.8ms (= 11159.1ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 80.7ms (= 16143.2ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.45 (= 80.7ms / 55.8ms)

OneFlow resnet50 time: 51.0ms (= 10198.4ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 67.8ms (= 13566.2ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.33 (= 67.8ms / 51.0ms)

github-actions · 2022-04-19T19:58:01Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/7862/

github-actions · 2022-04-19T20:05:34Z

CI failed when running job: cuda-module. PR label automerge has been removed

…dev_cc_thread_leak

github-actions · 2022-04-20T13:19:25Z

Speed stats:

GPU Name: GeForce GTX 1080 

❌ OneFlow resnet50 time: 129.5ms (= 12945.5ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 145.0ms (= 14501.1ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.12 (= 145.0ms / 129.5ms)

OneFlow resnet50 time: 82.4ms (= 8240.6ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 88.0ms (= 8798.8ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.07 (= 88.0ms / 82.4ms)

OneFlow resnet50 time: 55.7ms (= 11133.1ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 57.2ms (= 11435.5ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.03 (= 57.2ms / 55.7ms)

OneFlow resnet50 time: 43.1ms (= 8628.1ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 44.5ms (= 8890.9ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.03 (= 44.5ms / 43.1ms)

OneFlow resnet50 time: 39.9ms (= 7983.8ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 44.3ms (= 8858.8ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.11 (= 44.3ms / 39.9ms)

OneFlow swin dataloader time: 0.254s (= 50.710s / 200, num_workers=1)
PyTorch swin dataloader time: 0.152s (= 30.300s / 200, num_workers=1)
Relative speed: 0.598 (= 0.152s / 0.254s)

OneFlow swin dataloader time: 0.069s (= 13.886s / 200, num_workers=4)
PyTorch swin dataloader time: 0.044s (= 8.759s / 200, num_workers=4)
Relative speed: 0.631 (= 0.044s / 0.069s)

OneFlow swin dataloader time: 0.037s (= 7.433s / 200, num_workers=8)
PyTorch swin dataloader time: 0.022s (= 4.361s / 200, num_workers=8)
Relative speed: 0.587 (= 0.022s / 0.037s)

❌ OneFlow resnet50 time: 139.2ms (= 13919.8ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 164.0ms (= 16398.4ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.18 (= 164.0ms / 139.2ms)

OneFlow resnet50 time: 91.7ms (= 9167.4ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 112.7ms (= 11273.0ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.23 (= 112.7ms / 91.7ms)

OneFlow resnet50 time: 65.3ms (= 13050.3ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 79.3ms (= 15869.2ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.22 (= 79.3ms / 65.3ms)

OneFlow resnet50 time: 56.1ms (= 11224.2ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 63.5ms (= 12692.6ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
❌ Relative speed: 1.13 (= 63.5ms / 56.1ms)

OneFlow resnet50 time: 50.9ms (= 10182.2ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 68.0ms (= 13596.0ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.34 (= 68.0ms / 50.9ms)

github-actions · 2022-04-20T13:43:16Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/7862/

github-actions · 2022-04-21T03:54:23Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/7862/

github-actions · 2022-04-21T04:03:52Z

Speed stats:

GPU Name: GeForce GTX 1080 

✔️ OneFlow resnet50 time: 128.9ms (= 12889.9ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 140.3ms (= 14028.0ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.09 (= 140.3ms / 128.9ms)

OneFlow resnet50 time: 79.9ms (= 7986.3ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 86.4ms (= 8640.9ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.08 (= 86.4ms / 79.9ms)

OneFlow resnet50 time: 56.9ms (= 11384.4ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 57.6ms (= 11515.3ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.01 (= 57.6ms / 56.9ms)

OneFlow resnet50 time: 43.2ms (= 8644.9ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 42.5ms (= 8500.2ms / 200, input_shape=[2, 3, 224, 224])
❌ Relative speed: 0.98 (= 42.5ms / 43.2ms)

OneFlow resnet50 time: 41.8ms (= 8369.6ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 40.4ms (= 8071.8ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 0.96 (= 40.4ms / 41.8ms)

OneFlow swin dataloader time: 0.259s (= 51.857s / 200, num_workers=1)
PyTorch swin dataloader time: 0.153s (= 30.584s / 200, num_workers=1)
Relative speed: 0.590 (= 0.153s / 0.259s)

OneFlow swin dataloader time: 0.067s (= 13.385s / 200, num_workers=4)
PyTorch swin dataloader time: 0.043s (= 8.535s / 200, num_workers=4)
Relative speed: 0.638 (= 0.043s / 0.067s)

OneFlow swin dataloader time: 0.037s (= 7.362s / 200, num_workers=8)
PyTorch swin dataloader time: 0.022s (= 4.438s / 200, num_workers=8)
Relative speed: 0.603 (= 0.022s / 0.037s)

✔️ OneFlow resnet50 time: 136.3ms (= 13631.4ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 164.9ms (= 16490.5ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.21 (= 164.9ms / 136.3ms)

OneFlow resnet50 time: 97.2ms (= 9721.8ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 103.8ms (= 10376.1ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
❌ Relative speed: 1.07 (= 103.8ms / 97.2ms)

OneFlow resnet50 time: 66.0ms (= 13195.6ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 83.7ms (= 16746.6ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.27 (= 83.7ms / 66.0ms)

OneFlow resnet50 time: 54.4ms (= 10878.7ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 78.0ms (= 15591.4ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.43 (= 78.0ms / 54.4ms)

OneFlow resnet50 time: 49.6ms (= 9918.6ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 65.9ms (= 13187.2ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.33 (= 65.9ms / 49.6ms)

github-actions · 2022-04-21T04:53:44Z

CI failed when running job: cuda-misc. PR label automerge has been removed

github-actions · 2022-04-22T00:27:26Z

Speed stats:

github-actions · 2022-04-22T02:27:51Z

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/7862/

github-actions · 2022-04-22T02:37:53Z

Speed stats:

GPU Name: GeForce GTX 1080 

✔️ OneFlow resnet50 time: 128.6ms (= 12856.2ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 140.9ms (= 14087.6ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.10 (= 140.9ms / 128.6ms)

OneFlow resnet50 time: 81.4ms (= 8136.8ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 84.0ms (= 8395.9ms / 100, input_shape=[8, 3, 224, 224])
❌ Relative speed: 1.03 (= 84.0ms / 81.4ms)

OneFlow resnet50 time: 54.4ms (= 10872.8ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 53.7ms (= 10735.3ms / 200, input_shape=[4, 3, 224, 224])
❌ Relative speed: 0.99 (= 53.7ms / 54.4ms)

OneFlow resnet50 time: 46.1ms (= 9226.4ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 44.9ms (= 8979.6ms / 200, input_shape=[2, 3, 224, 224])
❌ Relative speed: 0.97 (= 44.9ms / 46.1ms)

OneFlow resnet50 time: 39.1ms (= 7810.6ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 40.6ms (= 8129.0ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.04 (= 40.6ms / 39.1ms)

OneFlow swin dataloader time: 0.258s (= 51.546s / 200, num_workers=1)
PyTorch swin dataloader time: 0.153s (= 30.673s / 200, num_workers=1)
Relative speed: 0.595 (= 0.153s / 0.258s)

OneFlow swin dataloader time: 0.069s (= 13.727s / 200, num_workers=4)
PyTorch swin dataloader time: 0.042s (= 8.495s / 200, num_workers=4)
Relative speed: 0.619 (= 0.042s / 0.069s)

OneFlow swin dataloader time: 0.037s (= 7.477s / 200, num_workers=8)
PyTorch swin dataloader time: 0.023s (= 4.550s / 200, num_workers=8)
Relative speed: 0.609 (= 0.023s / 0.037s)

✔️ OneFlow resnet50 time: 135.7ms (= 13570.2ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 160.3ms (= 16026.6ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.18 (= 160.3ms / 135.7ms)

OneFlow resnet50 time: 92.6ms (= 9256.7ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 106.4ms (= 10643.0ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.15 (= 106.4ms / 92.6ms)

OneFlow resnet50 time: 64.4ms (= 12881.9ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 89.2ms (= 17840.9ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.38 (= 89.2ms / 64.4ms)

OneFlow resnet50 time: 55.8ms (= 11163.9ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 69.7ms (= 13944.3ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.25 (= 69.7ms / 55.8ms)

OneFlow resnet50 time: 53.6ms (= 10710.6ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 72.1ms (= 14424.2ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.35 (= 72.1ms / 53.6ms)

chengtbf and others added 30 commits February 28, 2022 22:29

Clear empty thread when graph destroy

5feb16b

Merge branch 'master' into dev_cc_thread_leak

7e3f93a

fix bug of thread empty

458e678

Rollback NNGraph weak_ptr hold by MultiClientSessCtx and fix bug of t…

2116953

…hread resume

limit thread num 5k

de27358

limit threads num 3000

08b4104

distributed run limit threads 1000

87cc7f0

Merge branch 'master' into dev_cc_thread_leak

9767719

fix

1b7224b

:wq

1e9e7a1

erge branch 'dev_cc_thread_leak' of https://github.com/Oneflow-Inc/oneflow into dev_cc_thread_leak

refine code for review 1

f1b6d1a

add note

545827c

Merge branch 'master' into dev_cc_thread_leak

57ea016

Merge branch 'master' into dev_cc_thread_leak

7f2afb2

Merge branch 'master' into dev_cc_thread_leak

26f5417

different thread limit for cuda and cpu

415ec63

Merge branch 'master' into dev_cc_thread_leak

803d55c

Merge branch 'master' into dev_cc_thread_leak

190ab62

Merge branch 'master' into dev_cc_thread_leak

ac22e06

Add lock of thread del. And using blocking cnt to make sure graph des…

2a3742e

…troy order with session

Merge branch 'master' into dev_cc_thread_leak

207afe3

remove IsMultiClient() and single client logic

0bae07a

Signed-off-by: daquexian <daquexian566@gmail.com>

rename eager.multi_client to eager

959346b

Signed-off-by: daquexian <daquexian566@gmail.com>

auto format by CI

9a03edb

Fix bug of graph destroy order in bad case

0f895c9

add py ref

c59716f

refine new session

d531d43

clean code

ade9d9c

make scope api inner use

e0d7dbb

use session with ref cnt

ba9e31d

github-actions bot removed the automerge label Apr 19, 2022

Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow into …

79685a3

…dev_cc_thread_leak

refine thread limit

ced21bf

chengtbf added the automerge label Apr 20, 2022

mergify bot added 2 commits April 20, 2022 14:49

Merge branch 'master' into dev_cc_thread_leak

8f46976

Merge branch 'master' into dev_cc_thread_leak

0f700f2

chengtbf requested review from oneflow-ci-bot and removed request for oneflow-ci-bot April 21, 2022 02:59

github-actions bot removed the automerge label Apr 21, 2022

chengtbf removed the request for review from oneflow-ci-bot April 21, 2022 06:56

chengtbf added the automerge label Apr 21, 2022

chengtbf requested a review from oneflow-ci-bot April 21, 2022 06:56

chengtbf and others added 5 commits April 21, 2022 14:56

Merge branch 'master' into dev_cc_thread_leak

3463d0a

Merge branch 'master' into dev_cc_thread_leak

3bdb9c3

Merge branch 'master' into dev_cc_thread_leak

3715a27

Merge branch 'master' into dev_cc_thread_leak

fb6ab61

Merge branch 'master' into dev_cc_thread_leak

cf38d66

mergify bot merged commit c8c6d35 into master Apr 22, 2022

mergify bot deleted the dev_cc_thread_leak branch April 22, 2022 03:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clear independent thread when graph destroy #7862

Clear independent thread when graph destroy #7862

chengtbf commented Mar 22, 2022 •

edited

Loading

github-actions bot commented Apr 19, 2022

github-actions bot commented Apr 19, 2022

github-actions bot commented Apr 19, 2022

github-actions bot commented Apr 20, 2022

github-actions bot commented Apr 20, 2022

github-actions bot commented Apr 21, 2022

github-actions bot commented Apr 21, 2022

github-actions bot commented Apr 21, 2022

github-actions bot commented Apr 22, 2022

github-actions bot commented Apr 22, 2022

github-actions bot commented Apr 22, 2022

Clear independent thread when graph destroy #7862

Clear independent thread when graph destroy #7862

Conversation

chengtbf commented Mar 22, 2022 • edited Loading

github-actions bot commented Apr 19, 2022

github-actions bot commented Apr 19, 2022

github-actions bot commented Apr 19, 2022

github-actions bot commented Apr 20, 2022

github-actions bot commented Apr 20, 2022

github-actions bot commented Apr 21, 2022

github-actions bot commented Apr 21, 2022

github-actions bot commented Apr 21, 2022

github-actions bot commented Apr 22, 2022

github-actions bot commented Apr 22, 2022

github-actions bot commented Apr 22, 2022

chengtbf commented Mar 22, 2022 •

edited

Loading