CUDA Graphs delayed capture

[liujuncheng]

No description provided.

[github-actions]

CI failed when running job: cuda-module. PR label automerge has been removed

[github-actions]

Speed stats:

[github-actions]

Speed stats:

GPU Name: GeForce GTX 1080 

✔️ OneFlow resnet50 time: 128.6ms (= 12860.0ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 142.5ms (= 14253.9ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.11 (= 142.5ms / 128.6ms)

OneFlow resnet50 time: 76.1ms (= 7609.8ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 85.8ms (= 8579.2ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.13 (= 85.8ms / 76.1ms)

OneFlow resnet50 time: 48.9ms (= 9770.9ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 58.9ms (= 11780.4ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.21 (= 58.9ms / 48.9ms)

OneFlow resnet50 time: 39.6ms (= 7911.2ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 45.9ms (= 9176.2ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.16 (= 45.9ms / 39.6ms)

OneFlow resnet50 time: 34.5ms (= 6898.8ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 37.3ms (= 7456.6ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.08 (= 37.3ms / 34.5ms)

OneFlow swin dataloader time: 0.247s (= 49.485s / 200, num_workers=1)
PyTorch swin dataloader time: 0.150s (= 30.012s / 200, num_workers=1)
Relative speed: 0.606 (= 0.150s / 0.247s)

OneFlow swin dataloader time: 0.074s (= 14.872s / 200, num_workers=4)
PyTorch swin dataloader time: 0.040s (= 8.020s / 200, num_workers=4)
Relative speed: 0.539 (= 0.040s / 0.074s)

OneFlow swin dataloader time: 0.059s (= 11.883s / 200, num_workers=8)
PyTorch swin dataloader time: 0.021s (= 4.250s / 200, num_workers=8)
Relative speed: 0.358 (= 0.021s / 0.059s)

❌ OneFlow resnet50 time: 137.9ms (= 13794.2ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 163.9ms (= 16390.5ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.19 (= 163.9ms / 137.9ms)

OneFlow resnet50 time: 87.1ms (= 8710.1ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 103.7ms (= 10373.1ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.19 (= 103.7ms / 87.1ms)

OneFlow resnet50 time: 57.3ms (= 11450.5ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 77.3ms (= 15453.3ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.35 (= 77.3ms / 57.3ms)

OneFlow resnet50 time: 47.1ms (= 9411.0ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 67.6ms (= 13527.2ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.44 (= 67.6ms / 47.1ms)

OneFlow resnet50 time: 41.8ms (= 8358.3ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 76.0ms (= 15199.5ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.82 (= 76.0ms / 41.8ms)