Token seq id

[lixinqi]

不再固定token传输main线程上的数据。而是会为每个token在实际传输的时候附带序列号。
序列号在(src_token, dst_token, thread_consistent_id, rank_group)下分类独立自增。

[lixinqi]

这会在main线程上同步元信息。

[lixinqi]

这会在scheduler县城上同步张量数据。

[github-actions]

CI failed, removing label automerge

[github-actions]

CI failed, removing label automerge

[github-actions]

CI failed, removing label automerge

[github-actions]

CI failed, removing label automerge

[github-actions]

Speed stats:

GPU Name: GeForce GTX 1080 

PyTorch resnet50 time: 139.0ms (= 6951.8ms / 50, input_shape=[16, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 128.1ms (= 6404.7ms / 50, input_shape=[16, 3, 224, 224], backward is enabled)
Relative speed: 1.09 (= 139.0ms / 128.1ms)

PyTorch resnet50 time: 82.8ms (= 4137.7ms / 50, input_shape=[8, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 74.5ms (= 3723.2ms / 50, input_shape=[8, 3, 224, 224], backward is enabled)
Relative speed: 1.11 (= 82.8ms / 74.5ms)

PyTorch resnet50 time: 61.2ms (= 3062.0ms / 50, input_shape=[4, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 47.4ms (= 2368.2ms / 50, input_shape=[4, 3, 224, 224], backward is enabled)
Relative speed: 1.29 (= 61.2ms / 47.4ms)

PyTorch resnet50 time: 47.9ms (= 2392.8ms / 50, input_shape=[2, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 43.5ms (= 2177.4ms / 50, input_shape=[2, 3, 224, 224], backward is enabled)
Relative speed: 1.10 (= 47.9ms / 43.5ms)

PyTorch resnet50 time: 41.8ms (= 2091.7ms / 50, input_shape=[1, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 39.4ms (= 1972.3ms / 50, input_shape=[1, 3, 224, 224], backward is enabled)
Relative speed: 1.06 (= 41.8ms / 39.4ms)