Add manually mem gc python api

[lixiang007666]

This PR is done:

• 新增 flow.cuda.empty_cache() 的 python 接口， 类似 https://pytorch.org/docs/stable/generated/torch.cuda.empty_cache.html 。
• 在测试中增加 oneflow gpu mem 变化的验证，依赖 pynvml，在ci容器里安装。
• 新增 CUDAGetMemoryUsed()。

#8452 的后续，其中，在 C++ 层 VM Sync + GC 的有效性已经得到验证（graph上）。

Test:

脚本：

#import torch
import oneflow as torch
device = torch.device('cuda')

def get_gpu_mem_info(gpu_id=0):
    import pynvml
    pynvml.nvmlInit()
    if gpu_id < 0 or gpu_id >= pynvml.nvmlDeviceGetCount():
        print(r'gpu_id {} is not exist'.format(gpu_id))
        return 0, 0, 0

    handler = pynvml.nvmlDeviceGetHandleByIndex(gpu_id)
    meminfo = pynvml.nvmlDeviceGetMemoryInfo(handler)
    total = round(meminfo.total / 1024 / 1024, 2)
    used = round(meminfo.used / 1024 / 1024, 2)
    free = round(meminfo.free / 1024 / 1024, 2)
    print("total: ",total," used: ",used," free: ",free)

x = torch.randn(512, 3, 512, 512).to(device)
get_gpu_mem_info(1)

x = torch.randn(1, 3, 512, 512).to(device)
get_gpu_mem_info(1)

torch.cuda.empty_cache()
get_gpu_mem_info(1)

观察三次显存变化如下：

Torch:

total	used	free
7982.31	2520.0	5462.31
7982.31	2520.0	5462.31
7982.31	1004.0	6978.31

Oneflow:

total	used	free
7982.31	1976.0	6006.31
7982.31	1976.0	6006.31
7982.31	796.0	7186.31

当脚本执行结束，再次查看 nvidia-smi，固定的显存 used 才释放。

下面是多次执行申请释放的完整例子：

import oneflow
import oneflow.nn as nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv = oneflow.nn.Conv2d(3, 28, (3, 3))
        self.maxpool1 = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
        self.maxpool2 = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
    def forward(self, x):
        x = self.conv(x)
        x = self.maxpool1(x)
        x = self.maxpool2(x)
        return x

# x = oneflow.randn(120, 3, 512, 512).to('cuda')
# x = x.cpu()

def train():
    net = Net().to('cuda')
    for i in range(100):
        with oneflow.no_grad():
            frames_batches = oneflow.randn(120, 3, 224, 224).to('cuda')
            pred = net(frames_batches)
            # oneflow.cuda.empty_cache()

if __name__ == '__main__':
    train()

当不加 oneflow.cuda.empty_cache() 时，GPU Memory Usage 为 3011 MiB。
使用 oneflow.cuda.empty_cache() 时，GPU Memory Usage 为 1675 MiB。

[github-actions]

Static analysis with clang failed. PR label automerge has been removed

[github-actions]

CI failed when running job: Build cpu. PR label automerge has been removed

[github-actions]

Speed stats:

[github-actions]

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/8482/

[github-actions]

Speed stats:

GPU Name: NVIDIA GeForce GTX 1080 

❌ OneFlow resnet50 time: 129.2ms (= 12922.3ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 144.3ms (= 14431.3ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.12 (= 144.3ms / 129.2ms)

OneFlow resnet50 time: 75.8ms (= 7582.3ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 84.7ms (= 8472.7ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.12 (= 84.7ms / 75.8ms)

OneFlow resnet50 time: 49.6ms (= 9922.4ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 57.2ms (= 11442.3ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.15 (= 57.2ms / 49.6ms)

OneFlow resnet50 time: 40.0ms (= 8007.4ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 45.6ms (= 9110.8ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.14 (= 45.6ms / 40.0ms)

OneFlow resnet50 time: 35.1ms (= 7015.9ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 40.3ms (= 8052.0ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.15 (= 40.3ms / 35.1ms)

OneFlow swin dataloader time: 0.260s (= 51.957s / 200, num_workers=1)
PyTorch swin dataloader time: 0.152s (= 30.476s / 200, num_workers=1)
Relative speed: 0.587 (= 0.152s / 0.260s)

OneFlow swin dataloader time: 0.076s (= 15.176s / 200, num_workers=4)
PyTorch swin dataloader time: 0.042s (= 8.385s / 200, num_workers=4)
Relative speed: 0.553 (= 0.042s / 0.076s)

OneFlow swin dataloader time: 0.041s (= 8.183s / 200, num_workers=8)
PyTorch swin dataloader time: 0.023s (= 4.585s / 200, num_workers=8)
Relative speed: 0.560 (= 0.023s / 0.041s)

❌ OneFlow resnet50 time: 144.1ms (= 14405.5ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 168.9ms (= 16889.5ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.17 (= 168.9ms / 144.1ms)

OneFlow resnet50 time: 93.4ms (= 9343.0ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 111.8ms (= 11178.5ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.20 (= 111.8ms / 93.4ms)

OneFlow resnet50 time: 72.0ms (= 14398.8ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 87.9ms (= 17582.0ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.22 (= 87.9ms / 72.0ms)

OneFlow resnet50 time: 58.1ms (= 11623.3ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 76.8ms (= 15367.4ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.32 (= 76.8ms / 58.1ms)

OneFlow resnet50 time: 49.2ms (= 9848.0ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 74.3ms (= 14865.6ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.51 (= 74.3ms / 49.2ms)