fix files not actually removed when building for multiple python versions

[jackalcooper]

No description provided.

[github-actions]

Speed stats:

GPU Name: GeForce GTX 1080 

PyTorch resnet50 time: 141.9ms (= 7095.0ms / 50, input_shape=[16, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 128.0ms (= 6400.9ms / 50, input_shape=[16, 3, 224, 224], backward is enabled)
Relative speed: 1.11 (= 141.9ms / 128.0ms)

PyTorch resnet50 time: 84.3ms (= 4214.1ms / 50, input_shape=[8, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 75.1ms (= 3754.1ms / 50, input_shape=[8, 3, 224, 224], backward is enabled)
Relative speed: 1.12 (= 84.3ms / 75.1ms)

PyTorch resnet50 time: 53.7ms (= 2685.3ms / 50, input_shape=[4, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 48.6ms (= 2428.9ms / 50, input_shape=[4, 3, 224, 224], backward is enabled)
Relative speed: 1.11 (= 53.7ms / 48.6ms)

PyTorch resnet50 time: 47.6ms (= 2381.9ms / 50, input_shape=[2, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 43.2ms (= 2162.0ms / 50, input_shape=[2, 3, 224, 224], backward is enabled)
Relative speed: 1.10 (= 47.6ms / 43.2ms)

PyTorch resnet50 time: 45.0ms (= 2251.3ms / 50, input_shape=[1, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 36.6ms (= 1832.0ms / 50, input_shape=[1, 3, 224, 224], backward is enabled)
Relative speed: 1.23 (= 45.0ms / 36.6ms)