[CUTLASS] Add conv2d profiler

[masahi]

Adds a dedicated profiler for conv2d. The source template is modified from the cutlass example in https://github.com/NVIDIA/cutlass/tree/master/examples/16_ampere_tensorop_conv2dfprop

The table below shows selected kernels and their runtime on resnet50 workload, using TVM's profiler I added and cutlass's cutlass_profiler. Scripts I used to get those numbers are available at https://github.com/masahi/tvm-cutlass-eval/tree/master/conv2d.

The reported runtimes mostly agree between two profilers. One major issue I'm aware of is that our profiler is much slower than the cutlass one: While cutlass_profiler takes only 4 min to generate all numbers below, the TVM profiler using this script took 40 min!! I don't see an obvious cause of slowdown in our profiler source at contrib/cutlass/conv2d_profiler.py, but I haven't investigated deeply yet.

cc @comaniac @Laurawly

workload	TVM selected kernel name	CUTLASS selected kernel name	TVM selected kernel runtime	CUTLASS selected kernel runtime
{'n': 256, 'h': 56, 'w': 56, 'c': 64, 'k': 256, 'r': 1, 's': 1, 'pad_h': 0, 'pad_w': 0, 'stride_h': 1, 'stride_w': 1}	cutlass_tensorop_h1688fprop_optimized_64x64_32x2_nhwc_align8	cutlass_tensorop_h1688fprop_optimized_64x64_32x2_nhwc_align8	1.27407	1.27324
{'n': 256, 'h': 56, 'w': 56, 'c': 64, 'k': 64, 'r': 1, 's': 1, 'pad_h': 0, 'pad_w': 0, 'stride_h': 1, 'stride_w': 1}	cutlass_tensorop_h1688fprop_optimized_64x128_32x2_nhwc_align2	cutlass_tensorop_h16816fprop_optimized_128x64_32x6_nhwc_align8	0.5039	0.503762
{'n': 256, 'h': 56, 'w': 56, 'c': 64, 'k': 64, 'r': 3, 's': 3, 'pad_h': 1, 'pad_w': 1, 'stride_h': 1, 'stride_w': 1}	cutlass_tensorop_h1688fprop_optimized_128x64_32x2_nhwc_align8	cutlass_tensorop_h1688fprop_optimized_128x64_32x2_nhwc_align8	1.01041	0.994296
{'n': 256, 'h': 56, 'w': 56, 'c': 256, 'k': 64, 'r': 1, 's': 1, 'pad_h': 0, 'pad_w': 0, 'stride_h': 1, 'stride_w': 1}	cutlass_tensorop_h1688fprop_optimized_128x64_32x2_nhwc_align8	cutlass_tensorop_h1688fprop_optimized_64x64_32x2_nhwc_align8	1.23902	1.23865
{'n': 256, 'h': 56, 'w': 56, 'c': 256, 'k': 512, 'r': 1, 's': 1, 'pad_h': 0, 'pad_w': 0, 'stride_h': 2, 'stride_w': 2}	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	0.835523	0.828407
{'n': 256, 'h': 56, 'w': 56, 'c': 256, 'k': 128, 'r': 1, 's': 1, 'pad_h': 0, 'pad_w': 0, 'stride_h': 2, 'stride_w': 2}	cutlass_tensorop_h16816fprop_optimized_64x128_32x6_nhwc_align8	cutlass_tensorop_h16816fprop_optimized_64x128_32x6_nhwc_align8	0.383695	0.38307
{'n': 256, 'h': 28, 'w': 28, 'c': 128, 'k': 128, 'r': 3, 's': 3, 'pad_h': 1, 'pad_w': 1, 'stride_h': 1, 'stride_w': 1}	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	0.740372	0.753655
{'n': 256, 'h': 28, 'w': 28, 'c': 128, 'k': 512, 'r': 1, 's': 1, 'pad_h': 0, 'pad_w': 0, 'stride_h': 1, 'stride_w': 1}	cutlass_tensorop_h1688fprop_optimized_128x128_32x2_nhwc_align8	cutlass_tensorop_h1688fprop_optimized_128x128_32x2_nhwc_align8	0.645284	0.646561
{'n': 256, 'h': 28, 'w': 28, 'c': 512, 'k': 128, 'r': 1, 's': 1, 'pad_h': 0, 'pad_w': 0, 'stride_h': 1, 'stride_w': 1}	cutlass_tensorop_h16816fprop_optimized_64x128_32x6_nhwc_align8	cutlass_tensorop_h16816fprop_optimized_64x128_32x6_nhwc_align8	0.626012	0.626975
{'n': 256, 'h': 28, 'w': 28, 'c': 512, 'k': 1024, 'r': 1, 's': 1, 'pad_h': 0, 'pad_w': 0, 'stride_h': 2, 'stride_w': 2}	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	0.774748	0.766843
{'n': 256, 'h': 28, 'w': 28, 'c': 512, 'k': 256, 'r': 1, 's': 1, 'pad_h': 0, 'pad_w': 0, 'stride_h': 2, 'stride_w': 2}	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	0.208896	0.202144
{'n': 256, 'h': 14, 'w': 14, 'c': 256, 'k': 256, 'r': 3, 's': 3, 'pad_h': 1, 'pad_w': 1, 'stride_h': 1, 'stride_w': 1}	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	0.737978	0.743775
{'n': 256, 'h': 14, 'w': 14, 'c': 256, 'k': 1024, 'r': 1, 's': 1, 'pad_h': 0, 'pad_w': 0, 'stride_h': 1, 'stride_w': 1}	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	0.408544	0.410949
{'n': 256, 'h': 14, 'w': 14, 'c': 1024, 'k': 256, 'r': 1, 's': 1, 'pad_h': 0, 'pad_w': 0, 'stride_h': 1, 'stride_w': 1}	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	0.362004	0.359183
{'n': 256, 'h': 14, 'w': 14, 'c': 1024, 'k': 2048, 'r': 1, 's': 1, 'pad_h': 0, 'pad_w': 0, 'stride_h': 2, 'stride_w': 2}	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	0.707604	0.694497
{'n': 256, 'h': 14, 'w': 14, 'c': 1024, 'k': 512, 'r': 1, 's': 1, 'pad_h': 0, 'pad_w': 0, 'stride_h': 2, 'stride_w': 2}	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	0.18526	0.174509
{'n': 256, 'h': 7, 'w': 7, 'c': 512, 'k': 512, 'r': 3, 's': 3, 'pad_h': 1, 'pad_w': 1, 'stride_h': 1, 'stride_w': 1}	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	0.729611	0.736473
{'n': 256, 'h': 7, 'w': 7, 'c': 512, 'k': 2048, 'r': 1, 's': 1, 'pad_h': 0, 'pad_w': 0, 'stride_h': 1, 'stride_w': 1}	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	0.38751	0.382707
{'n': 256, 'h': 7, 'w': 7, 'c': 2048, 'k': 512, 'r': 1, 's': 1, 'pad_h': 0, 'pad_w': 0, 'stride_h': 1, 'stride_w': 1}	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	cutlass_tensorop_h16816fprop_optimized_128x128_32x3_nhwc_align8	0.353987	0.333313

[jroesch]

cc @tkonolige

[tkonolige]

Don't we still want to consider our built in schedules if cudnn is enabled?

[masahi]

cuDNN requires a different kernel layout than AutoTVM when the input layout is NHWC, in which case two implementations are not compatible.

But there is no problem with the NCHW layout, so I've refined the condition to

    elif is_depthwise_conv2d(data.shape, layout, kernel.shape, kernel_layout, groups) and (
        layout == "NCHW" or "cudnn" not in target.libs):

[tkonolige]

Do we have to write a new class like this for each op we want to support? If so, we might want to think about making something more reusable.

[masahi]

Yes, but there is not much to share between conv2d and gemm profilers. For now, these twos ops are the only ones we want to support.

[comaniac]

I raised this topic before in the GEMM profiler PR, but I agreed with @masahi that it seems not much to share and CUTLASS basically only supports GEMM and Conv2D. Accordingly, it might be a bit overkill to have a common base class at least for now.

[tkonolige]

What is the reasoning for putting the timing code inside this template instead of having the template being just the kernel and using our built in timers instead?

[masahi]

We don't invoke the full TVM compilation pipeline with BYOC yet at this point: The goal of these profiler templates are just to select the best implementation given a workload. So we can't make use of the built-in profiler.

We could invoke the TVM compilation for each candidate kernel, run and record the execution time using the built-in profiler. The advantage of the current approach is that we can compile profiler binaries once and cache them to a work directory, so the compilation cost is amortized over different workloads / networks. We could do the similar thing with the TVM compilation approach, but that requires compiling each module with dynamic shapes, and store *.so files instead of executables.

Anyway, this is the way GEMM profiler was already written by @Laurawly, so I inherited the same approach.

[tkonolige]

You don't need to need to invoke the whole compiler pipeline to use the timing code. It will accept any packedfunc. Here the packedfunc would just run the kernel.

If we only are every going to conv2d and gemm, then it probably doesn't matter.

[tkonolige]

Using cudaEventRecord is good, but I notice the gemm_profiler does not do that. Maybe we should fix it?

[masahi]

Yeah, I noticed that too and we should use cuda events there as well. I didn't look deeply into gemm_profiler or compare the selected gemm kernels with cutlass_profiler like I did with the covn2d profiler in this PR. Any comment @Laurawly?

[masahi]

Anyway, I want to defer addressing this problem in future PR.

[tkonolige]

@masahi Maybe the reason why the TVM script takes so long is that you are doing 100 iterations per benchmark where as the cutlass script is only doing 20? Also the TVM script is running through the whole tvm compilation pipeline for each workload.

[masahi]

@masahi Maybe the reason why the TVM script takes so long is that you are doing 100 iterations per benchmark where as the cutlass script is only doing 20?

I believe cutlass_profiler is also doing 100 iterations by default: https://github.com/NVIDIA/cutlass/blob/808c25337a3ed4c97ac21895257b1addc72d6ca8/tools/profiler/src/options.cu#L386

Also the TVM script is running through the whole tvm compilation pipeline for each workload.

As I commented in #9737 (comment), we don't invoke the tvm pipeline when we select cutlass kernels.

One major difference with two scripts are that cutlass compiles all kernels into one giant profiler executable, while we generate separate executables for each kernel. So cutlass can allocate / deallocate memory once and loop through each kernel for a given workload to select the best one. Also, I remember that there is a non-trivial initialization cost (close to 1 sec) for any CUDA apps, when we invoke the first CUDA API call - for cutlass_profiler this happens only once while we pay that cost for each profiler binary (there about 60 of them).

But this still doesn't explain 10x difference, so I believe there is something else going on. We could adopt the same approach as cutlass_profiler and compile all candidate kernels into one executable. I didn't do that for conv2d_profiler because I just followed how gemm_profiler is implemented, but that could be a possible improvement.

[comaniac]

LGTM. Thanks.
I don't have the preference of using the current approach or TVM timer. It seems to me that the current implementation is simple enough and could keep the CUTLASS implementation standalone, but reusing TVM timer is also a fair concern.

[comaniac]

I raised this topic before in the GEMM profiler PR, but I agreed with @masahi that it seems not much to share and CUTLASS basically only supports GEMM and Conv2D. Accordingly, it might be a bit overkill to have a common base class at least for now.

[comaniac]

IIUC, now you changed evaluate to return float("inf") when invalid. Then the fist invalid kernel will be selected since float("inf") > 0, right?

[masahi]

oops you are right, changed to out < float("inf")