Invoke hipify from op builder for JIT extension builds

[jithunnair-amd]

cc @jeffra

[jithunnair-amd]

Here's all the hipified files that get copied to the DeepSpeed install directory with the updated manifest:

root@ff499528957c:/data/DeepSpeed# find /opt/conda/lib/python3.8/site-packages/deepspeed -name \*hip\*
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/adagrad/cpu_adagrad_hip.cpp
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/common/custom_hip_kernel.hip
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/quantization/quantizer.hip
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/quantization/pt_binding_hip.cpp
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/lamb/fused_lamb_hip_kernel.hip
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/transformer/softmax_kernels.hip
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/transformer/gelu_kernels.hip
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/transformer/general_kernels.hip
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/transformer/transform_kernels.hip
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/transformer/ds_transformer_hip.cpp
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/transformer/normalize_kernels.hip
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/transformer/cublas_wrappers.hip
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/normalize.hip
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/apply_rotary_pos_emb.hip
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/dequantize.hip
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/pt_binding_hip.cpp
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/gelu.hip
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/softmax.hip
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/transformer/inference/includes/cublas_wrappers_hip.h
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/transformer/inference/includes/custom_hip_layers.h
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/transformer/inference/includes/context_hip.h
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/transformer/dropout_kernels.hip
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/cublas_wrappers_hip.h
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/patch/hip
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/patch/hip/hcc_detail/hip_cooperative_groups_helper.h
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/patch/hip/hcc_detail/hip_cooperative_groups.h
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/general_kernels_hip.h
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/custom_hip_layers.h
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/dropout_hip.h
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/Timer_hip.h
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/softmax_hip.h
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/context_hip.h
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/cpu_adam_hip.h
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/normalize_layer_hip.h
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/gelu_hip.h
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/feed_forward_hip.h
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/type_shim_hip.h
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/strided_batch_gemm_hip.h
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/gemm_test_hip.h
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/cpu_adagrad_hip.h
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/ds_transformer_hip.h
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/adam/multi_tensor_adam.hip
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/adam/multi_tensor_apply_hip.cuh
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/adam/cpu_adam_hip.cpp

[jithunnair-amd]

This is the testcase I used:

import deepspeed
deepspeed.ops.op_builder.FusedAdamBuilder().load()
deepspeed.ops.op_builder.CPUAdamBuilder().load()
deepspeed.ops.op_builder.FusedLambBuilder().load()
deepspeed.ops.op_builder.TransformerBuilder().load()
deepspeed.ops.op_builder.StochasticTransformerBuilder().load()
deepspeed.ops.op_builder.UtilsBuilder().load()

Log file: DeepSpeed_JIT_build.log

The only files that seem to get hipified afresh are:

fpadmin@rocm-framework-12:~$ grep '\[ok]' ~/tmp/DeepSpeed_JIT_build.log
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/quantizer.h -> /opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/quantizer_hip.h [ok]
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/patch/hip/hcc_detail/hip_cooperative_groups_helper.h -> /opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/patch/hip/hcc_detail/hip_cooperative_groups_helper_hip.h [ok]
/opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/patch/hip/hcc_detail/hip_cooperative_groups.h -> /opt/conda/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/patch/hip/hcc_detail/hip_cooperative_groups_hip.h [ok]

The quantizer.h seems to be not included by any of the DeepSpeed source files, that's why it didn't get hipified originally.
As for the patched cooperative groups headers, they are never meant to be used from the patch dir anyway, instead being copied to the ROCm install path during DeepSpeed build.
In other words, even if these files run into a Permission Denied error, it shouldn't affect the user.

[jithunnair-amd]

@jeffra Rebased on master

[jeffra]

Thank you @jithunnair-amd!