[Bug]: Running the punica lora on Qwen1.5 32B model encountered RuntimeError: No suitable kernel. h_in=64 h_out=3424 dtype=Float out_dtype=BFloat16

[victorzhz111]

Your current environment

3090 8 gpus
vllm 0.4.1

🐛 Describe the bug

Hello vllm team,

When I try to run qwen1.5 32B model with punica lora request with tensor parallel size = 8, I have encountered the following error:
punica_kernels.dispatch_bgmv_low_level(
(RayWorkerWrapper pid=1566630) ERROR 04-26 16:28:41 worker_base.py:157] RuntimeError: No suitable kernel. h_in=64 h_out=3424 dtype=Float out_dtype=BFloat16

I am using 3090 machine with 8 gpus and I wish to use all my gpus. Can you have any chance to take look at this issue I am facing? Thank you very much!

[jeejeelee]

FYI #3793 (comment)

getting the same issue with Qwen1.5-14B when enable lora, did you solve this problem?

[victorzhz111]

getting the same issue with Qwen1.5-14B when enable lora, did you solve this problem?

in bgmv_impl.cuh

1. add the following check at line 243: ''' feat_in % (vec_size * 4) == 0 ||); ''' to bypass the assert

2. add the following case at line line 284:
   '''
   else if constexpr (feat_in % (vec_size / 2 * 8) == 0) {
   constexpr int tx = 8;
   constexpr int ty = 4;

   dim3 nblks(feat_out, batch_size);
   dim3 nthrs(tx, ty);

   bgmv_shrink_kernel<feat_in, feat_out, vec_size / 2,
   vec_size * sizeof(in_T) / 2,
   vec_size * sizeof(W_T) / 2, tx, ty, tz>
   <<<nblks, nthrs, 0, stream>>>(Y, X, W, indicies, y_offset,
   full_y_size, num_layers, layer_idx,
   scale);
   }
   }
   '''

3. add the corresponding kernel at bgmv_config.h.

4. rebuild the vllm.

I am able to get qwen1.5 14B to run but the inference becomes slow.

[zhanghanweii]

FYI #3793 (comment)

FYI #3793 (comment)

你好，我是使用pip install vllm安装的vllm包，你说的vllm/blob/main/csrc/punica/bgmv/bgmv_config.h没有找到在哪

[wicsaax]

FYI #3793 (comment)

FYI #3793 (comment)

你好，我是使用pip install vllm安装的vllm包，你说的vllm/blob/main/csrc/punica/bgmv/bgmv_config.h没有找到在哪

在源码里，必须
用源码编译安装

[mgoin]

This should be resolved with the new landed Triton kernels #5036