[ToDo]: V0.1.5 Iteration Plan

[iofu728]

V0.1.5 Iteration Plan

New Model Support

• meta-llama/Meta-Llama-3.1-8B-Instruct
  • Supported in Feature(MInference): support llama 3.1 #54

Feature Support

• Remove the pycuda dependency; [Question]: Question about KV-cache storage #20
  • Supported in Hotfix(MInference): fix the import warnings, fix the apply_rotary_pos… #30
• Change the flash_attn dependency to optional; [Question]: Is A6000 supported? #23 @liyucheng09
• Add unittest. @liyucheng09
  • Supported in Feature(MInference): add unittest #31
• Support multi-gpu [Question]: RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cuda:1) #25;
  • Supported in Hotfix(MInference): fix the import warnings, fix the apply_rotary_pos… #30
• Add end-to-end benchmark script using vLLM [Question]: MInference Pre filling is slower than the vllm original version #18;
  • Supported in Feature(MInference): add e2e benchmark using vllm #49

Bugfix

• Fix the apply_rotary_pos_emb_single function. [Question]: RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cuda:1) #25.
  • Fixed in Hotfix(MInference): fix the import warnings, fix the apply_rotary_pos… #30
• Fix the import warning; [Bug]: missing warnings import in setup.py #28
  • Fixed in Hotfix(MInference): fix the import warnings, fix the apply_rotary_pos… #30
• Fix the vLLM >= 0.4.1; [Bug]: NameError: name 'cache_ops' is not defined #42
  • Fixed in Hotfix(MInference): fix vllm>=0.4.1 #44
• Fix the is_flash_attn_2_available issue;
  • Fixed in Feature(MInference): support llama 3.1 #54