Support Lora Adapter generated from mistral-finetune

[tensimixt]

Feature request

Recent mistral models inlcuding mistral 7b v0.3 instruct have consolidated.safetensors which have different weights key names compared to what LoRAx expects. Also there are keys like lm_head, embed_tokens, layernorm and postattention_layernorm that vllm finds difficult to deal with.

Are you able to implement an update where a user who has generated a lora safetensors file from mistral-finetune can simply load this directly as a lora adapter into LoRAx and it just works instead of having to first try to map the weights to another weights key name convention as well as figuring out how to deal with unfamiliar keys such as layernorm and postattention_layernorm.

Motivation

Mistral-Finetune will become widely used so users who have generated lora safetensors from this should be able to simply plug and play their lora adapters into LoRAx.

Your contribution

I am happy to provide a lora safetensors if needed to help you understand the problem better.

[vgkavayah]

Is an update available for this issue?