Mistral v0.3 Weight Loading

[huyiwen]

Failed to launch the sglang server:

Run command: python -m sglang.launch_server --model-path Mistral-7B-Instruct-v0.3 --port 30000

Error message: KeyError: 'layers.0.attention.wk.weight'

This issue is also reported and fixed in vLLM.

[zhyncs]

Hi @huyiwen May you try the latest version? I couldn't reproduce your issue on A100.

git clone https://github.com/sgl-project/sglang.git
cd sglang

pip install --upgrade pip
pip install -e "python[all]"

pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.3/

python3 -m sglang.launch_server --model mistralai/Mistral-7B-Instruct-v0.3

[zhyncs]

This issue has been closed because it cannot be reproduced in the latest version. Please feel free to reopen it if needed. Thanks.