[LLM Runtime] Enable Mistral-7b

[intellinjun]

Type of Change

feature or bug fix or documentation or others
API changed or not:not
huggingface models:

• Mistral-7B

Description

detail description
JIRA ticket: 917
TODO

• n_head_kv
• extension tests

Expected Behavior & Potential Risk

the expected behavior that triggered by this PR

How has this PR been tested?

how to reproduce the test (including hardware information)

Dependency Change?

any library dependency introduced or removed

[intellinjun]

[zhenwei-intel]

Do we support sliding window attention (SWA) ?
https://mistral.ai/news/announcing-mistral-7b/

[hshen14]

Do we support sliding window attention (SWA) ? https://mistral.ai/news/announcing-mistral-7b/

Talked with Jun, SWA is not supported yet since the code actually reuses the llama.cpp without SWA support. We can enable StreamingLLM on this model through a separate PR.

[zhenwei-intel]

Do we support sliding window attention (SWA) ? https://mistral.ai/news/announcing-mistral-7b/

Talked with Jun, SWA is not supported yet since the code actually reuses the llama.cpp without SWA support. We can enable StreamingLLM on this model through a separate PR.

The StreamingLLM function is already available, just specify n_ keep=4 and n_discard=-1

[intellinjun]

without MHA fusion, use gcc version=12, if use gcc version=13 will go to MHA fusion(mistral-7b use group query attention same as llama2-70b, it is unsupport in llama.cpp now, next week will support gqa fusion)