using bigdl-llm fused rope for llama #9066
Conversation
| q_embed = torch.empty(query_states.shape, dtype=query_states.dtype, device=query_states.device) | ||
| k_embed = torch.empty(key_states.shape, dtype=key_states.dtype, device=key_states.device) | ||
|
|
||
| linear_q4_0.apply_rotary_embedding_half_qk(query_states, key_states, position_ids, q_embed, k_embed) |
There was a problem hiding this comment.
this only works on GPU, yes?
There was a problem hiding this comment.
I think cpu ipex has a similar function.
There was a problem hiding this comment.
I think cpu ipex has a similar function.
We can add our GPU optimizations first
|
I think I am going to separate rms_norm, rope and other changes into different PRs with rms_norm being the first. |
yangw1234
changed the title
|
performance updated here https://github.com/analytics-zoo/nano/issues/606 @jason-dai would you mind reviewing again? |
|
These optimizations does not work for training. Should we check if it is in the training mode in every place? |
added checking for now |
* optimize llama xpu rope * fix bug * fix style * refine append cache * remove check * do not cache cos sin * remove unnecessary changes * clean up * fix style * check for training
Description
using bigdl-llm fused rope for llama to reduce generation latency