RoPE scaling support?

[nivibilla]

HF Merged RoPE scaling into their library. This allows to increase context length by 4x without retraining.

[WoosukKwon]

Hi @nivibilla, thanks for letting us know the integration! We are interested in the RoPe scaling. Will investigate it.

[nivibilla]

Thank you! Looking forward to it.

[lucasjinreal]

@WoosukKwon Same request here, another issue: #479

NTK could performance much more better then PI. It make vllm able to inference more than 16k without users finetune their models.

[nivibilla]

@lucasjinreal yes that's right. And I've seen your issue. Can you make a PR? I'd be happy to test it out!

[lucasjinreal]

@nivibilla I have pasted main modification in that issue. Please make some adoption to your vllm base. (my base has messy codes pr would be cubersome), feel free to ask me any question if you got a chance to try.

[nivibilla]

@lucasjinreal Sure, no problem. I will try it out

[kir152]

hey any update on this?

[viktor-ferenczi]

I will give it a try, because I have to learn into positional encoding anyway due to my planned work on #1161

For my reference, this former PR shows where the RoPE code is: #1004