Allow user to define the string to concatenate the role name and prompt in DefaultHistoryTransform

[AsakusaRinne]

Currently in DefaultHistoryTransform we use ":" by defualt to connect the role name and the prompt. However in some language, ":" is not a commonly used string, for example, Chinese. In some other conditions, users may just want to use a self-defined connector. Therefore we'd better allow user to define the connector themselves.

[patrick-hovsepian]

I'm new to the space of LLMs and was trying to get Llama 3 to run. At first, I copied the chat example in the readme but got some random odd behavior like run off prompts and gibberish replies or replies to older queries interleaved with others -- all sorts of whack. I then tried running with phi-3 and got weirdness there too. I finally came across the example from #708 and things started working for Llama. This made me realize I needed a transform for phi3 as well and ended up creating one for it and things have been working much better.

All that's to say, is it worth it to provide a variant of DefaultHistoryTransform for different models or at least a best guess if possible and then fallback to the existing Default? I was looking a bit at the llama.cpp source and saw they had some checks to detect the absence of those special tokens and add them but maybe it's worthwhile to do it from here as a transformer?

I might be conflating a few issues or just using the tools incorrectly but figured I'd bring it up after a weekend of debugging as a total newcomer to the space 😅

[AsakusaRinne]

Yes, what you are talking about is often called chat template. The DefaultHistoryTransform is an old thing developed a year ago and we'll replace it with chat templates, which will have different template for different models. Thank you for your suggestions.

[patrick-hovsepian]

I'd be happy to help with development if possible. Let me know if there's any existing design or if you're interested in seeing a PR to attempt a first pass.

[AsakusaRinne]

HI, sorry for the late reply. A bit busy these days. It will be great if you'd like to contribute for it!

I haven't had a design of it yet but here're something I will consider.

1. In Llama Text Templater #715 a wrapper for the template in llama.cpp was introduced.
2. llama.cpp-python has a chat template which may help you as a reference.
3. Different templates are required by different models. So when doing this, it should be able to serve different formats while keep the API for users stable.

Please let me know if you need any help.

[github-actions]

This issue has been automatically marked as stale due to inactivity. If no further activity occurs, it will be closed in 7 days.