-
-
Notifications
You must be signed in to change notification settings - Fork 863
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Using native chat_template from tokenizer config in chat_template
strategy
#1689
Comments
see truefoundry@5ba183d for rough implementation |
hi i'm trying to fine-tune gemma 2 for function calling and our tool definitions are in the system prompt but gemma chat template doesn't support system role, how can we update the chat_template to support system role?
|
Thinking on this, I would suggest two ways to generalize the current chat template stuff:
|
The code fix is I think as simple as extending this function to take in the full config object, check the two fields, and return the final template jinja. Then the few primary call paths would just take the string and naively insert it into the tokenizer. We could also embed the system prompt change for SFT into the function to consolidate |
Sure, I'll can make this change |
I'm actually testing the changes in a branch of mine: https://github.com/fozziethebeat/axolotl/tree/fozziethebeat-flexible-chat-template |
Oh, I didn't realize, just reading this comment 😅 |
that looks to be functionally equivalent! great. I'm a bit busy to tidy mine up for merging so if you can get yours merged we'll have a great answer |
👋 @fozziethebeat @chiragjn Just chiming in as I'm starting to get my hands dirty learning how to use I'm hoping to have essentially drop in replacement for the LORA fine tunes we make using |
The stuff being worked on in this request gives a bit more flexibility but for Llama3, you can most likely use this config and just change a few parts of the dataset. I would suggest looking at it and the dataset I made. You should see the relationship between the dataset config and how the dataset itself actually looks. I personally always load things up onto HuggingFace, so that's one of the easier ways to get things rolling but I know the jsonl reader works well, and it'd just require changing the dataset stanza a tiny bit |
@fozziethebeat thanks! Yeah, I actually found those examples yesterday. I can train on your dataset no problem, but I am having issues with my dataset, which includes a I added it as a role to the I suspect it is different from the chat template on the |
To include system you probably need to rewrite
to be
You actually can likely delete that whole roles section as what is above is the default and it should support |
I have a dataset, which has a system prompt, but its same for the entire dataset. |
Solved by #1970 |
🔖 Feature description
Currently when using
type: chat_template
we have to chose a chat template which are encoded in the axolotl codebase (defaults to chatml)https://github.com/OpenAccess-AI-Collective/axolotl/blob/a82a7115224b7aef14301387a11ad4729fd6ca52/src/axolotl/prompt_strategies/chat_template.py#L99-L102
https://github.com/OpenAccess-AI-Collective/axolotl/blob/a82a7115224b7aef14301387a11ad4729fd6ca52/src/axolotl/prompt_strategies/chat_template.py#L115-L126
https://github.com/OpenAccess-AI-Collective/axolotl/blob/a82a7115224b7aef14301387a11ad4729fd6ca52/src/axolotl/utils/chat_templates.py#L21-L29
I would like to use the tokenizer
tokenizer.chat_template
as that would help the model start from a familiar place.✔️ Solution
I would like to pass
chat_template
asNone
in which case it should be picked from tokenizer or raise error / fallback to chatml iftokenizer.chat_template
missingI can work on this, seems like a small change.
❓ Alternatives
No response
📝 Additional Context
No response
Acknowledgements
Fixed by #1970
The text was updated successfully, but these errors were encountered: