Skip to content

Server: use llama_chat_apply_template to format the chat #5575

Closed
@ngxson

Description

@ngxson

Depends on #5538 to be merged

In #5425 , I mentioned that the chat template can be (ideally) detected using model metadata tokenizer.chat_template, but at that time, I didn't know that it is possible to access the metadata

Now that we have llama_chat_apply_template, we no longer have to worry about metadata. We can use this new function to format the chat supplied to /v1/chat/completions

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions