Refactor: Split Chat Format Init/Parse into Separate Files

[aldehir]

Background Description

The common/chat.cpp and common/chat-parser.cpp files continue to grow as we add support for models. To help with maintainability, I recommend we use a registry to identify which init/parse functions to use for a given chat template. We can then move implementations into separate source files.

Possible Refactor Approaches

Change this:

static common_chat_params common_chat_templates_apply_jinja(
    const struct common_chat_templates        * tmpls,
    const struct common_chat_templates_inputs & inputs)
{
    // ...

    // DeepSeek V3.1: detect based on specific patterns in the template
    if (src.find("message['prefix'] is defined and message['prefix'] and thinking") != std::string::npos &&
        params.json_schema.is_null()) {
        return common_chat_params_init_deepseek_v3_1(tmpl, params);
    }

    // DeepSeek R1: use handler in all cases except json schema (thinking / tools).
    if (src.find("<｜tool▁calls▁begin｜>") != std::string::npos && params.json_schema.is_null()) {
        return common_chat_params_init_deepseek_r1(tmpl, params);
    }

    // Command R7B: : use handler in all cases except json schema (thinking / tools).
    if (src.find("<|END_THINKING|><|START_ACTION|>") != std::string::npos && params.json_schema.is_null()) {
        return common_chat_params_init_command_r7b(tmpl, params);
    }

    // ... 
}

static common_chat_params common_chat_params_init_deepseek_v3_1(const common_chat_template & tmpl, const struct templates_params & inputs) {
    // ...
}

static void common_chat_parse_deepseek_v3_1(common_chat_msg_parser & builder) {
    // ...
}

Into something like this:

common/chat-formats/
  deepseek.cpp
  command_r7b.cpp

static const auto _ = common_chat_format_register([](auto & handler) {
    handler.format = COMMON_CHAT_FORMAT_DEEPSEEK_V3_1;
    handler.priority = 50;

    // Define markers for efficient search by an Aho-Corasick Automaton
    handler.markers = {
        "message['prefix'] is defined and message['prefix'] and thinking",
    };

    handler.detect = [](const std::string & src, const templates_params & params) {
        // Any additional detection logic not covered by `markers`. This should only run if all defined markers are found.
        return params.json_schema.is_null();
    };

    handler.init = [](const common_chat_template & tmpl, const templates_params & inputs) {
        // ... deepseek v3.1 init ...
        return data;
    };

    handler.parse = [](const std::string & input, bool is_partial, const common_chat_syntax & syntax) {
        return parse_with_builder([](common_chat_msg_parser & builder) {
            // ... common_chat_parse_deepseek_v3_1  implementation moves here ...
        });
    };
});

Or through inheritance.

Benefits

• Reduce compilation times of incremental builds.
• Efficiently scan markers for all formats using an Aho-Corasick automaton, instead of multiple find() calls.
• Simplifies code review

[aldehir]

@pwilkin @ngxson thoughts?

[ngxson]

IMO each parser should be a subclass instead (polymorphism). This make it more extensible in the future.

[aldehir]

IMO each parser should be a subclass instead (polymorphism). This make it more extensible in the future.

That's fine too. I'm more so interested in extracting the logic into separate files.

However, right now the common API is not very conducive to a class-based interface.

// server-context.cpp:819
oai_parser_opt = {
    /* use_jinja             */ params_base.use_jinja,
    /* prefill_assistant     */ params_base.prefill_assistant,
    /* reasoning_format      */ params_base.reasoning_format,
    /* chat_template_kwargs  */ params_base.default_template_kwargs,
    /* common_chat_templates */ chat_templates.get(),
    /* allow_image           */ mctx ? mtmd_support_vision(mctx) : false,
    /* allow_audio           */ mctx ? mtmd_support_audio (mctx) : false,
    /* enable_thinking       */ enable_thinking,
    /* media_path            */ params_base.media_path,
};

// server-common:1027
auto chat_params = common_chat_templates_apply(opt.tmpls, inputs); // params get returned in a nlohmann::json object

// server-task.cpp:718
auto new_msg = common_chat_parse(
    generated_text,
    is_partial,
    oaicompat_chat_syntax);

Chat format detection happens in step 2 here. Seems like we would need to refactor the interface to pass an instance along.

[pwilkin]

Yes, this refactoring is very needed :) I will add more detailed thoughts later, but we might have to refactor the core logic for it because as you noticed it is right now very split.

[aldehir]

Not opposed, my snippet was just an example informed by maintaining the existing common API. It can still be maintained with classes, but it would give the illusion that state is preserved between init and parse. I'm happy to implement whatever is agreed upon.

[ngxson]

the common API for chat template / parsing is not very well-thought and quite complicated to follow, it's better to refactor them

Ideally we will have only 3 objects:

• common_chat --> responsible for deciding which template and parser to use. it has these members:
  • common_chat_formatter --> responsible for applying chat template, it calls minja internally
  • common_chat_parser --> responsible for parsing chat, this is base class and the impl will be selected via polymorphism

[pwilkin]

I'm working on an automatic parser generator that will extract the tool call markers and reasoning markers from the template, that way we can focus on just having dedicated parsers for the few templates / models that are too complicated to be automatically handled that way, but from what I've checked the vast majority of model templates can be automatized this way and then we can get rid of most of the parsing code.