Step 16: Chat template engine — Jinja2-subset interpreter #30

Roadmap Step

Step 14 — Chat template engine

Dependencies

• Depends on Step 4 (BPE tokenizer — ITokenizer, GGUF vocabulary loading)

Scope

• Jinja2-subset parser — tokenize and parse a subset of Jinja2 syntax: variable interpolation ({{ }}), for loops ({% for %}), conditionals ({% if/elif/else %}), string filters (trim, strip), raise_exception(), basic expressions and comparisons
• Template compiler — parse template string into an AST of template nodes (text, expression, for, if, etc.), then evaluate against a context dictionary
• IChatTemplate implementation — JinjaChatTemplate implementing Apply(messages, options) → string
• GGUF integration — load template from tokenizer.chat_template metadata key
• Context variables — messages (list of ChatMessage), add_generation_prompt (bool), bos_token, eos_token, special tokens
• Known formats — verify correct output for Llama 3, ChatML (Qwen), Mistral template formats
• Fallback — if no template in model metadata, use ChatML as configurable default with a warning

Key Files

• src/DotLLM.Tokenizers/ChatTemplates/JinjaLexer.cs (new)
• src/DotLLM.Tokenizers/ChatTemplates/JinjaParser.cs (new)
• src/DotLLM.Tokenizers/ChatTemplates/JinjaEvaluator.cs (new)
• src/DotLLM.Tokenizers/ChatTemplates/JinjaChatTemplate.cs (new — implements IChatTemplate)
• src/DotLLM.Tokenizers/IChatTemplate.cs (existing interface)
• src/DotLLM.Tokenizers/ChatMessage.cs (existing)
• src/DotLLM.Tokenizers/ChatTemplateOptions.cs (existing)

Tests

• tests/DotLLM.Tests.Unit/Tokenizers/ChatTemplates/JinjaLexerTests.cs
• tests/DotLLM.Tests.Unit/Tokenizers/ChatTemplates/JinjaParserTests.cs
• tests/DotLLM.Tests.Unit/Tokenizers/ChatTemplates/JinjaChatTemplateTests.cs
• Known-answer tests for Llama 3, ChatML, Mistral template formats
• Edge cases: empty system message, multi-turn conversation, add_generation_prompt true/false

References

• docs/TOKENIZERS.md — "Chat Template Engine" section
• HuggingFace tokenizer_config.json for real template strings
• Jinja2 documentation (subset only)

Acceptance Criteria

• Jinja2 lexer: tokenizes {{ }}, {% %}, {# #}, text literals, strings, identifiers, operators
• Jinja2 parser: produces AST for variable interpolation, for loops, if/elif/else, filters, expressions
• Jinja2 evaluator: evaluates AST against context dictionary, handles string operations, comparisons, method calls (.strip(), .role, array indexing)
• JinjaChatTemplate correctly formats Llama 3 template output
• JinjaChatTemplate correctly formats ChatML (Qwen) template output
• JinjaChatTemplate correctly formats Mistral template output
• add_generation_prompt appends assistant turn prefix when true
• Template loaded from GGUF tokenizer.chat_template metadata
• Fallback to ChatML with warning when no template found
• raise_exception() in template produces clear error
• Unit tests with known-answer verification for all supported formats