Step 16: Jinja2 chat template engine and dotllm chat REPL

[kkokosa]

Closes #30

Summary

• Jinja2-subset interpreter — three-stage pipeline (lexer → parser → evaluator) that parses chat templates from GGUF tokenizer.chat_template metadata and formats multi-turn conversations into prompt strings
• Special token pre-splitting in BpeTokenizer — control tokens (type 3) and user-defined tokens (type 6) are matched and emitted as single token IDs before BPE encoding, with EncodeSegment for continuation segments without SentencePiece BOS space prepend
• dotllm chat REPL — interactive multi-turn chat command with streaming output, system prompt, stop sequences, and per-turn timing

Jinja2 Subset

Covers the constructs used by real HuggingFace model templates (Llama 3.1/3.2, Qwen2/ChatML, Mistral, SmolLM):

Category	Features
Control flow	{% for %}, {% if/elif/else %}, {% set %}, {% set ns.attr %}
Expressions	String/int/bool/none literals, binary ops, comparisons, not/and/or, ternary x if cond else y, in/not in
Access	obj.attr, obj['key'], list[idx], list[start:stop] slicing
Literals	Lists [a, b], dicts {'key': value}
Filters	trim, tojson, length, default, join, replace, upper, lower, first, last, string, list, items, selectattr, map
Tests	is defined, is not defined, is none, is not none, is mapping, is string, is sequence
Functions	raise_exception(), namespace(), range(), dict(), strftime_now()
Other	Whitespace control ({%-/-%}), comments {# #}, loop variable (index, index0, first, last, length), string methods (.strip(), .replace(), .split(), .startswith()), keyword filter args (tojson(indent=4))

Architecture

Template string → JinjaLexer → tokens → JinjaParser → AST → JinjaEvaluator → output string
                  (parse once at construction)              (evaluate per Apply() call)

• JinjaLexer — two-mode lexer (text vs tag), brace depth tracking for dict literals inside {{ }}
• JinjaParser — recursive-descent with precedence climbing, normalizes msg['role'] to member access
• JinjaEvaluator — tree-walking evaluator with scope chain, Python-style truthiness, Undefined sentinel
• JinjaChatTemplate — public IChatTemplate impl, parses once in constructor, evaluates per Apply() call
• GgufChatTemplateFactory — bridge from GGUF metadata + tokenizer → JinjaChatTemplate

New Files

Path	Purpose
src/DotLLM.Tokenizers/ChatTemplates/JinjaException.cs	Exception with line/column info
src/DotLLM.Tokenizers/ChatTemplates/JinjaToken.cs	Token types enum + token record
src/DotLLM.Tokenizers/ChatTemplates/JinjaLexer.cs	Template → token list
src/DotLLM.Tokenizers/ChatTemplates/JinjaAst.cs	All AST node types
src/DotLLM.Tokenizers/ChatTemplates/JinjaParser.cs	Recursive-descent parser
src/DotLLM.Tokenizers/ChatTemplates/JinjaEvaluator.cs	Tree-walking evaluator
src/DotLLM.Tokenizers/ChatTemplates/JinjaChatTemplate.cs	Public IChatTemplate implementation
src/DotLLM.Models/Gguf/GgufChatTemplateFactory.cs	GGUF metadata → template bridge
src/DotLLM.Cli/Commands/ChatCommand.cs	Interactive chat REPL command

Modified Files

Path	Change
src/DotLLM.Tokenizers/Bpe/BpeTokenizer.cs	Special token pre-splitting (control/user-defined tokens emitted as single IDs)
src/DotLLM.Tokenizers/Bpe/BpeCore.cs	EncodeSegment default interface method on IBpeEncoding
src/DotLLM.Tokenizers/Bpe/SentencePieceEncoding.cs	EncodeSegment impl (no BOS space prepend)
src/DotLLM.Cli/Program.cs	Register chat command

Test Plan

• 123 unit tests — lexer (20), parser (25), evaluator (48), end-to-end chat template (30)
• 4 integration tests — SmolLM2-135M-Instruct (ChatML) and Llama-3.2-1B-Instruct (complex Llama template with dict literals, slicing, strftime_now)
• Known-answer tests for ChatML, Llama 3.1, and Mistral template formats
• Manual testing: dotllm chat bartowski/SmolLM2-135M-Instruct-GGUF and dotllm chat bartowski/Llama-3.2-1B-Instruct-GGUF

🤖 Generated with Claude Code

[kkokosa]

@codex please review this PR thoroughly - performance issues and possible improvements, potential bugs, code quality and architecture

[kkokosa]

@gemini please review this PR thoroughly - performance issues and possible improvements, potential bugs, code quality and architecture

[github-actions]

Gemini ✦

Overall, this is an excellent PR. The architecture for the Jinja2 subset (Lexer → Parser → Evaluator) is clean, standard, and easy to maintain. The approach to pre-splitting special tokens before BPE encoding is robust and correctly handles the longest-match-first requirement.

While the chat template evaluation and REPL are not on the strict token-generation hot path, there are a few areas where we can reduce GC pressure and fix potential edge cases.

Here is my review broken down by category:

🚀 Performance Improvements

1. Tokenizer Segment Allocations (src/DotLLM.Tokenizers/Bpe/BpeTokenizer.cs)
In EncodeWithSpecialTokens, you are allocating a new string for every text segment between special tokens:

string segment = text[pos..nextSpecialPos];

Since IBpeEncoding is internal, you can change Encode and EncodeSegment to accept ReadOnlySpan<char> instead of string. This would allow you to pass the sliced span directly to the underlying BPE encoder, completely eliminating string allocations during special token pre-splitting.

2. Jinja Evaluator Scope Allocations (src/DotLLM.Tokenizers/ChatTemplates/JinjaEvaluator.cs)
PushScope() allocates a new Dictionary every time it is called:

private void PushScope() => _scopes.Add(new Dictionary<string, object?>());

Because PushScope() is called on every iteration of a {% for %} loop, this will generate significant GC garbage for templates that iterate over many items (e.g., long message histories or tool call lists).
Fix: Use a pre-allocated pool or a reusable list of dictionaries with a _scopeDepth pointer. When pushing, if a dictionary exists at _scopeDepth, just .Clear() it and reuse it.

3. Fast-path in FindNextSpecialToken
The nested loop in FindNextSpecialToken checks SequenceEqual at every character position. You can speed this up by checking if the current character matches the first character of any special token before slicing and calling SequenceEqual.

🐛 Potential Bugs & Edge Cases

1. range() Function Exception (JinjaEvaluator.cs, line 285)

2 => Enumerable.Range(ToInt(args[0]), ToInt(args[1]) - ToInt(args[0])).Cast<object?>().ToList(),

If the template calls range(5, 2), ToInt(args[1]) - ToInt(args[0]) will be negative, causing Enumerable.Range to throw an ArgumentOutOfRangeException. Jinja2 normally returns an empty list in this scenario.
Fix: Wrap the count in Math.Max(0, count):

int count = Math.Max(0, ToInt(args[1]) - ToInt(args[0]));
return Enumerable.Range(ToInt(args[0]), count).Cast<object?>().ToList();

2. Stop Sequence Stripping (src/DotLLM.Cli/Commands/ChatCommand.cs, line 245)

foreach (var seq in options.StopSequences) {
    if (assistantText.EndsWith(seq, StringComparison.Ordinal))
        assistantText = assistantText[..^seq.Length];
}

If a model outputs multiple overlapping stop sequences (rare, but happens with misconfigured models), stripping one might expose another, but the foreach loop only evaluates each sequence once. A while loop or a regex might be safer, though the current implementation is likely fine for a simple REPL.

🧹 Code Quality & Architecture

• AST Design: The JinjaAst.cs records are beautifully structured. Using records for AST nodes is exactly the right tool for the job in modern C#.
• Truthiness Checks: In JinjaEvaluator.IsTruthy, you check if (value is ICollection c) and then if (value is IList l). Since System.Collections.IList implements ICollection, the second check is redundant and can be safely removed.
• JSON Parsing: In JinjaChatTemplate.ParseJsonToDict, using using var doc = JsonDocument.Parse(json); and recursively converting it to standard dictionaries is a great way to bridge the gap between HuggingFace's JSON tool schemas and the Jinja evaluator without leaking JsonElement into the AST logic.
• Special Token Sorting: Sorting _specialTokens by descending length in BuildSpecialTokenTable is a subtle but critical detail for correctness (ensuring <|im_start|> matches before <|im_>). Great catch.

Overall, the architecture is solid and perfectly aligns with the project's goals. I recommend applying the Math.Max fix for the range() function to prevent crashes on malformed templates, and considering the ReadOnlySpan<char> tokenizer optimization if you want to maintain strict zero-allocation on the text processing path.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4f188fd239

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Use correct token-type IDs for special token matching

This condition hard-codes tokenTypes[i] == 3 || tokenTypes[i] == 6, which does not match the tokenizer type mapping documented in this file (CreateSentencePiece docs list user-defined as a different value), so user-defined chat markers can be skipped while unrelated token classes are scanned as “special.” In chat prompts that rely on added control markers (for stop strings like <|im_end|>/<|eot_id|>), that prevents pre-splitting into single IDs and can leave partial stop fragments in generated text while also adding unnecessary per-character scan overhead.

Useful? React with 👍 / 👎.

[dotllm-claude-code-bot]

Fixed in d78d7f2. Changed tokenTypes[i] == 6 → 4 to match the GGUF llama_token_type enum (CONTROL=3, USER_DEFINED=4, BYTE=6). Also updated all comments and docstrings to reflect the correct spec values.

[chatgpt-codex-connector]

Guard dispose calls against partial initialization

The finally block always disposes model and gguf, but both are initialized inside the status callback; if GgufFile.Open, tokenizer loading, or LlamaModel.LoadFromGguf throws before assignment completes, this finally path raises NullReferenceException and masks the real load failure. Wrapping these disposals with null checks avoids turning recoverable initialization errors into secondary crashes.

Useful? React with 👍 / 👎.

[dotllm-claude-code-bot]

Fixed in d78d7f2. Changed declarations to nullable (GgufFile? gguf = null) and the finally block now uses model?.Dispose() / gguf?.Dispose(). Added ! null-forgiving operators after the Status block where initialization is guaranteed complete.

[chatgpt-codex-connector]

Honor tojson filter arguments during evaluation

The tojson filter ignores all passed arguments, so templates that use tojson(indent=4) (common in tool-calling chat templates) are rendered in a different format than expected. That changes prompt shape from the trained template layout and can degrade tool-call prompting quality even though parsing succeeds.

Useful? React with 👍 / 👎.

[dotllm-claude-code-bot]

Fixed in d78d7f2. The tojson filter now accepts an optional indent argument. tojson(indent=4) produces properly indented JSON with newlines, matching Python/Jinja2 behavior. The serialization methods (SerializeDictToJson, SerializeListToJson) gained indent and depth parameters for recursive indentation. Compact output (no args) is unchanged.

[dotllm-claude-code-bot]

Thanks for the thorough review! Here's what was addressed in d78d7f2:

Fixed:

• range() crash — Wrapped in Math.Max(0, ...) so range(5, 2) returns [] instead of throwing.
• FindNextSpecialToken fast-path — Added first-char check before SequenceEqual to skip unnecessary span comparisons.
• Redundant IsTruthy check — Removed the dead IList branch since ICollection already covers it.

Deferred (out of scope for review fix):

• ReadOnlySpan<char> for Encode/EncodeSegment — Requires changing the IBpeEncoding interface signature, which is too invasive for a review fix commit.
• Scope pooling — Chat template evaluation is not on the inference hot path; the added complexity isn't justified yet.
• Stop sequence stripping loop — Current single-pass approach is correct for the REPL use case.