Step 34: ASP.NET server — OpenAI-compatible API endpoints

[kkokosa]

Closes #84

Summary

• 6 API endpoints in DotLLM.Server: /v1/chat/completions (SSE streaming + non-streaming), /v1/completions, /v1/models, /v1/tokenize, /v1/detokenize, plus /health and /ready probes
• Full OpenAI API compatibility: request/response DTOs with [JsonPropertyName] snake_case, streaming via Server-Sent Events, finish_reason, usage token counts, tool call detection
• Chat template + tool calling: formats prompt via model's Jinja template, detects tool calls via IToolCallParser, returns finish_reason: "tool_calls" with structured tool_calls array
• Constrained decoding: response_format support for json_object and json_schema
• Testing: scripts/test_server.py (10/10 automated smoke tests), samples/server-api.http (VS Code REST Client)

Technical Details

The server loads a single model at startup via --model/--device/--quant CLI args (same pattern as RunCommand/ChatCommand). Model, tokenizer, chat template, and TextGenerator are registered as DI singletons.

Sequential request processing via SemaphoreSlim(1,1) in ServerState — Step 35 (continuous batching) will replace this with IScheduler. CORS is enabled permissively for Step 53 (Chat UI).

RequestConverter handles all DTO ↔ engine type conversions: ChatMessageDto[] → ChatMessage[], ToolDefinitionDto[] → ToolDefinition[], tool_choice JSON → ToolChoice record, response_format JSON → ResponseFormat record.

Model path resolution is inlined (local .gguf path or ~/.dotllm/models/ cache lookup) to avoid a dependency on DotLLM.Cli's GgufFileResolver.

Key Files

File	Changes
src/DotLLM.Server/Program.cs	Model loading, DI registration, endpoint mapping
src/DotLLM.Server/Endpoints/ChatCompletionEndpoint.cs	/v1/chat/completions — streaming SSE + non-streaming, tool call detection
src/DotLLM.Server/Endpoints/CompletionEndpoint.cs	/v1/completions — raw prompt completion
src/DotLLM.Server/RequestConverter.cs	OpenAI DTOs ↔ engine types
src/DotLLM.Server/Models/	6 DTO files for requests/responses/chunks
src/DotLLM.Server/ServerState.cs	Concurrency control, readiness
src/DotLLM.Server/ServerOptions.cs	CLI arg parsing
scripts/test_server.py	10 automated smoke tests
samples/server-api.http	VS Code REST Client test file

Test Plan

• Solution builds: dotnet build
• Unit tests pass (992/992): dotnet test tests/DotLLM.Tests.Unit/
• Server smoke tests (10/10): python scripts/test_server.py
• Manual test: streaming, non-streaming, tools, JSON mode, tokenize
• OpenAI Python SDK compatibility: openai.ChatCompletion.create(base_url=...)

🤖 Generated with Claude Code

[kkokosa]

@gemini please review this PR thoroughly - performance issues and possible improvements, potential bugs, code quality and architecture

[github-actions]

Gemini ✦

Overall, this is a solid implementation of the OpenAI API surface. The architecture cleanly separates the DTOs from the engine types, and the SSE streaming implementation is standard.

However, there are a few performance improvements that can be made to reduce GC pressure, particularly on the streaming hot path where allocations per token can quickly add up.

Performance & Allocations

1. Avoid string allocations in SSE streaming
In ChatCompletionEndpoint.cs and CompletionEndpoint.cs, the SSE chunks are serialized to a string and then interpolated:

string json = JsonSerializer.Serialize(chunk, JsonOptions);
await ctx.Response.WriteAsync($"data: {json}\n\n", ct);

This allocates multiple strings per generated token. You can eliminate these allocations by writing directly to the response body stream:

private static async Task WriteSseChunk<T>(HttpContext ctx, T chunk, CancellationToken ct)
{
    await ctx.Response.WriteAsync("data: ", ct);
    await JsonSerializer.SerializeAsync(ctx.Response.Body, chunk, JsonOptions, ct);
    await ctx.Response.WriteAsync("\n\n", ct);
    await ctx.Response.Body.FlushAsync(ct);
}

2. Cache static stop sequences
In ChatCompletionEndpoint.cs, BuildStopSequences allocates a new array and a new List<string> on every request:

private static List<string> BuildStopSequences(ServerState state)
{
    var stops = new List<string>();
    foreach (var marker in new[] { "<|im_end|>", "<|eot_id|>", "<|eom_id|>", "<|end|>", "</s>", "</tool_call>" })
        stops.Add(marker);
    return stops;
}

Change this to a private static readonly string[] CommonStopSequences = [ ... ]; to avoid per-request allocations.

3. Use JSON Source Generators
Since dotLLM prioritizes low GC pressure and high performance, consider using System.Text.Json Source Generators for the DTOs in DotLLM.Server.Models. Reflection-based serialization incurs overhead and allocates more memory.

[JsonSerializable(typeof(ChatCompletionRequest))]
[JsonSerializable(typeof(ChatCompletionResponse))]
[JsonSerializable(typeof(ChatCompletionChunk))]
// ...
internal partial class ServerJsonContext : JsonSerializerContext { }

Then pass ServerJsonContext.Default to your JsonSerializerOptions.

Code Quality

• LINQ in Tokenize Endpoint: In TokenizeEndpoint.cs, tokens.Select(t => tokenizer.DecodeToken(t)).ToArray() is fine for a utility endpoint, but if you want to maintain the zero-allocation philosophy, a simple for loop over the pre-sized array avoids the LINQ enumerator allocation.
• Cancellation Tokens: You are correctly passing HttpContext.RequestAborted down to the generator. Ensure that generator.GenerateStreamingTokensAsync and generator.Generate properly observe this token to halt inference immediately if the client disconnects.

[dotllm-claude-code-bot]

All three performance suggestions addressed in c866b38:

1. SSE string allocations — Fixed

WriteSseChunk now uses JsonSerializer.SerializeAsync directly to the response body stream. No intermediate string allocation per token.

2. Cache static stop sequences — Fixed

Replaced per-request List<string> + new[] with private static readonly string[] CommonStopSequences.

3. JSON Source Generators — Fixed

Added ServerJsonContext with [JsonSerializable] for all 11 DTO types + [JsonSourceGenerationOptions] for snake_case naming. All serialization calls now use the source-generated context (e.g., ServerJsonContext.Default.ChatCompletionChunk). Zero reflection overhead, AOT-compatible.

4. LINQ in Tokenize — Skipped

Utility endpoint called rarely. One enumerator + one array is acceptable for readability.

5. Cancellation tokens — Already handled

GenerateStreamingTokensAsync checks ThrowIfCancellationRequested() at the top of each decode step (TextGenerator.cs:368). No fix needed.

All 10 server smoke tests still passing.