feat: Microsoft.Extensions.AI IChatClient adapter (DotLLM.Extensions.AI) (#327)

CLAUDE.md

@@ -146,6 +146,7 @@ dotLLM/
 | Diagnostics & interpretability | [docs/DIAGNOSTICS.md](docs/DIAGNOSTICS.md) | Hooks, logit lens, SAE |
 | Telemetry & observability | [docs/TELEMETRY.md](docs/TELEMETRY.md) | Metrics, request tracing |
 | Server & API | [docs/SERVER.md](docs/SERVER.md) | Endpoints, rate limiting, warm-up |
+| Microsoft.Extensions.AI / Agent Framework | [docs/EXTENSIONS_AI.md](docs/EXTENSIONS_AI.md) | `IChatClient` adapter, MAF integration, type mapping |
 | Warm-up | [docs/WARMUP.md](docs/WARMUP.md) | Startup warm-up, JIT compilation, CUDA warm-up |
 | GPU inference | [docs/GPU.md](docs/GPU.md) | GPU forward pass, weight loading, KV-cache, CLI |
 | CUDA backend | [docs/CUDA.md](docs/CUDA.md) | PTX architecture, P/Invoke, kernel conventions, build |

Directory.Packages.props

@@ -5,6 +5,7 @@
   <ItemGroup>
     <!-- Runtime -->
     <PackageVersion Include="System.Numerics.Tensors" Version="9.0.3" />
+    <PackageVersion Include="Microsoft.Extensions.AI.Abstractions" Version="10.7.0" />
     <!-- Testing -->
     <PackageVersion Include="xunit" Version="2.9.3" />
     <PackageVersion Include="xunit.runner.visualstudio" Version="3.0.2" />

README.md

@@ -37,6 +37,7 @@ dotLLM is a ground-up LLM inference engine for .NET — not a wrapper around lla
 
 ### Serving
 - **OpenAI-compatible API** — `/v1/chat/completions`, `/v1/completions`, tool calling, streaming via ASP.NET
+- **Microsoft Agent Framework ready** — native `Microsoft.Extensions.AI.IChatClient` adapter (`DotLLM.Extensions.AI`) — drop dotLLM into MAF agents in-process, no HTTP
 - **Paged KV-cache** — PagedAttention with block-level allocation, prefix caching, and copy-on-write
 - **Speculative decoding** — draft-verify-accept with KV-cache rollback (greedy mode today; non-greedy planned — see issue #121)
 - **Structured output** — FSM/PDA-based constrained decoding guaranteeing valid JSON, JSON Schema, regex, and grammar
@@ -585,6 +586,7 @@ dotLLM ships as a set of NuGet packages so you can reference only what you need
 | [`DotLLM.Tokenizers`](https://www.nuget.org/packages/DotLLM.Tokenizers) | BPE, SentencePiece, HuggingFace tokenizer.json, Jinja2-subset chat templates |
 | [`DotLLM.Engine`](https://www.nuget.org/packages/DotLLM.Engine) | Inference engine — KV-cache, scheduler, samplers, constrained decoding, speculative decoding |
 | [`DotLLM.Server`](https://www.nuget.org/packages/DotLLM.Server) | OpenAI-compatible HTTP server, tool calling, built-in chat UI |
+| [`DotLLM.Extensions.AI`](https://www.nuget.org/packages/DotLLM.Extensions.AI) | `Microsoft.Extensions.AI.IChatClient` adapter — native Microsoft Agent Framework / MEAI backend |
 | [`DotLLM.HuggingFace`](https://www.nuget.org/packages/DotLLM.HuggingFace) | HuggingFace Hub search and GGUF download/caching |
 | [`DotLLM.Diagnostics`](https://www.nuget.org/packages/DotLLM.Diagnostics) | Interpretability hooks — activation capture, logit lens, logprobs |
 | [`DotLLM.Telemetry`](https://www.nuget.org/packages/DotLLM.Telemetry) | Placeholder package; `System.Diagnostics.Metrics` counters and `Activity`-based tracing are planned (Phase 7 / Phase 9) |
@@ -672,6 +674,7 @@ Both modes transparently reuse the embedded chat UI assets if `serveUi: true`. T
 
 ## News
 
+- **2026-06** — **Microsoft Agent Framework compatibility** — new `DotLLM.Extensions.AI` package implementing `Microsoft.Extensions.AI.IChatClient` (`DotLLMChatClient`) over the dotLLM engine. Makes dotLLM a native, in-process backend for the Microsoft Agent Framework (`chatClient.CreateAIAgent(...)`) and the wider `Microsoft.Extensions.AI` ecosystem — streaming, tool calling (`FunctionCallContent`), structured output, and token usage all map onto the engine with no HTTP round-trip. AOT-clean, targets .NET 10, depends only on `Microsoft.Extensions.AI.Abstractions`. 22 unit tests; see `docs/EXTENSIONS_AI.md` ([#327](https://github.com/kkokosa/dotLLM/issues/327))
 - **2026-04** — **First public release (v0.1.0-preview.1)** — dotLLM goes public. [NuGet packages](#nuget-packages) for all 10 libraries + `DotLLM.Cli` as a global `dotnet tool`. Self-contained single-file downloads for Windows / Linux / macOS (Apple Silicon) and experimental Native AOT builds for Linux / Windows attached to every [GitHub Release](https://github.com/kkokosa/dotLLM/releases). Companion website at [dotllm.dev](https://dotllm.dev/) ([#119](https://github.com/kkokosa/dotLLM/issues/119))
 - **2026-04** — **Wave 7**: CPU performance cleanup pass — `TopKSampler` replaces full `Array.Sort` with a hand-rolled size-K min-heap (`O(N log K)`, stack-resident scratch); `JsonSchemaConstraint` adds first-char bucketing to skip the ~160 MB of struct clones per mask build when the tracker rejects most leading characters, plus LRU eviction instead of the previous full-flush cache overflow; `Dequantize.Q5_0` gains an AVX2 path matching Q8_0's throughput (reuses `MatMulQ5_0.ExtractQ5HighBits` / `vpshufb` bit-extraction); `BpeTokenizer` pre-splits special tokens via the existing `Trie.TryMatchLongest` instead of the O(n × m) linear scan; `ComputeThreadPool` now pins the caller (inference) thread to the first candidate P-core on first `Dispatch`, eliminating the hybrid-CPU stall where pinned P-core workers idled at the barrier waiting for an E-core caller. New BenchmarkDotNet suites for TopK sampling, schema mask build, and special-token encode ([#109](https://github.com/kkokosa/dotLLM/issues/109))
 - **2026-04** — **Phase 7 begins**: Logprobs — OpenAI-compatible `logprobs: true` + `top_logprobs: N` (0-20) on `/v1/chat/completions` and `/v1/completions`. Per-token log-softmax captured before sampling, returned in both streaming SSE chunks and non-streaming responses. Chat UI gains opt-in logprobs visualization: color-coded token confidence (green/lime/yellow/orange/red), hover tooltips with top-K alternatives and probabilities, diagnostic cues for low confidence, ambiguity, and sampling effect. `DotLLM.Sample.Logprobs` console sample with ANSI-colored output ([#101](https://github.com/kkokosa/dotLLM/issues/101))

docs/EXTENSIONS_AI.md

@@ -0,0 +1,89 @@
+# Microsoft.Extensions.AI integration — dotLLM
+
+`DotLLM.Extensions.AI` makes dotLLM a native, **in-process** backend for the
+**Microsoft Agent Framework** (MAF) and the broader `Microsoft.Extensions.AI`
+(MEAI) ecosystem by implementing **`IChatClient`** — the single interface that
+MAF agents, the function-invocation middleware, response caching, OpenTelemetry,
+and DI all consume.
+
+Implementation: `DotLLMChatClient` (the adapter) and `ChatClientMapping` (the
+pure type translation). The package depends only on `DotLLM.Engine` and
+`Microsoft.Extensions.AI.Abstractions`; it targets .NET 10 and adds no
+reflection-based serialization (AOT-clean).
+
+## Usage
+
+```csharp
+using DotLLM.Extensions.AI;
+using Microsoft.Extensions.AI;
+
+// `generator` (TextGenerator) and `chatTemplate` (IChatTemplate) come from
+// loading a model with the dotLLM engine; `toolCallParser` is optional.
+IChatClient client = new DotLLMChatClient(generator, chatTemplate, modelId: "phi-4-q4_k_m", toolCallParser);
+
+// Use directly as a Microsoft.Extensions.AI chat client...
+ChatResponse response = await client.GetResponseAsync("Tell me a joke.");
+
+// ...or as a Microsoft Agent Framework agent (zero glue):
+AIAgent agent = client.CreateAIAgent(instructions: "You are concise.", name: "Assistant");
+var reply = await agent.RunAsync("Summarize the plot of Hamlet.");
+
+await foreach (var update in client.GetStreamingResponseAsync("Stream this."))
+    Console.Write(update.Text);
+```
+
+## Integration paths
+
+| Path | How | When |
+|------|-----|------|
+| **Native `IChatClient`** (this package) | Wrap `TextGenerator` in-process — no HTTP, shares mmap'd weights, surfaces usage/timings directly | Embedded/edge .NET agent apps; highest fidelity & lowest latency |
+| **OpenAI-compatible HTTP** | Point `Microsoft.Extensions.AI.OpenAI` at dotLLM's `/v1/` base URL (`OpenAIClient ... .AsIChatClient()`) | Out-of-process / language-agnostic consumers (the server already exposes this) |
+
+Both expose the same `IChatClient` surface to MAF, so agent code is identical.
+
+## Mapping reference
+
+**Request (Microsoft.Extensions.AI → engine)**
+
+| MEAI | dotLLM |
+|------|--------|
+| `IEnumerable<ChatMessage>` (System/User/Assistant/Tool) | `ChatMessage[]` (chat-template rendered to a prompt) |
+| `TextContent` | concatenated message content |
+| `FunctionCallContent` | assistant `ToolCall` |
+| `FunctionResultContent` | `tool`-role message keyed by `CallId` |
+| `ChatOptions.{Temperature,TopP,TopK,MaxOutputTokens,StopSequences,Seed}` | `InferenceOptions` |
+| `ChatOptions.ResponseFormat` (Json / JSON-schema) | `ResponseFormat.JsonObject` / `JsonSchema` |
+| `ChatOptions.Tools` (`AIFunction`) | `ToolDefinition[]` (via the chat template) |
+| `AdditionalProperties["min_p" / "repetition_penalty"]` | `InferenceOptions.MinP` / `RepetitionPenalty` (dotLLM-specific knobs) |
+
+**Response (engine → Microsoft.Extensions.AI)**
+
+| dotLLM | MEAI |
+|--------|------|
+| generated text | `TextContent` |
+| detected `ToolCall`s | `FunctionCallContent` (text dropped, `FinishReason = ToolCalls`) |
+| `FinishReason` (Stop/Length/ToolCalls) | `ChatFinishReason` (Stop/Length/ToolCalls) |
+| prompt/generated token counts | `UsageDetails` (Input/Output/Total) |
+| streaming `GenerationToken` | `ChatResponseUpdate` (`text_delta`) |
+
+## Notes / limitations
+
+- The underlying `TextGenerator` is stateful and single-request; calls are
+  serialized through an internal gate (streaming holds it for the enumeration),
+  matching the dotLLM server's single-request model.
+- Streaming tool calls are detected post-generation (like the server endpoints),
+  so `FunctionCallContent` is emitted in the final update rather than incrementally.
+  Consequently, **when a tool-call parser is configured and the request carries tools,
+  text is buffered instead of streamed** and delivered in that same final update. Streaming
+  it would leak the model's raw tool-call syntax to the caller, and would make the coalesced
+  stream (`ToChatResponse()`) disagree with `GetResponseAsync`, which drops text when tool
+  calls are present. Requests without tools (or without a parser) stream token-by-token as
+  usual.
+- `GetResponseAsync` offloads the synchronous `TextGenerator.Generate` to the thread pool so
+  it does not block the caller. The `CancellationToken` therefore only gates scheduling; it
+  cannot abort a generation already in flight. Use `GetStreamingResponseAsync`, which
+  cancels cooperatively between decode steps, when mid-generation cancellation matters.
+- `ChatOptions.FrequencyPenalty`/`PresencePenalty` have no direct engine
+  equivalent and are not applied; use `RepetitionPenalty` via `AdditionalProperties`.
+- For automatic tool execution, wrap the client with MEAI's
+  `FunctionInvokingChatClient` — no custom loop needed.

docs/ROADMAP.md

@@ -97,6 +97,7 @@ Step 22 (done) ──────► Step 30 (NUMA + Spin-wait)
 | 41 | **Regex + CFG** :white_check_mark: | `RegexConstraint` (DFA-based) and `GrammarConstraint` (PDA, GBNF-style). | 39 |
 | 42 | **Tool calling** :white_check_mark: | `IToolCallParser`, chat template tool integration, structured output for function arguments. `finish_reason: "tool_calls"`. Parallel tool calls. | 16, 40 |
 | 34 | **ASP.NET server** :white_check_mark: | Minimal API endpoints: `/v1/chat/completions`, `/v1/completions`, `/v1/models`, `/v1/embeddings`, `/v1/tokenize`, `/v1/detokenize`. Health + readiness probes. | 16, 17 |
+| 34c | **Microsoft.Extensions.AI `IChatClient`** :white_check_mark: | `DotLLM.Extensions.AI` package — `DotLLMChatClient : IChatClient` over `TextGenerator`. Native, in-process Microsoft Agent Framework / MEAI backend (`chatClient.CreateAIAgent(...)`): streaming, tool calling (`FunctionCallContent`), structured output, usage. AOT-clean, depends only on `Microsoft.Extensions.AI.Abstractions`. See `docs/EXTENSIONS_AI.md`. | 16, 42 |
 | 53 | **Chat UI (`serve` command)** :white_check_mark: | Built-in web chat interface served by the ASP.NET host. `dotllm serve model.gguf` starts the API server and opens a browser to a bundled single-page chat UI. Streaming responses via SSE, model/parameter selection, conversation history. Similar to `ollama serve` + Open WebUI or `llama.cpp --server` built-in UI. | 34 |
 | 54 | **Simple prompt caching** :white_check_mark: | Hash-based prefix cache for multi-turn conversations. Hash the token prefix, store/reuse the KV-cache state. On cache hit, skip prefill for the matching prefix and only process new tokens. Works with existing `SimpleKvCache` — no paged attention required. LRU eviction with configurable max cached sessions. `--prompt-cache` CLI flag. Dramatically reduces TTFT for multi-turn chat. | 7, 34 |
 

dotLLM.slnx

@@ -8,6 +8,7 @@
     <File Path="docs/BENCHMARKS.md" />
     <File Path="docs/CONSTRAINED_DECODING.md" />
     <File Path="docs/DIAGNOSTICS.md" />
+    <File Path="docs/EXTENSIONS_AI.md" />
     <File Path="docs/GGUF_FORMAT.md" />
     <File Path="docs/KV_CACHE.md" />
     <File Path="docs/LORA.md" />
@@ -37,6 +38,7 @@
     <Project Path="src/DotLLM.Cuda/DotLLM.Cuda.csproj" />
     <Project Path="src/DotLLM.Diagnostics/DotLLM.Diagnostics.csproj" />
     <Project Path="src/DotLLM.Engine/DotLLM.Engine.csproj" />
+    <Project Path="src/DotLLM.Extensions.AI/DotLLM.Extensions.AI.csproj" />
     <Project Path="src/DotLLM.HuggingFace/DotLLM.HuggingFace.csproj" />
     <Project Path="src/DotLLM.Models/DotLLM.Models.csproj" />
     <Project Path="src/DotLLM.Server/DotLLM.Server.csproj" />