Step 54: Simple prompt caching — hash-based prefix cache for multi-turn conversations (#54) #90

Summary

Hash-based prefix cache for multi-turn conversations. Keep the KV-cache alive between calls, match the new prompt's token prefix against what's already cached, and only prefill the new suffix. Dramatically reduces TTFT for multi-turn chat.

Acceptance Criteria

• PrefixCache class with LRU eviction and configurable max entries
• PrefixCacheEntry storing live KV-cache + token prefix
• SimpleKvCache.SetCurrentLength() to truncate visible cache length
• TextGenerator accepts optional PrefixCache, skips prefill for cached prefix
• InferenceTimings.CachedTokenCount reports cached token stats
• Enabled by default in chat CLI and serve commands
• --no-prompt-cache flag to disable, --prompt-cache-size N to configure max entries
• /clear in chat CLI resets the prefix cache
• TimingsDto.CachedTokens in API response
• Chat UI displays cached token count in stats bar
• Unit tests for PrefixCache (match, eviction, clear)
• CPU SimpleKvCache only for V1 (QuantizedKvCache/GPU fall back to no caching)

Key Design Decision

Live KV-cache reuse (not snapshots): The KV-cache object persists across calls. No memcpy on cache hit — just truncate CurrentLength and prefill only suffix tokens. Ownership transfers to PrefixCache after generation, disposed on eviction.

Dependencies

• Step 7: Simple KV-cache ✅
• Step 34: ASP.NET server ✅

Key Files

• src/DotLLM.Engine/PromptCache/PrefixCache.cs (new)
• src/DotLLM.Engine/PromptCache/PrefixCacheEntry.cs (new)
• src/DotLLM.Engine/TextGenerator.cs (modify)
• src/DotLLM.Engine/KvCache/SimpleKvCache.cs (modify)
• src/DotLLM.Engine/InferenceTimings.cs (modify)
• src/DotLLM.Cli/Commands/ChatCommand.cs (modify)
• src/DotLLM.Cli/Commands/ServeCommand.cs (modify)
• src/DotLLM.Server/ (modify: ServerOptions, ServerState, ServerStartup, TimingsDto, endpoints, UI)