LLM response caching for tests

[elroy-bot]

I would like to have a cached version of the LLM client, which writes output to local files within test flows. This implies converting the existing client into a class that would need to be initialized and incorporated into ElroyContext

The test llm client should still be able to call real endpoints like it does currently, but it should write responses to a fixtures dir in the test dir. This can be checked into version control (not automated, user can do this), so remote runs can just used cached data rather than re-running llm queries.

This caching mechanism should ONLY affect tests, there should not be caching that affects production

The cache should be written in json files, in a test/fixtures/llm_cache dir

[elroy-bot]

@claude implement

[claude]

Claude finished @elroy-bot's task —— View job • claude/issue-484-20250903-2251 • Create PR ➔

---

LLM Response Caching Implementation ✅

I have successfully implemented LLM response caching for tests as requested.

Todo List

• Read repository setup instructions (CLAUDE.md)
• Explore current LLM client implementation
• Understand ElroyContext structure
• Convert LLM client to class-based architecture
• Create cached LLM client for tests
• Create test fixtures directory structure
• Integrate cached client with ElroyContext for tests
• Update test configuration to use cached client
• Create test for cached client functionality
• Create documentation for cache functionality
• Run linting/testing if available
• Commit and push changes

Implementation Summary

Key Components Created:

1. CachedLlmClient (elroy/llm/cached_client.py):

   • Extends the existing LlmClient class
   • Caches responses for all LLM methods: query_llm, get_embedding, generate_chat_completion_message, etc.
   • Uses MD5 hashing of parameters to create consistent cache keys
   • Stores cache files as JSON in tests/fixtures/llm_cache/

2. MockStreamParser (elroy/llm/mock_stream_parser.py):

   • Replays cached streaming responses for generate_chat_completion_message
   • Maintains compatibility with existing code expecting StreamParser

3. Test Integration:

   • Added cached_ctx fixture to conftest.py for easy test usage
   • Updated george_ctx fixture to use cached responses
   • Created comprehensive tests in tests/llm/test_cached_client.py

4. ElroyContext Enhancement:

   • Added set_custom_llm_client() method to allow overriding the default LLM client
   • Maintains backward compatibility

Features

• Test-only caching: Only affects test environment, production code unchanged
• Offline testing: Tests can run without API access once responses are cached
• Deterministic results: Same inputs always produce same outputs
• CI/CD friendly: Cache files can be committed to version control
• Automatic cache directory creation: Creates tests/fixtures/llm_cache/ as needed
• Parameter-based caching: Different requests create separate cache entries

Usage

Tests can now use the cached_ctx fixture instead of ctx to get automatic LLM response caching:

def test_something(cached_ctx):
    response = cached_ctx.llm.query_llm("What is 2+2?", "You are helpful")
    # First run makes API call and caches response
    # Subsequent runs use cached response

All changes committed and pushed to the branch.

---