[Draft] Add Response Caching Middleware #1845

strawgate · 2025-09-17T03:04:05Z

Description

Add a response caching middleware + in memory cache implementation + on disk cache implementation

cache list calls
cache call/read tool/prompt/resource
custom ttl by method
cache bust lists on update notifications

Todo:

Add ES implementation
Add redis implementation
Support max item size
support resource templates

Contributors Checklist

My change closes Response Caching Middleware #1844
I have followed the repository's development workflow
I have tested my changes manually and by adding relevant tests
I have performed all required documentation updates

Review Checklist

I have self-reviewed my changes
My Pull Request is ready for review

Copilot

Pull Request Overview

This PR implements a comprehensive response caching middleware system for FastMCP. It adds caching capabilities for various MCP operations including tool calls, resource reads, and prompt requests to improve performance and reduce server load.

Implements both in-memory and disk-based caching backends
Adds configurable TTL settings and filtering options for different operation types
Includes cache invalidation through MCP notifications and comprehensive test coverage

Reviewed Changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`tests/server/middleware/test_caching.py`	Comprehensive test suite covering cache implementations, middleware functionality, and integration tests
`src/fastmcp/server/middleware/middleware.py`	Updated middleware base class to use proper return type for resource reading operations
`src/fastmcp/server/middleware/caching.py`	Core caching middleware implementation with cache protocols, backends, and operation handlers
`pyproject.toml`	Added caching dependencies and test dependencies
`docs/servers/middleware.mdx`	Documentation for the new caching middleware functionality

src/fastmcp/server/middleware/caching.py

docs/servers/middleware.mdx

strawgate · 2025-09-18T01:59:07Z

TypedDicts are a lot less fun than I was expecting and I'm not sure the cache entry model is going to stick around

The implementation for the cache entries is mostly for the benefit of distributed cache implementations that can't just pickle

…ith docs

Copilot

Pull Request Overview

Copilot reviewed 8 out of 11 changed files in this pull request and generated 3 comments.

Copilot · 2025-09-18T18:49:05Z

tests/server/middleware/test_caching.py

+class TestResponseCachingMiddlewareIntegration:
+    """Integration tests with real FastMCP server."""
+
+    @pytest.fixture(params=["memory", "disk", "elasticsearch"])


The test fixture includes 'elasticsearch' as a parameter but the actual implementation in the fixture method only handles 'memory' and 'disk' cases, which will cause the tests to fail when 'elasticsearch' is selected.

Copilot · 2025-09-18T18:49:05Z

src/fastmcp/server/middleware/caching.py

+
+def get_size_of_value(value: CachableValueTypes) -> int:
+    """Get the size of a cache entry."""
+    if isinstance(value, BaseModel | ToolResult | ReadResourceContents):


The isinstance check uses union syntax with | but this will not work correctly. The condition should use a tuple: isinstance(value, (BaseModel, ToolResult, ReadResourceContents))

Suggested change

if isinstance(value, BaseModel | ToolResult | ReadResourceContents):

if isinstance(value, (BaseModel, ToolResult, ReadResourceContents)):

Copilot · 2025-09-18T18:49:06Z

src/fastmcp/contrib/middleware/caching/elasticsearch/elasticsearch_cache.py

+            body={
+                "query": {
+                    "range": {
+                        "expires_at": {"lt": datetime.now(tz=timezone.utc).timestamp()},


The comparison uses timestamp() but the mapping defines expires_at as a date type, not a numeric timestamp. This should use an ISO format string instead: datetime.now(tz=timezone.utc).isoformat()

Suggested change

"expires_at": {"lt": datetime.now(tz=timezone.utc).timestamp()},

"expires_at": {"lt": datetime.now(tz=timezone.utc).isoformat()},

jlowin · 2025-09-19T17:58:41Z

Love this idea! The CachedPrompt (and similar) classes feel a little overwrought -- I'm not totally clear on what makes the call_next results inherently uncacheable already? Or perhaps I'm missing something you're designing around

strawgate · 2025-09-19T18:03:31Z

Love this idea! The CachedPrompt (and similar) classes feel a little overwrought -- I'm not totally clear on what makes the call_next results inherently uncacheable already? Or perhaps I'm missing something you're designing around

Prompt cannot be instantiated directly as its an ABC and so what comes through the middleware is a FunctionPrompt which can't be serialized/deserialized due to the function reference -- so the CachedPrompt offers a serializable/deserializable model

Would love other ideas if you have them

jlowin · 2025-09-21T19:46:00Z

Oh! what do you think about removing the ABC from all components and just raising NotImplementedError() instead? Less self-documenting, more compatible?

Add initial implementation of response caching middleware

9a0c248

marvin-context-protocol bot added enhancement Improvement to existing functionality. For issues and smaller PR improvements. server Related to FastMCP server implementation or server-side functionality. labels Sep 17, 2025

strawgate added 2 commits September 16, 2025 22:23

Add tests for caching

daa0eb7

Add disk cache

09369b5

strawgate force-pushed the responsecachingmiddleware branch from 6850047 to 09369b5 Compare September 17, 2025 03:47

strawgate changed the title ~~Add Response Caching Middleware~~ [Draft] Add Response Caching Middleware Sep 17, 2025

Adding response caching with tests

84b3e0f

strawgate changed the title ~~[Draft] Add Response Caching Middleware~~ Add Response Caching Middleware Sep 18, 2025

strawgate changed the title ~~Add Response Caching Middleware~~ [Draft] Add Response Caching Middleware Sep 18, 2025

More progress

c600a16

strawgate requested a review from Copilot September 18, 2025 00:45

Add docs

d5f01da

strawgate force-pushed the responsecachingmiddleware branch from 03da107 to d5f01da Compare September 18, 2025 00:46

Copilot AI reviewed Sep 18, 2025

View reviewed changes

src/fastmcp/server/middleware/caching.py Outdated Show resolved Hide resolved

src/fastmcp/server/middleware/caching.py Outdated Show resolved Hide resolved

src/fastmcp/server/middleware/caching.py Outdated Show resolved Hide resolved

docs/servers/middleware.mdx Show resolved Hide resolved

strawgate mentioned this pull request Sep 18, 2025

as_proxy substantially slower #1583

Closed

strawgate added 2 commits September 18, 2025 08:20

PR Clean-up

00fbc97

Refactor cache, add Elasticsearch cache backend as a contrib module w…

a9113d0

…ith docs

strawgate self-assigned this Sep 18, 2025

strawgate requested a review from Copilot September 18, 2025 18:48

Copilot AI reviewed Sep 18, 2025

View reviewed changes

small fix for caching

e84f944

strawgate mentioned this pull request Sep 23, 2025

Add Redis support for the KVStorage protocol #1896

Open

marvin-context-protocol bot mentioned this pull request Oct 5, 2025

Add response limiting middleware #2004

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Draft] Add Response Caching Middleware #1845

[Draft] Add Response Caching Middleware #1845

Uh oh!

strawgate commented Sep 17, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

strawgate commented Sep 18, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Sep 18, 2025

Uh oh!

Copilot AI Sep 18, 2025

Uh oh!

Copilot AI Sep 18, 2025

Uh oh!

jlowin commented Sep 19, 2025

Uh oh!

strawgate commented Sep 19, 2025 •

edited

Loading

Uh oh!

jlowin commented Sep 21, 2025

Uh oh!

Uh oh!

	if isinstance(value, BaseModel \| ToolResult \| ReadResourceContents):
	if isinstance(value, (BaseModel, ToolResult, ReadResourceContents)):

	"expires_at": {"lt": datetime.now(tz=timezone.utc).timestamp()},
	"expires_at": {"lt": datetime.now(tz=timezone.utc).isoformat()},

[Draft] Add Response Caching Middleware #1845

Are you sure you want to change the base?

[Draft] Add Response Caching Middleware #1845

Uh oh!

Conversation

strawgate commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

strawgate commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

jlowin commented Sep 19, 2025

Uh oh!

strawgate commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jlowin commented Sep 21, 2025

Uh oh!

Uh oh!

strawgate commented Sep 17, 2025 •

edited

Loading

strawgate commented Sep 18, 2025 •

edited

Loading

strawgate commented Sep 19, 2025 •

edited

Loading