When my MCP server needs to deploy multiple nodes, how should I handle our load balancing?

[loong-coder]

WebMvcSseServerTransportProvider.sessions is a private variable. When my MCP service has multiple identical nodes, requests to mcp/message might be routed to machines that don't hold the client connection sessions. How should this problem be resolved?

[SightStudio]

I'm experiencing the same issue. Currently, when using Spring AI in a remote MCP environment,
It's not possible to scale out the server.

To address this, I'm working on creating a custom implementation that extends McpServerTransportProvider.
But this is a very critical issue. when servering in remote mcp

public class MyWebMvcSseServerTransportProvider implements McpServerTransportProvider {

}

and then when you using spring ai

@Configuration
public class CustomMvcSseConfig {

    @Bean
    public MyWebMvcSseServerTransportProvider webMvcSseServerTransportProvider(
            ObjectProvider<ObjectMapper> objectMapperProvider, McpServerProperties serverProperties
    ) {
        ObjectMapper objectMapper = objectMapperProvider.getIfAvailable(ObjectMapper::new);
        return new MyWebMvcSseServerTransportProvider(objectMapper, serverProperties.getBaseUrl(),
                serverProperties.getSseMessageEndpoint(), serverProperties.getSseEndpoint());
    }

    @Bean
    public RouterFunction<ServerResponse> mvcMcpRouterFunction(MyWebMvcSseServerTransportProvider transportProvider) {
        return transportProvider.getRouterFunction();
    }
}

cc @tzolov @chemicL

[SightStudio]

@loong-coder

As someone who has spent more time trying to work around this, I can confirm that there's also this issue:

WebMvcSseServerTransportProvider causes Tomcat worker thread to hang #342

[loong-coder]

@loong-coder

As someone who has spent more time trying to work around this, I can confirm that there's also this issue:

WebMvcSseServerTransportProvider causes Tomcat worker thread to hang #342

I plan to store the sessionId in Redis. When an mcp/message request carries a sessionId that is not on the current machine, I will use Redis to broadcast this event. Eventually, the machine where the sessionId resides will consume the message and handle the request.

[SightStudio]

@loong-coder I'm using a similar approach to work around the issue

[Zizo-Vi]

Maybe u can use session sticky load balance policy.