-
Notifications
You must be signed in to change notification settings - Fork 3k
Description
** Please make sure you read the contribution guide and file the issues in the right place. **
Contribution guide.
Describe the bug
When running an AgentEvaluator.evaluate test using pytest-asyncio, the evaluation consistently fails during the teardown phase with a RuntimeError: Attempted to exit cancel scope in a different task than it was entered in. This error originates from anyio within the mcp.client.stdio.stdio_client asynchronous generator, indicating a problem with how asynchronous resources and their cancellation scopes are managed and shut down by the adk framework. The agent under test is configured to use MCPToolset with StdioServerParameters for its communication.
To Reproduce
Steps to reproduce the behavior:
- Install necessary packages:
mcp
sse-starlette
starlette
google-adk
google-adk[eval]
pytest
pytest-asyncio
- Create the agent definition file
import logging
from google.adk.agents.llm_agent import LlmAgent
from google.adk.tools.mcp_tool.mcp_session_manager import StdioServerParameters
from google.adk.tools.mcp_tool.mcp_toolset import MCPToolset
PATH_TO_MCP_SERVER = "shadowblade/mcp_server.py"
root_agent = LlmAgent(
model="gemini-2.5-flash",
# Agent name and detailed instructions omitted for brevity, but agent is standard LlmAgent
tools=[
MCPToolset(
connection_params=StdioServerParameters(
command="python3",
args=[PATH_TO_MCP_SERVER],
)
)
],
)- Create a pytest test file
from google.adk.evaluation.agent_evaluator import AgentEvaluator
import pytest
@pytest.mark.asyncio
async def test_with_single_test_file():
"""Test the agent's basic ability via a session file."""
evaluation_result = await AgentEvaluator.evaluate(
agent_module="[AGENT_MODULE_NAME]",
eval_dataset_file_path_or_dir="[AGENT_MODULE_NAME]/the_static.evalset.json",
)
pytest test_agent_evaluation.py
Alternatively, the error is also observed when running adk eval directly, confirming it's not specific to pytest's interaction but inherent to the adk teardown with MCPToolset
adk eval [AGENT_MODULE_NAME] [AGENT_MODULE_NAME]/sample.evalset.json --config_file_path [AGENT_MODULE_NAME]/test_config.json
Expected behavior
The test should complete successfully without any RuntimeError during the teardown phase, and the assert evaluation_result.passed should pass if the agent's evaluation was successful. All asynchronous resources, including the MCPToolset connections, should be cleanly closed by the adk framework.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
- Python version(python -V): 3.12.3
- ADK version 1.8
Model Information:
For example, which model is being used.
Additional context
The error points to anyio's cancel_scope being exited in a different task than it was entered, which is a common issue in asynchronous programming when tasks and their associated contexts are not managed carefully during shutdown. This suggests a potential issue with how adk or mcp manages asynchronous task lifecycles during agent communication and cleanup, specifically when MCPToolset is used with StdioServerParameters.
Full Error Traceback:
--------------------------------------------------------------- Captured log teardown -----------------------------------------------------------------
ERROR asyncio:base_events.py:1821 an error occurred during closing of asynchronous generator <async_generator object stdio_client at 0x79875f4b8d60>
asyncgen: <async_generator object stdio_client at 0x79875f4b8d60>
+ Exception Group Traceback (most recent call last):
| File "[YOUR_PROJECT_PATH]/env/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 772, in __aexit__
| raise BaseExceptionGroup(
| BaseExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "[YOUR_PROJECT_PATH]/env/lib/python3.12/site-packages/mcp/client/stdio/__init__.py", line 187, in stdio_client
| yield read_stream, write_stream
| GeneratorExit
+------------------------------------
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "[YOUR_PROJECT_PATH]/env/lib/python3.12/site-packages/mcp/client/stdio/__init__.py", line 180, in stdio_client
async with (
File "[YOUR_PROJECT_PATH]/env/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 778, in __aexit__
if self.cancel_scope.__exit__(type(exc), exc, exc.__traceback__):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "[YOUR_PROJECT_PATH]/env/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 457, in __exit__
raise RuntimeError(
RuntimeError: Attempted to exit cancel scope in a different task than it was entered in
=================================================================== warnings summary ===================================================================
[AGENT_MODULE_NAME]/test_the_static.py::test_with_single_test_file
[YOUR_PROJECT_PATH]/env/lib/python3.12/site-packages/google/adk/tools/mcp_tool/mcp_tool.py:87: UserWarning: [EXPERIMENTAL] BaseAuthenticatedTool: This feature is experimental and may change or be removed in future versions without notice. It may introduce breaking changes at any time.
super().__init__(
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=============================================================== short test summary info ================================================================
FAILED [AGENT_MODULE_NAME]/test_the_static.py::test_with_single_test_file - AssertionError: Following are all the test failures. If you looking to get more details on the failures, then please re-run this test with `print_d...
=============================================================================================================================== 1 passed, 1 warning in 37.37s ===============================================================================================================================