Users/robrandao/integration overhaul #300

rodrigobr-msft · 2026-02-01T14:44:39Z

No description provided.

…it tests

…rding

dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/aiohttp_callback_server.py

+            exchange = Exchange(error=str(e), response_at=response_at)
+            response = Response(
+                status=500,
+                text=str(e)


In general, to fix this type of issue you avoid sending raw exception details (messages, stack traces) to external clients. Instead you log or otherwise record the detailed error on the server side (for debugging) while returning a generic, non-sensitive error message in the HTTP response. This keeps useful diagnostics for developers while preventing attackers from learning about internals.

Here, the minimal change that preserves functionality is to keep recording the actual exception details in the Exchange object (so tests and transcripts can still see them), while changing the HTTP Response body from text=str(e) to a generic message such as "Internal server error" or "An internal error has occurred.". No additional imports are needed; we just adjust the response construction in the except block in AiohttpCallbackServer._handle_request.

Concretely, in dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/aiohttp_callback_server.py, in the except Exception as e: block, leave exchange = Exchange(error=str(e), response_at=response_at) intact, but change:

response = Response( status=500, text=str(e) )

to something like:

response = Response( status=500, content_type="application/json", text='{"message": "Internal server error"}', )

or a plain-text equivalent, depending on what is expected by the rest of the tests. This stops tainted exception text from flowing to the client while still recording it in the transcript.

Copilot

Pull request overview

This PR overhauls the microsoft-agents-testing developer tooling by replacing legacy integration/DDT/assertion utilities with a new scenario/transport-based core, plus a new CLI entry point.

Changes:

Introduces a new core architecture (Scenario/ClientFactory, transport layer with Transcript/Exchange, aiohttp Sender/CallbackServer).
Adds a click-based CLI (agt) with commands and predefined scenarios.
Removes legacy integration framework, data-driven tests, benchmark tooling, and setup.py-based packaging in favor of pyproject.toml.

Reviewed changes

Copilot reviewed 208 out of 253 changed files in this pull request and generated 15 comments.

Show a summary per file

File	Description
dev/microsoft-agents-testing/tests/core/fluent/backend/types/test_unset.py	Adds unit tests for the new `Unset` sentinel behavior.
dev/microsoft-agents-testing/tests/core/fluent/backend/types/test_readonly.py	Adds unit tests for the new `Readonly` mixin behavior.
dev/microsoft-agents-testing/tests/assertions/_common.py	Removes legacy test fixtures tied to the old assertions layer.
dev/microsoft-agents-testing/setup.py	Removes legacy setuptools packaging in favor of `pyproject.toml`.
dev/microsoft-agents-testing/pytest.ini	Adjusts warning filters for new aiohttp/pytest behaviors.
dev/microsoft-agents-testing/pyproject.toml	Defines runtime dependencies and adds `agt` console script entrypoint.
dev/microsoft-agents-testing/microsoft_agents/testing/utils/populate.py	Removes legacy activity-population helpers.
dev/microsoft-agents-testing/microsoft_agents/testing/utils/misc.py	Removes legacy misc helpers (URL parsing, model normalization).
dev/microsoft-agents-testing/microsoft_agents/testing/utils/init.py	Removes legacy `utils` package exports.
dev/microsoft-agents-testing/microsoft_agents/testing/utils.py	Adds new “quick interaction” helpers (`send`, `ex_send`) built on scenarios.
dev/microsoft-agents-testing/microsoft_agents/testing/transcript_logger.py	Adds transcript formatting/printing helpers used by the CLI.
dev/microsoft-agents-testing/microsoft_agents/testing/sdk_config.py	Removes legacy `SDKConfig` class (config now passed as dict).
dev/microsoft-agents-testing/microsoft_agents/testing/integration/data_driven/load_ddts.py	Removes legacy data-driven-test loader.
dev/microsoft-agents-testing/microsoft_agents/testing/integration/data_driven/ddt.py	Removes legacy DDT decorator system.
dev/microsoft-agents-testing/microsoft_agents/testing/integration/data_driven/data_driven_test.py	Removes legacy DDT runner.
dev/microsoft-agents-testing/microsoft_agents/testing/integration/data_driven/init.py	Removes legacy DDT exports.
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/sample.py	Removes legacy integration sample abstraction.
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/integration.py	Removes legacy pytest fixture-based integration harness.
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/environment.py	Removes legacy integration environment abstraction.
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/client/response_client.py	Removes legacy response server client.
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/client/auto_client.py	Removes unused legacy stub client.
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/client/agent_client.py	Removes legacy agent client implementation.
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/client/init.py	Removes legacy integration client exports.
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/application_runner.py	Removes legacy threaded server runner.
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/aiohttp/aiohttp_runner.py	Removes legacy aiohttp runner.
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/aiohttp/aiohttp_environment.py	Removes legacy aiohttp integration environment.
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/aiohttp/init.py	Removes legacy aiohttp integration exports.
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/init.py	Removes legacy integration exports.
dev/microsoft-agents-testing/microsoft_agents/testing/integration/init.py	Removes legacy integration public surface.
dev/microsoft-agents-testing/microsoft_agents/testing/core/utils.py	Adds core exchange helpers and config-based token generation; retains token generation.
dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/transcript/transcript.py	Adds new Transcript tree model for recording exchanges.
dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/transcript/exchange.py	Adds new Exchange model and response parsing helpers.
dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/transcript/init.py	Exposes Transcript/Exchange from the new transport layer.
dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/sender.py	Adds Sender interface for pluggable HTTP senders.
dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/callback_server.py	Adds CallbackServer interface for async callbacks.
dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/aiohttp_sender.py	Adds aiohttp Sender implementation.
dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/aiohttp_callback_server.py	Adds aiohttp TestServer-based callback server implementation.
dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/init.py	Exposes transport layer public API.
dev/microsoft-agents-testing/microsoft_agents/testing/core/scenario.py	Adds Scenario/ClientFactory abstractions and a convenience `client()` context manager.
dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/utils.py	Adds model normalization/flattening helpers for fluent APIs.
dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/types/unset.py	Adds `Unset` sentinel singleton implementation.
dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/types/readonly.py	Adds shared `Readonly` mixin.
dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/types/init.py	Exposes fluent backend “types” helpers.
dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/quantifier.py	Adds quantifier helpers (all/any/none/one/n).
dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/model_predicate.py	Adds predicate evaluation engine producing per-item match details.
dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/init.py	Exposes fluent backend internal building blocks.
dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/init.py	Exposes fluent API entry points.
dev/microsoft-agents-testing/microsoft_agents/testing/core/external_scenario.py	Adds ExternalScenario for testing against externally hosted agents.
dev/microsoft-agents-testing/microsoft_agents/testing/core/config.py	Adds ScenarioConfig/ClientConfig configuration dataclasses.
dev/microsoft-agents-testing/microsoft_agents/testing/core/_aiohttp_client_factory.py	Adds internal aiohttp-based client factory for scenarios.
dev/microsoft-agents-testing/microsoft_agents/testing/core/init.py	Re-exports core APIs for external consumption.
dev/microsoft-agents-testing/microsoft_agents/testing/cli/scenarios/basic_scenario.py	Adds a basic echo scenario for the CLI.
dev/microsoft-agents-testing/microsoft_agents/testing/cli/scenarios/auth_scenario.py	Adds an auth-testing scenario for the CLI.
dev/microsoft-agents-testing/microsoft_agents/testing/cli/scenarios/init.py	Registers CLI scenarios by name.
dev/microsoft-agents-testing/microsoft_agents/testing/cli/main.py	Adds the top-level CLI command group and initialization.
dev/microsoft-agents-testing/microsoft_agents/testing/cli/core/output.py	Adds reusable CLI output formatting utilities.
dev/microsoft-agents-testing/microsoft_agents/testing/cli/core/executor/thread_executor.py	Adds thread-based executor to run async tasks concurrently.
dev/microsoft-agents-testing/microsoft_agents/testing/cli/core/executor/executor.py	Adds base executor strategy for CLI operations.
dev/microsoft-agents-testing/microsoft_agents/testing/cli/core/executor/execution_result.py	Adds result container for CLI executions.
dev/microsoft-agents-testing/microsoft_agents/testing/cli/core/executor/coroutine_executor.py	Adds asyncio coroutine executor.
dev/microsoft-agents-testing/microsoft_agents/testing/cli/core/executor/init.py	Exposes executor utilities.
dev/microsoft-agents-testing/microsoft_agents/testing/cli/core/decorators.py	Adds `async_command` decorator for click commands.
dev/microsoft-agents-testing/microsoft_agents/testing/cli/core/init.py	Exposes CLI core utilities.
dev/microsoft-agents-testing/microsoft_agents/testing/cli/config.py	Adds CLI configuration loader for .env and process env.
dev/microsoft-agents-testing/microsoft_agents/testing/cli/commands/validate.py	Adds config validation command.
dev/microsoft-agents-testing/microsoft_agents/testing/cli/commands/run.py	Adds scenario runner command.
dev/microsoft-agents-testing/microsoft_agents/testing/cli/commands/post.py	Adds a command to post an activity payload to an agent.
dev/microsoft-agents-testing/microsoft_agents/testing/cli/commands/chat.py	Adds interactive chat REPL command.
dev/microsoft-agents-testing/microsoft_agents/testing/cli/commands/init.py	Registers CLI commands.
dev/microsoft-agents-testing/microsoft_agents/testing/cli/init.py	Exposes CLI entrypoint module.
dev/microsoft-agents-testing/microsoft_agents/testing/auth/init.py	Removes legacy auth module exports (token now in core utils).
dev/microsoft-agents-testing/microsoft_agents/testing/assertions/type_defs.py	Removes legacy assertion type definitions.
dev/microsoft-agents-testing/microsoft_agents/testing/assertions/model_selector.py	Removes legacy assertion model selector.
dev/microsoft-agents-testing/microsoft_agents/testing/assertions/model_assertion.py	Removes legacy model assertion implementation.
dev/microsoft-agents-testing/microsoft_agents/testing/assertions/check_model.py	Removes legacy deep model checker.
dev/microsoft-agents-testing/microsoft_agents/testing/assertions/check_field.py	Removes legacy field checker.
dev/microsoft-agents-testing/microsoft_agents/testing/assertions/assertions.py	Removes legacy assertion function wrappers.
dev/microsoft-agents-testing/microsoft_agents/testing/assertions/init.py	Removes legacy assertions exports.
dev/microsoft-agents-testing/microsoft_agents/testing/init.py	Updates package-level exports to new core/fluent/scenario APIs.
dev/microsoft-agents-testing/_manual_test/main.py	Removes manual test harness.
dev/microsoft-agents-testing/_manual_test/env.TEMPLATE	Removes manual test env template.
dev/integration/tests/test_expect_replies.py	Removes legacy integration test file(s).
dev/integration/tests/quickstart/test_quickstart_sample.py	Removes legacy quickstart integration tests.
dev/integration/tests/quickstart/directline/send_hi.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/quickstart/directline/send_hello.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/quickstart/directline/conversation_update.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/quickstart/directline/_parent.yaml	Removes legacy DDT yaml parent defaults.
dev/integration/tests/basic_agent/webchat/SendStreamActivity_SendStreamMessage_ExpectStreamResponses.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/webchat/SendInvoke_SendsInvokeActivityToAcExecute_ReturnsValidAdaptiveCardInvokeResponse.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/webchat/SendInvoke_SendBasicInvokeActivity_ReceiveInvokeResponse.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/webchat/SendInvoke_SelectItem_ReceiveItem.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/webchat/SendInvoke_QueryPackage_ReceiveInvokeResponse.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/webchat/SendInvoke_QueryLink_ReturnsText.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/webchat/SendExpectedRepliesActivity_SendsText_ReturnsPoem.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/webchat/SendExpectedRepliesActivity_SendsSeattleTodayWeather_ReturnsWeather.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/webchat/SendActivity_SimulateMessageLoop_ExpectQuestionAboutTimeAndReturnsWeather.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/webchat/SendActivity_SendsText_ReturnsPoem.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/webchat/SendActivity_SendsSeattleTodayWeather_ReturnsWeather.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/webchat/SendActivity_SendsMessageActivityToAcSubmit_ReturnValidResponse.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/webchat/SendActivity_SendsHi5_Returns5HiActivities.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/webchat/SendActivity_SendsHelloWorld_ReturnsHelloWorld.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/webchat/SendActivity_SendHeartMessageReaction_ReturnsMessageReactionHeart.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/webchat/SendActivity_RemoveHeartMessageReaction_ReturnsMessageReactionHeart.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/webchat/SendActivity_EndConversation_DeleteConversation.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/webchat/SendActivity_ConversationUpdate_ReturnsWelcomeMessage.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/test_basic_agent.py	Removes legacy integration test harness usage.
dev/integration/tests/basic_agent/msteams/SendStreamActivity_SendStreamMessage_ExpectStreamResponses.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/msteams/SendInvoke_SendsInvokeActivityToAcExecute_ReturnsValidAdaptiveCardInvokeResponse.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/msteams/SendInvoke_SendBasicInvokeActivity_ReceiveInvokeResponse.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/msteams/SendInvoke_SelectItem_ReceiveItem.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/msteams/SendInvoke_QueryPackage_ReceiveInvokeResponse.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/msteams/SendInvoke_QueryLink_ReturnsText.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/msteams/SendExpectedRepliesActivity_SendsText_ReturnsPoem.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/msteams/SendExpectedRepliesActivity_SendsSeattleTodayWeather_ReturnsWeather.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/msteams/SendActivity_StartTeamsMeeting_ExpectMessage.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/msteams/SendActivity_SimulateMessageLoop_ExpectQuestionAboutTimeAndReturnsWeather.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/msteams/SendActivity_SendsText_ReturnsPoem.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/msteams/SendActivity_SendsSeattleTodayWeather_ReturnsWeather.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/msteams/SendActivity_SendsMessageActivityToAcSubmit_ReturnValidResponse.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/msteams/SendActivity_SendsHi5_Returns5HiActivities.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/msteams/SendActivity_SendsHelloWorld_ReturnsHelloWorld.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/msteams/SendActivity_SendHeartMessageReaction_ReturnsMessageReactionHeart.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/msteams/SendActivity_RemoveHeartMessageReaction_ReturnsMessageReactionHeart.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/msteams/SendActivity_ParticipantJoinsTeamMeeting_ExpectMessage.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/msteams/SendActivity_EndTeamsMeeting_ExpectMessage.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/msteams/SendActivity_EndConversation_DeleteConversation.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/msteams/SendActivity_EditMessage_ReceiveUpdate.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/msteams/SendActivity_ConversationUpdate_ReturnsWelcomeMessage.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/directline/SendStreamActivity_SendStreamMessage_ExpectStreamResponses.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/directline/SendInvoke_SendsInvokeActivityToAcExecute_ReturnsValidAdaptiveCardInvokeResponse.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/directline/SendInvoke_SendBasicInvokeActivity_ReceiveInvokeResponse.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/directline/SendInvoke_SelectItem_ReceiveItem.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/directline/SendInvoke_QueryPackage_ReceiveInvokeResponse.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/directline/SendInvoke_QueryLink_ReturnsText.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/directline/SendExpectedRepliesActivity_SendsText_ReturnsPoem.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/directline/SendExpectedRepliesActivity_SendsSeattleTodayWeather_ReturnsWeather.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/directline/SendActivity_SimulateMessageLoop_ExpectQuestionAboutTimeAndReturnsWeather.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/directline/SendActivity_SendsText_ReturnsPoem.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/directline/SendActivity_SendsSeattleTodayWeather_ReturnsWeather.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/directline/SendActivity_SendsMessageActivityToAcSubmit_ReturnValidResponse.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/directline/SendActivity_SendsHi5_Returns5HiActivities.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/directline/SendActivity_SendsHelloWorld_ReturnsHelloWorld.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/directline/SendActivity_SendHeartMessageReaction_ReturnsMessageReactionHeart.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/directline/SendActivity_RemoveHeartMessageReaction_ReturnsMessageReactionHeart.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/directline/SendActivity_EndConversation_DeleteConversation.yaml	Removes legacy DDT yaml test definition.
dev/integration/tests/basic_agent/directline/SendActivity_ConversationUpdate_ReturnsWelcomeMessage.yaml	Removes legacy DDT yaml test definition.
dev/integration/samples/quickstart_sample.py	Removes legacy integration sample.
dev/integration/samples/init.py	Removes legacy samples export module.
dev/integration/pytest.ini	Removes separate integration pytest config.
dev/integration/agents/basic_agent/python/src/weather/plugins/weather_forecast_plugin.py	Removes legacy sample agent plugin code.
dev/integration/agents/basic_agent/python/src/weather/plugins/weather_forecast.py	Removes legacy sample agent model.
dev/integration/agents/basic_agent/python/src/weather/plugins/date_time_plugin.py	Removes legacy sample agent plugin code.
dev/integration/agents/basic_agent/python/src/weather/plugins/adaptive_card_plugin.py	Removes legacy sample agent plugin code.
dev/integration/agents/basic_agent/python/src/weather/plugins/init.py	Removes legacy plugin exports.
dev/integration/agents/basic_agent/python/src/weather/agents/weather_forecast_agent.py	Removes legacy agent implementation.
dev/integration/agents/basic_agent/python/src/config.py	Removes legacy sample agent config.
dev/integration/agents/basic_agent/python/src/app.py	Removes legacy sample agent host app.
dev/integration/agents/basic_agent/python/requirements.txt	Removes legacy sample dependency list.
dev/integration/agents/basic_agent/python/pre_requirements.txt	Removes legacy sample pre-req list.
dev/integration/agents/basic_agent/python/env.TEMPLATE	Removes legacy sample env template.
dev/integration/agents/basic_agent/python/README.md	Removes legacy sample agent README.
dev/benchmark/src/payload_sender.py	Removes benchmark tool implementation.
dev/benchmark/src/output.py	Removes benchmark output module.
dev/benchmark/src/main.py	Removes benchmark CLI entrypoint.
dev/benchmark/src/generate_token.py	Removes benchmark token utilities.
dev/benchmark/src/executor/thread_executor.py	Removes benchmark thread executor.
dev/benchmark/src/executor/executor.py	Removes benchmark executor interface.
dev/benchmark/src/executor/execution_result.py	Removes benchmark execution result model.
dev/benchmark/src/executor/coroutine_executor.py	Removes benchmark coroutine executor.
dev/benchmark/src/executor/init.py	Removes benchmark executor exports.
dev/benchmark/src/config.py	Removes benchmark config loader.
dev/benchmark/src/aggregated_results.py	Removes benchmark results aggregator.
dev/benchmark/requirements.txt	Removes benchmark dependencies list.
dev/benchmark/payload.json	Removes benchmark sample payload.
dev/benchmark/env.template	Removes benchmark env template.
dev/benchmark/README.md	Removes benchmark documentation.
dev/README.md	Simplifies dev README to focus on editable install.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-01T14:47:24Z

dev/microsoft-agents-testing/microsoft_agents/testing/cli/config.py

+# TODO: This import path is incorrect - set_defaults is in core.fluent.backend.utils
+# Should be: from microsoft_agents.testing.core.fluent.backend.utils import set_defaults
+from microsoft_agents.testing.utils import set_defaults


CLIConfig will fail at runtime: it imports set_defaults from microsoft_agents.testing.utils (which is a different module now) and then calls self._env_defaults which is not defined anywhere in this class. Fix by importing set_defaults from the actual module where it lives and either defining _env_defaults (as a dict) or removing the call if defaults are not required.

Copilot · 2026-02-01T14:47:24Z

dev/microsoft-agents-testing/microsoft_agents/testing/cli/config.py

+        # TODO: _env_defaults is not defined - this will raise AttributeError
+        set_defaults(self._env, self._env_defaults)


CLIConfig will fail at runtime: it imports set_defaults from microsoft_agents.testing.utils (which is a different module now) and then calls self._env_defaults which is not defined anywhere in this class. Fix by importing set_defaults from the actual module where it lives and either defining _env_defaults (as a dict) or removing the call if defaults are not required.

Copilot · 2026-02-01T14:47:24Z

dev/microsoft-agents-testing/microsoft_agents/testing/cli/config.py

+def load_environment(
+    env_path: str | None = None,
+) -> tuple[dict, str]:


load_environment() is annotated to return tuple[dict, str] but returns ({}, None) when the file doesn't exist. This is a concrete type/contract violation and can cause downstream errors. Update the return type to tuple[dict, str | None] (and adjust the docstring), or return an empty string instead of None.

Copilot · 2026-02-01T14:47:25Z

dev/microsoft-agents-testing/microsoft_agents/testing/cli/config.py

+    if not path.exists():
+        return {}, None


load_environment() is annotated to return tuple[dict, str] but returns ({}, None) when the file doesn't exist. This is a concrete type/contract violation and can cause downstream errors. Update the return type to tuple[dict, str | None] (and adjust the docstring), or return an empty string instead of None.

Copilot · 2026-02-01T14:47:25Z

dev/microsoft-agents-testing/microsoft_agents/testing/cli/commands/run.py

+    async with ins.client():
+        while True:
+            pass


This command will spin forever at 100% CPU due to while True: pass, effectively hanging the CLI. Replace this with actual scenario execution logic (e.g., send a message, stream transcript, or run until user quits) and/or block on a proper wait mechanism rather than a busy loop.

Copilot · 2026-02-01T14:47:26Z

dev/microsoft-agents-testing/microsoft_agents/testing/transcript_logger.py

+    @abstractmethod
+    def format(self, transcript: Transcript) -> str:
+        """Format the given Transcript into a string representation."""
+        exchanges = sorted(self._select(transcript), key=_exchange_node_dt_sort_key)
+        formatted_exchanges = [ self._format_exchange(e) for e in exchanges ]
+        return "\n".join(formatted_exchanges)


format() is marked @abstractmethod but also provides a complete implementation. This prevents instantiation of subclasses that don't override format(), even though the base implementation is usable. Consider removing @abstractmethod from format() and leaving only _select() / _format_exchange() abstract.

Copilot · 2026-02-01T14:47:27Z

dev/microsoft-agents-testing/microsoft_agents/testing/utils.py

+from microsoft_agents.testing.core import (
+    ActivityTemplate,
+    ScenarioConfig,
+    Exchange,
+    ExternalScenario,
+)


In this file, ActivityTemplate, ScenarioConfig, and Exchange are imported but not used anywhere in the shown module. Removing unused imports will reduce lint noise and avoid implying these types are required for the helper functions.

Copilot · 2026-02-01T14:47:27Z

dev/microsoft-agents-testing/pytest.ini

    ignore::PendingDeprecationWarning
    # pytest-asyncio warnings that are safe to ignore
    ignore:.*deprecated.*asyncio.*:DeprecationWarning:pytest_asyncio.*
+    ignore:pytest.PytestUnraisableExceptionWarning


This filterwarnings entry matches the warning message pattern, not the category, so it may not reliably suppress PytestUnraisableExceptionWarning. If the intent is to ignore by category, the usual pattern is ignore::pytest.PytestUnraisableExceptionWarning (or ignore::PytestUnraisableExceptionWarning with the appropriate importable name).

Suggested change

ignore:pytest.PytestUnraisableExceptionWarning

ignore::pytest.PytestUnraisableExceptionWarning

Copilot · 2026-02-01T14:47:27Z

dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/types/unset.py

+from .readonly import Readonly
+
+
+class Unset(Readonly):


This shadows the Unset class with an instance of itself at import time, making it impossible to refer to the class by name (and complicating type checking/debugging). A clearer pattern is to keep the class name (e.g., _Unset) and export an instance constant (e.g., UNSET or UnsetValue) to avoid the class/instance name collision.

Copilot · 2026-02-01T14:47:27Z

dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/types/unset.py

+        """Returns an empty iterator to prevent iteration hangs."""
+        return iter([])
+
+Unset = Unset()


This shadows the Unset class with an instance of itself at import time, making it impossible to refer to the class by name (and complicating type checking/debugging). A clearer pattern is to keep the class name (e.g., _Unset) and export an instance constant (e.g., UNSET or UnsetValue) to avoid the class/instance name collision.

Copilot

Pull request overview

Copilot reviewed 211 out of 261 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-02T23:39:56Z

dev/microsoft-agents-testing/microsoft_agents/testing/core/stream_collector.py

+        self._client = agent_client
+        self._stream_id = None
+
+    async def send(...):


async def send(...): is invalid Python syntax (ellipsis cannot be used as a parameter list). This file will raise a SyntaxError on import and break package imports. Replace it with a valid signature (e.g., async def send(self, ...) with explicit params) or remove the module until implemented.

Suggested change

async def send(...):

async def send(self, *args, **kwargs):

Copilot · 2026-02-02T23:39:56Z

dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/aiohttp_sender.py

+            async with self._session.post(
+                "api/messages",
+                json=activity.model_dump(
+                    by_alias=True, exclude_unset=True, exclude_none=True, mode="json"
+                ),
+                **kwargs
+            ) as response:
+                response_at = datetime.now(timezone.utc)
+                response_or_exception = response
+
+                exchange = await Exchange.from_request(
+                    request_activity=activity,
+                    response_or_exception=response_or_exception,
+                    request_at=request_at,
+                    response_at=response_at,
+                    **kwargs
+                )


**kwargs is forwarded both into ClientSession.post(...) and into Exchange.from_request(...)/Exchange(...). Any aiohttp-specific kwargs (e.g., timeout, ssl) are not part of the Exchange model and may be dropped silently or raise depending on Pydantic config. Pass only the fields that belong to Exchange (timestamps, etc.), and keep request kwargs separate.

Copilot · 2026-02-02T23:39:57Z

dev/microsoft-agents-testing/microsoft_agents/testing/core/external_scenario.py

+    def __init__(self, endpoint: str, config: ScenarioConfig | None = None) -> None:
+        super().__init__(config)
+        if not endpoint:
+            raise ValueError("endpoint must be provided.")
+        self._endpoint = endpoint


ExternalScenario takes an endpoint string, while AiohttpSender always posts to a hardcoded relative path (api/messages). This makes it ambiguous whether callers should pass a base URL (e.g., http://host:3978/) or a full message endpoint (e.g., http://host:3978/api/messages). Align the API by either (1) requiring endpoint to be the full message URL and posting to '' / that path, or (2) requiring a base URL and validating/normalizing it, and updating docs/examples accordingly.

Copilot · 2026-02-02T23:39:57Z

dev/microsoft-agents-testing/microsoft_agents/testing/utils.py

+    Example::
+
+        exchanges = await ex_send("Hello!", "http://localhost:3978/api/messages")


This example conflicts with the current transport implementation: the aiohttp sender posts to api/messages relative to the configured base URL. If callers pass .../api/messages as the base URL, requests will likely go to .../api/messages/api/messages. Update the example to match the expected URL form (or adjust the sender/factory so the example remains correct).

Copilot · 2026-02-02T23:39:57Z

dev/microsoft-agents-testing/pytest.ini

+    ignore:pytest.PytestUnraisableExceptionWarning
+    ignore::aiohttp.web_exceptions.NotAppKeyWarning


The new filterwarnings entry ignore:pytest.PytestUnraisableExceptionWarning looks malformed for warning-category filtering (it’s being treated as a message pattern, not a category). If the intent is to ignore by category, use the ignore::CategoryName form. Also, aiohttp.web_exceptions.NotAppKeyWarning may not match aiohttp’s actual warning class path (older configs commonly used aiohttp.web.NotAppKeyWarning).

Suggested change

ignore:pytest.PytestUnraisableExceptionWarning

ignore::aiohttp.web_exceptions.NotAppKeyWarning

ignore::pytest.PytestUnraisableExceptionWarning

ignore::aiohttp.web.NotAppKeyWarning

Copilot · 2026-02-02T23:39:57Z

dev/microsoft-agents-testing/microsoft_agents/testing/cli/config.py

+def load_environment(
+    env_path: str | None = None,
+) -> tuple[dict, str]:
+    """Load environment variables from a .env file.
+
+    Args:
+        env_path: Path to the .env file. Defaults to ".env" in current directory.
+        override: Whether to override existing environment variables.
+
+    Returns:
+        The resolved path to the loaded .env file.
+
+    Raises:
+        FileNotFoundError: If the specified .env file does not exist.
+    """
+    path = Path(env_path) if env_path else Path(".env")
+
+    if not path.exists():
+        return {}, None


The return type is annotated as tuple[dict, str], but on missing .env it returns ({}, None). This is a concrete type mismatch and can cause downstream None handling bugs. Update the annotation to tuple[dict, str | None] (and fix the docstring’s Returns: section and the unused override mention).

rodrigobr-msft added 30 commits January 31, 2026 10:10

Revamping assertion system

a1b3848

Revamped check system

75d2cbe

Adding more test cases

7bb29a1

Draft implementation of overhauled Checking engine with supporting un…

68086cd

…it tests

Added doc

e724353

Adding back old work

b040e3f

Redoing testing decorator flow

da0fc77

Removing unused files

d3785f8

Updating utils modules

c4e4ad7

Reworked response collection

aa19061

Improved separation of concerns for AgentClient and SenderClient

f1b2901

Removing unused imports

c9504a1

Data and model utils sophistication

134869f

Moving over token generation work

8871eeb

Adding fn.py integration for Check

6c8af5e

Changing Check quantifier selection methods to be properties

7cb736b

Removed functional utils, saving for later

7e65065

Adding updated utils and check tests

01b39cd

Added copyright comments and created tests for AgentClient subcomponents

c6cfdac

Fixed bug in ModelTemplate

d756032

Progress in testing AiohttpAgentScenario

f621053

Bug fix for aiohttp scenario and provided more tests

463467e

Added agent_test tests

2edbdce

Adding underscore utilities

d4ee441

Adding more tests and documenting limitations

2d48688

Updating docs

e7233f8

Update to Check API

2fecf56

Moving things around

baf3d36

Implement CLI commands

3a6bfb3

Cleaning up CLI imports/exports

9a83b2b

rodrigobr-msft added 19 commits January 31, 2026 10:10

Fixing small bugs

55f3798

More refactoring! Woo

3172bb8

Fixing imports

9c2aefa

Draft implementation of streamlined client backend with exchange reco…

a783623

…rding

Reorganization

8cc3334

Adding utils tests

ca9b09c

Adding more tests and fixes to client root module

556df51

Adding methods to access exchange on sends

71c5154

Fixed issues in implementation of AgentClient.ex_send_invoke

829d046

Adjustments to CLI

d593cc6

All tests pass (i never thought I'd see the day)

9534c1e

Adding doc

50bb3e6

Updated backend components and redoing unit tests

828121b

Core tests

290ee00

sketching out non-core components

d55d312

Updated to pyproject.toml

955fdaa

Reorganizing configuration

3a479b5

Adding docstrings

5e3e912

Pytest plugin integration

29677a0

Copilot AI review requested due to automatic review settings February 1, 2026 14:44

github-advanced-security bot found potential problems Feb 1, 2026

View reviewed changes

Copilot AI reviewed Feb 1, 2026

View reviewed changes

rodrigobr-msft added 2 commits February 1, 2026 11:48

Adding transcript formatter variants

99761bc

Adding streaming support

161a5b9

Copilot AI review requested due to automatic review settings February 2, 2026 23:37

Copilot AI reviewed Feb 2, 2026

View reviewed changes

rodrigobr-msft closed this Feb 4, 2026

		# TODO: _env_defaults is not defined - this will raise AttributeError
		set_defaults(self._env, self._env_defaults)

	ignore:pytest.PytestUnraisableExceptionWarning
	ignore::pytest.PytestUnraisableExceptionWarning

		Example::

		exchanges = await ex_send("Hello!", "http://localhost:3978/api/messages")

		ignore:pytest.PytestUnraisableExceptionWarning
		ignore::aiohttp.web_exceptions.NotAppKeyWarning

Users/robrandao/integration overhaul #300

Users/robrandao/integration overhaul #300

Uh oh!

Conversation

rodrigobr-msft commented Feb 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Check warning

Uh oh!

Copilot Autofix

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

rodrigobr-msft commented Feb 1, 2026 •

edited

Loading