Draft #6547

Youssef1313 · 2025-09-10T07:53:10Z

This PR adds a draft documentation for the pipe protocol, which is used when running Microsoft.Testing.Platform test applications with the new MTP-specific dotnet test implementation shipping in .NET 10.

Fixes #6187

Clarify message reception rules for TestHost and TestHostController.

drognanar · 2025-09-10T12:27:59Z

docs/pipe-protocol.md

+This document outlines the protocol used by 'dotnet test' CLI when communicating with Microsoft.Testing.Platform (MTP) applications.
+
+> [!NOTE]
+> Through the document, .NET CLI will be referred to for easy interpretation, but it's not necessarily ".NET CLI".


what does this sentence mean? what's the difference between .NET CLI and ".NET CLI"

I meant we are using .NET CLI as an example. But it could be whatever other client (or well, from pipe-point of view, server)

the JSON-RPC protocol has the following start

MSTest Runner projects builds into a self-contained executable that can be invoked to run tests. The protocol describes the communication between the client (IDE/CLI/CI) any the MSTest Runner executable (also refered to as the server).
The communication is based on JSON-RPC and describes the RPC messages sent in order to support running of tests.

if we'd use a similar preamble reading these docs side by side would be easier.

otherwise, a similar note could be added. that the document describes the communication between any client (dotnet test) and MTP runner. and that afterwards this document uses .NET CLI when referring to any compatible client

drognanar · 2025-09-10T12:29:51Z

docs/pipe-protocol.md

+      - In this case, should we have a "special" response that means "I didn't understand your request"?
+      - I think such special response gives us the most flexibility.
+
+## General protocol message format


Should we have under the General flow some reference that the messages are sent using a BinaryFormatter usign the payload defined in this section?

I'm not sure how BinaryFormatter is relevant here.

without seeing the reference implementation I'm unsure how the messages are encoded in practice. knowing the Stream/ StreamWriter/ Serializer APIs used to encode the messages would help me understand this better

drognanar · 2025-09-10T12:31:00Z

docs/pipe-protocol.md

+Handshake is expected to happen in all modes. Help, discovery, and execution.
+Today, there is an MTP bug that we don't handshake in help mode. This will be worked around in .NET CLI. A fix needs to be done in MTP.
+
+### Handshake request


It might be nice to align it with the MTP VS protocol spec and write down the direction of the request for brevity. for instance it looks like the request is first sent from the MTP server to the dotnet test CLI. maybe it's written somewhere above and I missed it

drognanar · 2025-09-10T12:48:10Z

docs/pipe-protocol.md

+- Supported protocol versions (required): The protocol versions that are supported by the test application, separated by semicolons.
+- Host type (required): The host type which is handshaking with us.
+- Module path (required): The path to the test module, either the assembly dll path, or the actual apphost (exe) path.
+- ExecutionId (required): Explained above.


if this is a markdown document it might be better to add hyperlinks here, so that I can click and navigate to the definition quickly

drognanar · 2025-09-10T12:51:13Z

docs/pipe-protocol.md

+
+### `ExecutionId`
+
+For the execution of a test app, the test app and all its child processes are uniquely identified by `ExecutionId`. Per **current implementation**, we can already identify the test app and all its child processes by knowing which pipe is receiving the message. But we are still including `ExecutionId` in the protocol, in case we needed it in future due to implementation changes.


so the ExecutionId is a generated GUID by the test runner? are currently messages validated that the ExecutionId does not change?

Yes. It's a Guid generated by MTP and is the same for child processes as well.
Today's implementation might be missing the validation. The purpose of this doc is to have a well-defined spec, then review the implementation and adjust it as needed.

drognanar · 2025-09-10T12:59:00Z

docs/pipe-protocol.md

+- SessionUid (required)
+- ExecutionId (required)
+
+## Command line options


it might be good to add "message" for consistency

drognanar · 2025-09-10T13:00:03Z

docs/pipe-protocol.md

+
+## Command line options
+
+This message is expected to be received only in help mode.


it would be useful to explain what this message means from user's point of view, i.e. that if a user calls dotnet test --help each test application is expected to report its command line options back

drognanar · 2025-09-10T13:01:17Z

docs/pipe-protocol.md

+- When running in "execution" mode, the absence of these messages mean that the test app doesn't have any tests.
+- When running in "help" mode, the test app MUST NOT send these messages.
+
+This message contains the following properties:


are in-progress tests somehow being relayed?

A while back, I added an "IDE mode" to the protocol (and I'm realizing now this part isn't documented and probably needs to be documented). With the "IDE" mode, in-progress is reported as "successful" with state denoting "in-progress". In IDE mode, we also send line number and file path. Maybe few more things that I don't recall now.

drognanar · 2025-09-10T13:07:48Z

docs/pipe-protocol.md

+  - What we could do?
+    - Keep InstanceId for compatibility.
+    - Add the metadata on test results as "another" way of knowing retries.
+    - In handshake request, tell .NET CLI that we have the "new" way of knowing retries.


are we considering using the VS server protocol for dotnet test at any point? we already have capabilities there and this would make it easier to include features such as retries to both CLI and VS/VsCode simulatenously

I personally would consider the opposite 😄
Any kind of "capability" can potentially be added as a property in handshake request.

the JSON-RPC provides a more general-purpose way of communicating between client and server and comes with out of the box implementations for many programming languages, decreasing the amount of work needed to create various implementations

and there's a couple of features that the JSON-RPC brings out of the box

allows server to respond with exceptions to the client (rather than just rely on console stderr)

allows client to cancel requests (rather than support cancellation just via Ctrl+C)

allows to overlap messages (where now requests cannot be processed in parallel)

allows to write a client that logs all RPC messages for debugging purposes (as by default everything is just JSON). if one of the endpoints doesn't know how to decode the binary, debugging through the message is much trickier

supports both request/response and notifications data formats

if we use well known JSON-RPC libraries we already can get the endianness handling for free

all of the serialization logic is already well known/documented since it's just JSON

of course these can be over added to the pipe protocol, but we're effectively reimplementing JSON-RPC with a custom serialization format

drognanar · 2025-09-10T13:14:01Z

docs/pipe-protocol.md

+
+- Youssef: Can we get rid of InstanceId completely? I personally don't believe it was the right design.
+  - If we want to simply track retries, that could simply have been a metadata on test results.
+  - Having it as a metadata on test results even allows for in-process retries to be reported.


I think the tricky part here is to align the tests with file artifacts.
For instance, if the tests get rerun 3 times, how should these be reported.
And when should a FileArtifact get a different SessionId.

So if a test runner runs test1 twice and collects the coverage once, perhaps there should be a single SessionId, but two different RunIds.
Whereas, if a test runner runs test1 once, collects the coverage and runs it again, perhaps there should be just one RunId, but two different SessionIds.

Hmm, for the purpose of dotnet test, it's sufficient to link artifacts to test results via TestNodeUid+SessionUid. But, this might limit us if, in future, we wanted to use the pipe protocol in VS. There is at least an ambiguity when an MSTest test method is folded and it reports multiple results. In this case, we can't link artifacts to individual results.

At the same time, TestNodeUid is supposedly to uniquely identify a "single" test. So, what MSTest does today with folding is already kinda a violation of MTP contract.

Youssef1313 added 8 commits September 8, 2025 10:10

Protocol draft

3d82951

Update

88472b6

Fix markdown

cc92e57

Markdownlint

fea6919

Fix indentation

a859c15

Progress

8acb309

Progress

04f40e5

Clarify message reception rules for TestHost and TestHostController.

Markdownlint

8468638

drognanar reviewed Sep 10, 2025

View reviewed changes


		### `ExecutionId`

		For the execution of a test app, the test app and all its child processes are uniquely identified by `ExecutionId`. Per current implementation, we can already identify the test app and all its child processes by knowing which pipe is receiving the message. But we are still including `ExecutionId` in the protocol, in case we needed it in future due to implementation changes.


		## Command line options

		This message is expected to be received only in help mode.

Draft #6547

Are you sure you want to change the base?

Draft #6547

Uh oh!

Conversation

Youssef1313 commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Youssef1313 commented Sep 10, 2025 •

edited

Loading