Skip to content

[ci] Use descriptive test run titles. #926

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Dec 6, 2021
Merged

[ci] Use descriptive test run titles. #926

merged 3 commits into from
Dec 6, 2021

Conversation

jpobst
Copy link
Contributor

@jpobst jpobst commented Dec 1, 2021

Add testRunTitle information to our .NET tests so they are easier to read in the Azure Pipelines Test tab.

Additionally, auditing the test suites we run revealed we are not currently running Java.Interop.Tools.JavaTypeSystem-Tests, so this test suite was also added to CI.

@jpobst jpobst force-pushed the test-run-title branch 2 times, most recently from e53e7f5 to 38fa5ca Compare December 2, 2021 19:35
@jpobst jpobst marked this pull request as ready for review December 2, 2021 20:08
@jonpryor
Copy link
Contributor

jonpryor commented Dec 4, 2021

One question/concern; consider when we view the Tests tab, showing all tests, not just failed tests: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=5512883&view=ms.vss-test-web.build-test-results-tab

Many of these names are good, e.g. Java.Interop (netstandard2.0 - .NET - MacOS).

However, many are not, in that they're inscrutable, such as NUnit_TestResults_5512883_4.

I suspect that the inscrutable ones are coming from the TestResult.xml files produced by the RunTests target, and not controllable by the testRunTitle value.

Is there any way to fix these as well?

The JUnit test results from java-source-utils appear to be in a similar bucket: we have 4 com.microsoft.android.ParameterNameGEneratorTest entries in the Tests tab, with ~nothing to differentiate them. (That might be a build.gradle change? Maybe?)

@jpobst jpobst force-pushed the test-run-title branch 3 times, most recently from 8bb25d8 to 5951939 Compare December 6, 2021 19:35
@jpobst
Copy link
Contributor Author

jpobst commented Dec 6, 2021

Is there any way to fix these as well?

Unfortunately this is caused by using the PublishTestResults with a wildcard specifier. In order to get good names, we would need to call it N times to give each run a good name.

I did make updates to make it a little better, and they now specify the platform in the title:

- NUnit test results (Mono - Mac)_1
- NUnit test results (.NET Framework - Windows)_1

I was able to give java-source-utils a good name because it is only running a single test suite.

@jonpryor jonpryor merged commit af91b9c into main Dec 6, 2021
@jonpryor jonpryor deleted the test-run-title branch December 6, 2021 22:08
@github-actions github-actions bot locked and limited conversation to collaborators Apr 13, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants