Test failure error message is unreadable in Azure DevOps

[BruceForstall]

When a test fails, and you view the Error Message in Azure DevOps, you get something like this:

\ncmdLine:/private/tmp/helix/working/B08009A5/w/A2FA08B1/e/Interop/ICustomMarshaler/Primitives/ICustomMarshaler/ICustomMarshaler.sh Timed Out\n\nReturn code: -100\nRaw output file: /private/tmp/helix/working/B08009A5/w/A2FA08B1/e/Interop/ICustomMarshaler/Reports/Interop.ICustomMarshaler/Primitives/ICustomMarshaler/ICustomMarshaler.output.txt\nRaw output:\nBEGIN EXECUTION\n/tmp/helix/working/B08009A5/p/corerun ICustomMarshaler.dll ''\nDiscovering...\n\ncmdLine:/private/tmp/helix/working/B08009A5/w/A2FA08B1/e/Interop/ICustomMarshaler/Primitives/ICustomMarshaler/ICustomMarshaler.sh Timed Out\nTest Harness Exitcode is : -100\nTo run the test:\n> set CORE_ROOT=/tmp/helix/working/B08009A5/p\n> /private/tmp/helix/working/B08009A5/w/A2FA08B1/e/Interop/ICustomMarshaler/Primitives/ICustomMarshaler/ICustomMarshaler.sh\nExpected: True\nActual: False

This is unreadable. We need proper line breaking in the view instead of the embedded \n characters.

@dotnet/runtime-infrastructure

[jaredpar]

@BruceForstall can you link to a build / run that has this issue?

[BruceForstall]

I think it's any test failure. E.g.,

https://dev.azure.com/dnceng/public/_build/results?buildId=481170&view=ms.vss-test-web.build-test-results-tab&runId=15269884&resultId=108254&paneView=debug

It looks like the Linux/Mac jobs have \n but the Windows jobs have \r\n text:

https://dev.azure.com/dnceng/public/_build/results?buildId=480582&view=ms.vss-test-web.build-test-results-tab&runId=15254946&paneView=debug&resultId=110332

[ericstj]

/cc @ViktorHofer

[BruceForstall]

To make it worse, the AzDO UI for viewing ends up creating a very wide window with a horizontal scrollbar, so you have to constantly scroll back-and-forth (and read through a mass of no-whitespace text) to find the relevant bits of information.

[dagood]

I always copy the text out and replace escaped newlines with newlines in np++. I would love to see this fixed.

[jaredpar]

It looks like the Linux/Mac jobs have \n but the Windows jobs have \r\n text:

I've complained to AzDO about this before (we have same issue in Roslyn). They constantly push back the issue is on our end not there's but even when digging in I could never see where we were doing extra escaping.

[jkoritzinsky]

Is it possible that it's the Helix xunit reporter that's doing the escaping? That's the common factor between Roslyn and dotnet/runtime that other repos using xunit on AzDO don't have.

[BruceForstall]

Is it possible that it's the Helix xunit reporter that's doing the escaping? That's the common factor between Roslyn and dotnet/runtime that other repos using xunit on AzDO don't have.

@MattGal Can you suggest someone with knowledge of the Helix xunit reporter upload to AzDO that might be able to investigate this?

[jaredpar]

@jkoritzinsky

Is it possible that it's the Helix xunit reporter that's doing the escaping? That's the common factor between Roslyn and dotnet/runtime that other repos using xunit on AzDO don't have.

Roslyn doesn't use Helix yet so it's not a common factor for this.

Note: it should be possible to see if we are or aren't escaping by looking at the raw xUnit logs. They should be available in XML form for every test that fails in Helix correct? If so we can just look there and see if the values are escaped. That would at least eliminate infra in dotnet/runtime as the culprit.

[MattGal]

Is it possible that it's the Helix xunit reporter that's doing the escaping? That's the common factor between Roslyn and dotnet/runtime that other repos using xunit on AzDO don't have.

@MattGal Can you suggest someone with knowledge of the Helix xunit reporter upload to AzDO that might be able to investigate this?

@jaredpar 's comment likely obviates this but you can read the full sources here: https://github.com/dotnet/arcade/blob/master/src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/azure_devops_result_publisher.py and @alexperovich wrote it.

[alexperovich]

The build linked in the issue is too old for the results file to still exist. I looked at the latest build but the xunit file isn't avaliable for the JIT.HardwareIntrinsics work item. If we can catch another instance I can grab the xunit file and check.

[BruceForstall]

@alexperovich How about https://dev.azure.com/dnceng/public/_build/results?buildId=498865&view=ms.vss-test-web.build-test-results-tab&runId=15859996&resultId=109037&paneView=debug

[alexperovich]

That workitem doesn't have the test results file uploaded for some reason. The older one did, it just doesn't exist anymore.