RemoteExecutor Swallows Errors and Throws Invalid Exceptions

[JimBobSquarePants]

• This issue is blocking
• This issue is causing unreasonable pain

I could be holding it wrong but it does not seem to be working as expected.

Microsoft.DotNet.RemoteExecutor Version 6.0.0-beta.20508.3 (Are there non beta version available?)

Installed via https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json

The following method runs without any assertation exceptions. I would expect it to fail.

            RemoteExecutor.Invoke(() =>
            {
                 Assert.True(false);
            }).Dispose();

The following code throws an exception. I would expect none.

            RemoteExecutor.Invoke(() =>
            {
                 return RemoteExecutor.SuccessExitCode;
            }).Dispose();

Message:
Exit code was -2147450749 but it should have been 42
Expected: True
Actual: False
Stack Trace:
RemoteInvokeHandle.Dispose(Boolean disposing) line 237
RemoteInvokeHandle.Dispose() line 58

[JimBobSquarePants]

ExecutorTest.zip

Ok... So I created a separate unit test project to confirm this.

[Fact]
public void Test1()
{
    Assert.True(RemoteExecutor.IsSupported);

    RemoteExecutor.Invoke(() =>
    {
        Assert.True(false);
    }).Dispose();

    RemoteExecutor.Invoke(() =>
    {
        return RemoteExecutor.SuccessExitCode;
    }, new RemoteInvokeOptions { CheckExitCode = true }).Dispose();
}

[JimBobSquarePants]

Pulling the source and running the tests leads to 3 out of 5 tests failing with the 2 passing false positives due to the symptoms described above.

[JimBobSquarePants]

Confirmed to not be isolated to my machine.

Linux Pass
Windows Fail

https://github.com/JimBobSquarePants/RemoteExecutorTests/runs/1232480114

[danmoseley]

-2147450749
https://github.com/dotnet/runtime/blob/6072e4d3a7a2a1493f514cdf4be75a3d56580e84/src/installer/corehost/error_codes.h#L18

• CoreHostLibMissingFailure (-2147450749 0x80008083) - One of the dependent libraries is missing. Typically when the hostfxr, hostpolicy or coreclr dynamic libraries are not present in the expected locations. Probably means corrupted or incomplete installation.

[danmoseley]

https://github.com/dotnet/runtime/search?q=CoreHostLibMissingFailure

Enabling host tracing might help? COREHOST_TRACE=1 COREHOST_TRACE_VERBOSITY=4
I think this goes goes to stderr so should be captured?
https://github.com/dotnet/runtime/blob/d14b50ae2186af77ad9b68370978182584b43e65/docs/design/features/host-tracing.md

[JimBobSquarePants]

Thanks @danmosemsft .

I thought it was something like that and have been attempting to determine whether there is a specific SDKs missing that should be there. It's very odd that the GitHub Ubuntu image works but Windows doesn't.

Is COREHOST_TRACE=1 something I pass to dotnet test?

[danmoseley]

It's an environment variable read by the host - hostfxr.dll, I think (this isn't my area). So you should be able to set it anywhere eg before launching dotnet test, or in your test just before launching the remote executor.

[JimBobSquarePants]

Thanks. I’ll have a crack and report back

[JimBobSquarePants]

No further info I could uncover with that variable. Just lots of build noise.

[danmoseley]

Does the logging not end with an error? Hmm. @vitek-karas I believe owns the host and can help advise from here.

[JimBobSquarePants]

https://github.com/JimBobSquarePants/RemoteExecutorTests/runs/1233024008?check_suite_focus=true#step:5:25756

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:05.66]     ExecutorTest.UnitTest1.ShouldNotThrow [FAIL]
[xUnit.net 00:00:05.68]     ExecutorTest.UnitTest1.ShouldThrow [FAIL]
  X ExecutorTest.UnitTest1.ShouldNotThrow [295ms]
  Error Message:
   Exit code was -2147450749 but it should have been 42
Expected: True
Actual:   False
  Stack Trace:
     at Microsoft.DotNet.RemoteExecutor.RemoteInvokeHandle.Dispose(Boolean disposing) in /_/src/Microsoft.DotNet.RemoteExecutor/src/RemoteInvokeHandle.cs:line 237
   at ExecutorTest.UnitTest1.ShouldNotThrow() in D:\a\RemoteExecutorTests\RemoteExecutorTests\ExecutorTest\UnitTest1.cs:line 23
  X ExecutorTest.UnitTest1.ShouldThrow [28ms]
  Error Message:
   Assert.Throws() Failure
Expected: typeof(Microsoft.DotNet.RemoteExecutor.RemoteExecutionException)
Actual:   (No exception was thrown)
  Stack Trace:
     at ExecutorTest.UnitTest1.ShouldThrow() in D:\a\RemoteExecutorTests\RemoteExecutorTests\ExecutorTest\UnitTest1.cs:line 13

Test Run Failed.
Total tests: 2
     Failed: 2
 Total time: 14.8005 Seconds
Execute managed assembly exit code: 0x1
Waiting for breadcrumb thread to exit...
Done waiting for breadcrumb thread to exit...
C:\Program Files\dotnet\sdk\3.1.402\Microsoft.TestPlatform.targets(32,5): error MSB4181: The "Microsoft.TestPlatform.Build.Tasks.VSTestTask" task returned false but did not log an error. [D:\a\RemoteExecutorTests\RemoteExecutorTests\ExecutorTest\ExecutorTest.csproj]
Execute managed assembly exit code: 0x1
Waiting for breadcrumb thread to exit...
Done waiting for breadcrumb thread to exit...
Execute managed assembly exit code: 0x1
Waiting for breadcrumb thread to exit...
Done waiting for breadcrumb thread to exit...
Execute managed assembly exit code: 0x1
Waiting for breadcrumb thread to exit...
Done waiting for breadcrumb thread to exit...
Error: Process completed with exit code 1.
3s

[danmoseley]

Ah, I don't see the host logging there. Either it's not enabled, or not captured. Does host-tracing.md above help?

[JimBobSquarePants]

There's definitely a lot of extra logging 100s of extra lines, just nothing relevant (that I could see). Here's a previous run.

https://github.com/JimBobSquarePants/RemoteExecutorTests/runs/1232480114?check_suite_focus=true#step:5:1

I'll try moving to Core 3.1 and adding the extra verbosity command

[JimBobSquarePants]

No improvement
https://github.com/JimBobSquarePants/RemoteExecutorTests/runs/1233287574?check_suite_focus=true#step:5:25736