Tests failing in master

[justinormont]

As noted on our repo's README.md, the build is failing (in a test), which appears due to MulticlassTreeFeaturizedLRTest on OS X:

MulticlassTreeFeaturizedLRTest:

machinelearning/test/Microsoft.ML.Predictor.Tests/TestPredictors.cs

Lines 199 to 207 in 9157cea

	public void MulticlassTreeFeaturizedLRTest()
	{
	RunMTAThread(() =>
	{
	RunOneAllTests(TestLearners.multiclassLogisticRegression, TestDatasets.irisTreeFeaturized, digitsOfPrecision: 4);
	RunOneAllTests(TestLearners.multiclassLogisticRegression, TestDatasets.irisTreeFeaturizedPermuted, digitsOfPrecision: 4);
	});
	Done();
	}

Test error: (full logs)

2018-10-18T23:35:15.5213890Z Failed   Microsoft.ML.Runtime.RunTests.TestPredictors.MulticlassTreeFeaturizedLRTest
2018-10-18T23:35:15.5214010Z Error Message:
2018-10-18T23:35:15.5214250Z  System.InvalidOperationException : MTA thread failed
2018-10-18T23:35:15.5215480Z ---- Assert.InRange() Failure
2018-10-18T23:35:15.5216230Z Range:  (-0.0001 - 0.0001)
2018-10-18T23:35:15.5216970Z Actual: -3
2018-10-18T23:35:15.5217100Z Stack Trace:
2018-10-18T23:35:15.5217280Z    at Microsoft.ML.Runtime.RunTests.BaseTestBaseline.RunMTAThread(ThreadStart fn) in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs:line 736
2018-10-18T23:35:15.5217490Z    at Microsoft.ML.Runtime.RunTests.TestPredictors.MulticlassTreeFeaturizedLRTest() in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.Predictor.Tests/TestPredictors.cs:line 201
2018-10-18T23:35:15.5218240Z ----- Inner Stack Trace -----
2018-10-18T23:35:15.5218440Z    at Microsoft.ML.Runtime.RunTests.BaseTestBaseline.MatchNumberWithTolerance(MatchCollection firstCollection, MatchCollection secondCollection, Int32 digitsOfPrecision) in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs:line 544
2018-10-18T23:35:15.5219030Z    at Microsoft.ML.Runtime.RunTests.BaseTestBaseline.GetNumbersFromFile(String& firstString, String& secondString, Int32 digitsOfPrecision) in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs:line 513
2018-10-18T23:35:15.5219310Z    at Microsoft.ML.Runtime.RunTests.BaseTestBaseline.CheckEqualityFromPathsCore(String relPath, String basePath, String outPath, Int32 skip, Int32 digitsOfPrecision) in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs:line 493
2018-10-18T23:35:15.5219590Z    at Microsoft.ML.Runtime.RunTests.BaseTestBaseline.CheckEqualityCore(String dir, String name, String nameBase, Boolean normalize, Int32 digitsOfPrecision) in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs:line 369
2018-10-18T23:35:15.5219830Z    at Microsoft.ML.Runtime.RunTests.BaseTestBaseline.CheckEqualityNormalized(String dir, String name, String nameBase, Int32 digitsOfPrecision) in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs:line 339
2018-10-18T23:35:15.5220090Z    at Microsoft.ML.Runtime.RunTests.TestCommandBase.OutputPath.CheckEqualityNormalized(Int32 digitsOfPrecision) in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.TestFramework/TestCommandBase.cs:line 76
2018-10-18T23:35:15.5220320Z    at Microsoft.ML.Runtime.RunTests.BaseTestPredictors.Run(RunContext ctx, Int32 digitsOfPrecision) in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.TestFramework/BaseTestPredictorsMaml.cs:line 167
2018-10-18T23:35:15.5220600Z    at Microsoft.ML.Runtime.RunTests.BaseTestPredictors.Run_TrainTest(PredictorAndArgs predictor, TestDataset dataset, String[] extraSettings, String extraTag, Boolean expectFailure, Boolean summary, Boolean saveAsIni, Int32 digitsOfPrecision) in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.TestFramework/BaseTestPredictorsMaml.cs:line 389
2018-10-18T23:35:15.5220900Z    at Microsoft.ML.Runtime.RunTests.BaseTestPredictors.RunOneAllTests(PredictorAndArgs predictor, TestDataset dataset, String[] extraSettings, String extraTag, Boolean summary, Int32 digitsOfPrecision) in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.TestFramework/BaseTestPredictorsMaml.cs:line 357
2018-10-18T23:35:15.5221170Z    at Microsoft.ML.Runtime.RunTests.TestPredictors.<MulticlassTreeFeaturizedLRTest>b__15_0() in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.Predictor.Tests/TestPredictors.cs:line 203
2018-10-18T23:35:15.5221390Z    at Microsoft.ML.Runtime.RunTests.BaseTestBaseline.<>c__DisplayClass92_0.<RunMTAThread>b__0() in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs:line 721
2018-10-18T23:35:15.5221830Z Standard Output Messages:
2018-10-18T23:35:15.5222710Z  Running 'MulticlassLogisticRegression' on 'iris-tree-featurized'
2018-10-18T23:35:15.5224910Z    Running as: TrainTest tr=MulticlassLogisticRegression{l1=0.001 l2=0.1 ot=1e-3 nt=1} data=/Users/buildagent/agent/_work/1/s/test/data/iris.txt seed=1 test=/Users/buildagent/agent/_work/1/s/test/data/iris.txt loader=Text{col=Label:U4[0-2]:0 col=Features:1-*} xf=TreeFeat{lps=0 trainer=ftr{iter=3}} xf=copy{col=Features:Leaves} out={/Users/buildagent/agent/_work/1/s/bin/AnyCPU.Debug/Microsoft.ML.Predictor.Tests/netcoreapp2.1/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris-tree-featurized-model.zip} dout={/Users/buildagent/agent/_work/1/s/bin/AnyCPU.Debug/Microsoft.ML.Predictor.Tests/netcoreapp2.1/TestOutput/MulticlassLogisticRegression/MulticlassLogisticRegression-TrainTest-iris-tree-featurized.txt} norm=no
2018-10-18T23:35:15.5226910Z  *** Failure: The test threw an exception - Xunit.Sdk.InRangeException: Assert.InRange() Failure
2018-10-18T23:35:15.5227350Z  Range:  (-0.0001 - 0.0001)
2018-10-18T23:35:15.5227700Z  Actual: -3
2018-10-18T23:35:15.5227810Z     at Xunit.Assert.InRange[T](T actual, T low, T high, IComparer`1 comparer) in C:\Dev\xunit\xunit\src\xunit.assert\Asserts\RangeAsserts.cs:line 41
2018-10-18T23:35:15.5228220Z     at Microsoft.ML.Runtime.RunTests.BaseTestBaseline.MatchNumberWithTolerance(MatchCollection firstCollection, MatchCollection secondCollection, Int32 digitsOfPrecision) in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs:line 544
2018-10-18T23:35:15.5228390Z     at Microsoft.ML.Runtime.RunTests.BaseTestBaseline.GetNumbersFromFile(String& firstString, String& secondString, Int32 digitsOfPrecision) in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs:line 513
2018-10-18T23:35:15.5228560Z     at Microsoft.ML.Runtime.RunTests.BaseTestBaseline.CheckEqualityFromPathsCore(String relPath, String basePath, String outPath, Int32 skip, Int32 digitsOfPrecision) in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs:line 493
2018-10-18T23:35:15.5228720Z     at Microsoft.ML.Runtime.RunTests.BaseTestBaseline.CheckEqualityCore(String dir, String name, String nameBase, Boolean normalize, Int32 digitsOfPrecision) in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs:line 369
2018-10-18T23:35:15.5228970Z     at Microsoft.ML.Runtime.RunTests.BaseTestBaseline.CheckEqualityNormalized(String dir, String name, String nameBase, Int32 digitsOfPrecision) in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs:line 339
2018-10-18T23:35:15.5229110Z     at Microsoft.ML.Runtime.RunTests.TestCommandBase.OutputPath.CheckEqualityNormalized(Int32 digitsOfPrecision) in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.TestFramework/TestCommandBase.cs:line 76
2018-10-18T23:35:15.5229250Z     at Microsoft.ML.Runtime.RunTests.BaseTestPredictors.Run(RunContext ctx, Int32 digitsOfPrecision) in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.TestFramework/BaseTestPredictorsMaml.cs:line 167
2018-10-18T23:35:15.5229410Z     at Microsoft.ML.Runtime.RunTests.BaseTestPredictors.Run_TrainTest(PredictorAndArgs predictor, TestDataset dataset, String[] extraSettings, String extraTag, Boolean expectFailure, Boolean summary, Boolean saveAsIni, Int32 digitsOfPrecision) in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.TestFramework/BaseTestPredictorsMaml.cs:line 389
2018-10-18T23:35:15.5229590Z     at Microsoft.ML.Runtime.RunTests.BaseTestPredictors.RunOneAllTests(PredictorAndArgs predictor, TestDataset dataset, String[] extraSettings, String extraTag, Boolean summary, Int32 digitsOfPrecision) in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.TestFramework/BaseTestPredictorsMaml.cs:line 357
2018-10-18T23:35:15.5229890Z     at Microsoft.ML.Runtime.RunTests.TestPredictors.<MulticlassTreeFeaturizedLRTest>b__15_0() in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.Predictor.Tests/TestPredictors.cs:line 203
2018-10-18T23:35:15.5230030Z     at Microsoft.ML.Runtime.RunTests.BaseTestBaseline.<>c__DisplayClass92_0.<RunMTAThread>b__0() in /Users/buildagent/agent/_work/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs:line 721
2018-10-18T23:35:15.5230140Z  Test MulticlassTreeFeaturizedLRTest: aborted: failed
2018-10-18T23:35:15.5230190Z 
2018-10-18T23:35:15.5230230Z 

[justinormont]

These look like another instance of random test failures (given that the tests passed in the previous PR).

[justinormont]

Previous guess was correct; after the next rebuild, our build status is green again. This appears to be a random test failure.

[Zruty0]

@TomFinley here's one more stack trace for your investigation.

[frank-dong-ms-zz]

fixed in #4950, this is a race condition issue