Skip to content

NaiveCalibratorOnnxConversionTest test failure in PR vallidation #6483

Open
@ericstj

Description

@ericstj

Observed a flaky test failure during PR validation.

    Microsoft.ML.Tests.OnnxConversionTest.NaiveCalibratorOnnxConversionTest [FAIL]
�[31;1m�[m�[37m      Assert.Equal() Failure
�[m�[37m      Expected: 0
�[m�[37m      Actual:   1
�[m�[30;1m      Stack Trace:
�[m�[37m        /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs(201,0): at Microsoft.ML.RunTests.BaseTestBaseline.Done()
�[m�[37m        /__w/1/s/test/Microsoft.ML.Tests/OnnxConversionTest.cs(328,0): at Microsoft.ML.Tests.OnnxConversionTest.CommonCalibratorOnnxConversionTest(IEstimator`1 calibrator, IEstimator`1 calibratorNonStandard)
�[m�[37m        /__w/1/s/test/Microsoft.ML.Tests/OnnxConversionTest.cs(349,0): at Microsoft.ML.Tests.OnnxConversionTest.NaiveCalibratorOnnxConversionTest()
�[m�[30;1m      Output:
�[m�[37m        *** Failure #1: Output and baseline mismatch at line .
�[m�[37m        Values to compare are 1 and 0.97826087474823
�[m�[37m        	 AllowedVariance: 0.001
�[m�[37m        	 delta: 0.022
�[m�[37m        	 delta2: 0.022
�[m�[37m        
�[m�[37m        		Void Fail(System.String, System.Object[]) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs 224
�[m�[37m        		Boolean CompareNumbersWithTolerance(Double, Double, System.Nullable`1[System.Int32], Int32, Boolean) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs 643
�[m�[37m        		Void CompareScalarValues[T](T, T, Int32) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs 787
�[m�[37m        		Void CompareSelectedColumns[T](System.String, System.String, Microsoft.ML.IDataView, Microsoft.ML.IDataView, Int32, Boolean) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs 771
�[m�[37m        		Void CompareResults(System.String, System.String, Microsoft.ML.IDataView, Microsoft.ML.IDataView, Int32, Boolean) /__w/1/s/test/Microsoft.ML.TestFramework/BaseTestBaseline.cs 698
�[m�[37m        		Void TestPipeline[TLastTransformer](Microsoft.ML.Data.EstimatorChain`1[TLastTransformer], Microsoft.ML.IDataView, System.String, ColumnComparison[], System.String, System.String) /__w/1/s/test/Microsoft.ML.Tests/OnnxConversionTest.cs 2261
�[m�[37m        		Void CommonCalibratorOnnxConversionTest(Microsoft.ML.IEstimator`1[Microsoft.ML.ITransformer], Microsoft.ML.IEstimator`1[Microsoft.ML.ITransformer]) /__w/1/s/test/Microsoft.ML.Tests/OnnxConversionTest.cs 315
�[m�[37m        		Void NaiveCalibratorOnnxConversionTest() /__w/1/s/test/Microsoft.ML.Tests/OnnxConversionTest.cs 349

It passed on rerun. Seems to be either non-deterministic or machine dependent.

Metadata

Metadata

Assignees

No one assigned

    Labels

    testrelated to tests

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions