Skip to content

Commit cacb3bf

Browse files
authored
[wasm] Add support for a random test case orderer, for xunit tests (dotnet#65628)
* [wasm] Add support for a random test case orderer, for xunit tests This is enabled by default for wasm with `$(XUnitUseRandomizedTestOrderer)=true`. When the library tests run, they print two messages like: ``` info: Using random seed for test cases: 700149826 info: Using random seed for collections: 700149826 info: Starting: System.Collections.Immutable.Tests.dll ``` These seeds are picked randomly every time the tests are run. To run the tests with a specific seed, use environment variable `XUNIT_RANDOM_ORDER_SEED`. When running with tests, this can be used as: `WasmXHarnessMonoArgs="--setenv=XUNIT_RANDOM_ORDER_SEED=<seed>` * Enable test orderer only for libraries tests * [wasm] Automatically pass XUNIT_RANDOM_ORDER_SEED envvar to wasm apps * Disable random test ordering for `System.Xml.RW.XmlWriterApi.Tests` `System.Xml.Tests.TCCloseOutput.*` tests seem to depend on the order of execution. ``` [06:35:14] fail: [FAIL] System.Xml.Tests.TCCloseOutput.CloseOutput_4(utils: XmlWriterUtils { Async = False, WriterType = UTF8Writer }, outputType: "Stream") [06:35:14] info: System.IO.FileNotFoundException : File Not Found: writer.out [06:35:14] info: at XmlCoreTest.Common.FilePathUtil.getStream(String filename) [06:35:14] info: at System.Xml.Tests.TCCloseOutput.CloseOutput_4(XmlWriterUtils utils, String outputType) [06:35:14] info: at System.Reflection.RuntimeMethodInfo.InvokeWorker(Object obj, BindingFlags invokeAttr, Span`1 parameters) ``` * [wasm] Disable random tests ordering for `System.Runtime.Loader` It seems to cause failures like, reproducible with `XUNIT_RANDOM_ORDER_SEED=2106784294`: ``` [06:25:43] fail: [FAIL] System.Runtime.Loader.Tests.SatelliteAssembliesTests.SatelliteLoadsCorrectly_FromName(alc: "Empty", assemblyName: "System.Runtime.Loader.Tests", culture: "en") [06:25:43] info: Assert.Same() Failure [06:25:43] info: Expected: "Default" System.Runtime.Loader.DefaultAssemblyLoadContext #0 [06:25:43] info: Actual: "Empty" System.Runtime.Loader.AssemblyLoadContext #4 [06:25:43] info: at System.Runtime.Loader.Tests.SatelliteAssembliesTests.SatelliteLoadsCorrectly_FromName(String alc, String assemblyName, String culture) [06:25:43] info: at System.Reflection.RuntimeMethodInfo.InvokeWorker(Object obj, BindingFlags invokeAttr, Span`1 parameters) ``` * [wasm] Disable random test ordering for `System.Runtime.Numerics` Randomized runs seem to fail with: ``` [03:29:54] info: Starting: System.Runtime.Numerics.Tests.dll [03:29:58] fail: [FAIL] System.Numerics.Tests.cast_toTest.RunDoubleExplicitCastToBigIntegerTests [03:29:58] info: Assert.Equal() Failure [03:29:58] info: ↓ (pos 0) [03:29:58] info: Expected: -0 [03:29:58] info: Actual: 0 [03:29:58] info: ↑ (pos 0) [03:29:58] info: at System.Numerics.Tests.cast_toTest.VerifyDoubleExplicitCastToBigInteger(Double value) [03:29:58] info: at System.Numerics.Tests.cast_toTest.RunDoubleExplicitCastToBigIntegerTests() [03:29:58] info: at System.Reflection.RuntimeMethodInfo.InvokeWorker(Object obj, BindingFlags invokeAttr, Span`1 parameters) ``` Reproducible with `XUNIT_RANDOM_SEED_ORDER=1883302047`. * Add issues for the failing tests
1 parent a01fe0b commit cacb3bf

File tree

11 files changed

+138
-3
lines changed

11 files changed

+138
-3
lines changed

eng/testing/WasmRunnerTemplate.cmd

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,10 @@ if [%XHARNESS_ARGS%] == [] (
5757
set "XHARNESS_ARGS=%JS_ENGINE% %JS_ENGINE_ARGS% %BROWSER_PATH% %MAIN_JS%"
5858
)
5959

60+
if [%XUNIT_RANDOM_ORDER_SEED%] NEQ [] (
61+
set "WasmXHarnessMonoArgs=%WasmXHarnessMonoArgs% --setenv=XUNIT_RANDOM_ORDER_SEED=%XUNIT_RANDOM_ORDER_SEED%"
62+
)
63+
6064
echo EXECUTION_DIR=%EXECUTION_DIR%
6165
echo SCENARIO=%SCENARIO%
6266
echo XHARNESS_OUT=%XHARNESS_OUT%

eng/testing/WasmRunnerTemplate.sh

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,10 @@ if [[ -z "$XHARNESS_ARGS" ]]; then
5353
XHARNESS_ARGS="$JS_ENGINE $JS_ENGINE_ARGS $MAIN_JS"
5454
fi
5555

56+
if [[ -n "$XUNIT_RANDOM_ORDER_SEED" ]]; then
57+
WasmXHarnessMonoArgs="${WasmXHarnessMonoArgs} --setenv=XUNIT_RANDOM_ORDER_SEED=${XUNIT_RANDOM_ORDER_SEED}"
58+
fi
59+
5660
echo EXECUTION_DIR=$EXECUTION_DIR
5761
echo SCENARIO=$SCENARIO
5862
echo XHARNESS_OUT=$XHARNESS_OUT
@@ -64,7 +68,6 @@ echo JS_ENGINE=$JS_ENGINE
6468
echo JS_ENGINE_ARGS=$JS_ENGINE_ARGS
6569
echo XHARNESS_ARGS=$XHARNESS_ARGS
6670

67-
6871
pushd $EXECUTION_DIR
6972

7073
# ========================= BEGIN Test Execution =============================
@@ -83,4 +86,4 @@ echo ----- end $(date) ----- exit code $_exitCode ------------------------------
8386

8487
echo "XHarness artifacts: $XHARNESS_OUT"
8588

86-
exit $_exitCode
89+
exit $_exitCode

eng/testing/tests.mobile.targets

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,10 @@
5151
<AdditionalXHarnessArguments Condition="'$(XUnitClassName)' != ''">$(AdditionalXHarnessArguments) -- -c=$(XUnitClassName)</AdditionalXHarnessArguments>
5252
</PropertyGroup>
5353

54+
<ItemGroup Condition="'$(XUnitUseRandomizedTestOrderer)' == 'true'">
55+
<Compile Include="$(RepoRoot)src\libraries\Common\tests\Tests\RandomizedTestOrderAssemblyInfo.cs" />
56+
</ItemGroup>
57+
5458
<UsingTask Condition="'$(RunAOTCompilation)' == 'true'" TaskName="MonoAOTCompiler" AssemblyFile="$(MonoAOTCompilerTasksAssemblyPath)" />
5559
<Import Condition="'$(RunAOTCompilation)' == 'true'" Project="$(MonoAOTCompilerDir)MonoAOTCompiler.props" />
5660

eng/testing/tests.wasm.targets

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
<_ShellCommandSeparator Condition="'$(OS)' != 'Windows_NT'">&amp;&amp;</_ShellCommandSeparator>
1616
<WasmNativeStrip>false</WasmNativeStrip>
1717
<_WasmMainJSFileName Condition="'$(WasmMainJSPath)' != ''">$([System.IO.Path]::GetFileName('$(WasmMainJSPath)'))</_WasmMainJSFileName>
18+
<XUnitUseRandomizedTestOrderer Condition="'$(XUnitUseRandomizedTestOrderer)' == '' and '$(IsTestProject)' == 'true'">true</XUnitUseRandomizedTestOrderer>
1819
</PropertyGroup>
1920

2021
<PropertyGroup>
Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
// Licensed to the .NET Foundation under one or more agreements.
2+
// The .NET Foundation licenses this file to you under the MIT license.
3+
4+
using System;
5+
using System.Collections.Generic;
6+
using System.Diagnostics.CodeAnalysis;
7+
using System.Linq;
8+
using System.Threading;
9+
using Xunit.Abstractions;
10+
using Xunit.Sdk;
11+
12+
#nullable enable
13+
14+
namespace TestUtilities;
15+
16+
// Based on https://github.com/xunit/xunit/blob/v2/src/xunit.execution/Sdk/DefaultTestCaseOrderer.cs
17+
18+
public class RandomTestCaseOrderer : ITestCaseOrderer
19+
{
20+
public const string RandomSeedEnvironmentVariableName = "XUNIT_RANDOM_ORDER_SEED";
21+
22+
public static readonly Lazy<int> LazySeed = new (GetSeed, LazyThreadSafetyMode.ExecutionAndPublication);
23+
private readonly IMessageSink _diagnosticMessageSink;
24+
25+
private static int GetSeed()
26+
{
27+
string? seedEnvVar = Environment.GetEnvironmentVariable(RandomSeedEnvironmentVariableName);
28+
if (string.IsNullOrEmpty(seedEnvVar) || !int.TryParse(seedEnvVar, out int seed))
29+
{
30+
seed = new Random().Next();
31+
}
32+
33+
return seed;
34+
}
35+
36+
public RandomTestCaseOrderer(IMessageSink diagnosticMessageSink)
37+
{
38+
diagnosticMessageSink.OnMessage(new DiagnosticMessage($"Using random seed for test cases: {LazySeed.Value}"));
39+
_diagnosticMessageSink = diagnosticMessageSink;
40+
}
41+
42+
public IEnumerable<TTestCase> OrderTestCases<TTestCase>(IEnumerable<TTestCase> testCases) where TTestCase : ITestCase
43+
=> TryRandomize(testCases.ToList(), _diagnosticMessageSink, out List<TTestCase>? randomizedTests)
44+
? randomizedTests
45+
: testCases;
46+
47+
public static bool TryRandomize<T>(List<T> tests, IMessageSink messageSink, [NotNullWhen(true)] out List<T>? randomizedTests)
48+
{
49+
randomizedTests = null;
50+
try
51+
{
52+
randomizedTests = Randomize(tests.ToList());
53+
return true;
54+
}
55+
catch (Exception ex)
56+
{
57+
messageSink.OnMessage(new DiagnosticMessage($"Failed to randomize test cases: {ex}"));
58+
return false;
59+
}
60+
61+
static List<T> Randomize(List<T> tests)
62+
{
63+
var result = new List<T>(tests.Count);
64+
65+
var randomizer = new Random(LazySeed.Value);
66+
67+
while (tests.Count > 0)
68+
{
69+
int next = randomizer.Next(tests.Count);
70+
result.Add(tests[next]);
71+
tests.RemoveAt(next);
72+
}
73+
74+
return result;
75+
}
76+
}
77+
}
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
// Licensed to the .NET Foundation under one or more agreements.
2+
// The .NET Foundation licenses this file to you under the MIT license.
3+
4+
using System.Collections.Generic;
5+
using System.Linq;
6+
using Xunit;
7+
using Xunit.Abstractions;
8+
using Xunit.Sdk;
9+
10+
#nullable enable
11+
12+
namespace TestUtilities;
13+
14+
public class RandomTestCollectionOrderer : ITestCollectionOrderer
15+
{
16+
private readonly IMessageSink _diagnosticMessageSink;
17+
18+
public RandomTestCollectionOrderer(IMessageSink diagnosticMessageSink)
19+
{
20+
diagnosticMessageSink.OnMessage(new DiagnosticMessage(
21+
$"Using random seed for collections: {RandomTestCaseOrderer.LazySeed.Value}"));
22+
_diagnosticMessageSink = diagnosticMessageSink;
23+
}
24+
25+
public IEnumerable<ITestCollection> OrderTestCollections(IEnumerable<ITestCollection> testCollections)
26+
=> RandomTestCaseOrderer.TryRandomize(testCollections.ToList(), _diagnosticMessageSink, out List<ITestCollection>? randomizedTests)
27+
? randomizedTests
28+
: testCollections;
29+
}

src/libraries/Common/tests/TestUtilities/TestUtilities.csproj

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,9 @@
4343
variant from the Common folder and adding the missing members manually.
4444
-->
4545
<Compile Include="Interop\Interop.Libraries.cs" />
46+
47+
<Compile Include="RandomTestCaseOrderer.cs" />
48+
<Compile Include="RandomTestCollectionOrderer.cs" />
4649
</ItemGroup>
4750
<!-- Windows imports -->
4851
<ItemGroup>
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
// Licensed to the .NET Foundation under one or more agreements.
2+
// The .NET Foundation licenses this file to you under the MIT license.
3+
4+
using Xunit;
5+
6+
[assembly: TestCaseOrderer("TestUtilities.RandomTestCaseOrderer", "TestUtilities")]
7+
[assembly: TestCollectionOrderer("TestUtilities.RandomTestCollectionOrderer", "TestUtilities")]

src/libraries/System.Private.Xml/tests/Writers/XmlWriterApi/System.Xml.RW.XmlWriterApi.Tests.csproj

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
<Project Sdk="Microsoft.NET.Sdk">
22
<PropertyGroup>
33
<TargetFrameworks>$(NetCoreAppCurrent)</TargetFrameworks>
4+
<XUnitUseRandomizedTestOrderer>false</XUnitUseRandomizedTestOrderer>
45
</PropertyGroup>
56
<ItemGroup>
67
<Compile Include="EndOfLineHandlingTests.cs" />

src/libraries/System.Runtime.Loader/tests/System.Runtime.Loader.Tests.csproj

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,9 @@
88
<GenerateDependencyFile>false</GenerateDependencyFile>
99
<!-- EnC tests on targets without a remote executor need the environment variable set before launching the test -->
1010
<WasmXHarnessMonoArgs>--setenv=DOTNET_MODIFIABLE_ASSEMBLIES=debug</WasmXHarnessMonoArgs>
11+
12+
<!-- disabled due to https://github.com/dotnet/runtime/issues/65672 -->
13+
<XUnitUseRandomizedTestOrderer>false</XUnitUseRandomizedTestOrderer>
1114
</PropertyGroup>
1215
<ItemGroup>
1316
<Compile Include="ApplyUpdateTest.cs" />

src/libraries/System.Runtime.Numerics/tests/System.Runtime.Numerics.Tests.csproj

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,9 @@
22
<PropertyGroup>
33
<IncludeRemoteExecutor>true</IncludeRemoteExecutor>
44
<TargetFrameworks>$(NetCoreAppCurrent)</TargetFrameworks>
5+
6+
<!-- disabled due to https://github.com/dotnet/runtime/issues/65671 -->
7+
<XUnitUseRandomizedTestOrderer>false</XUnitUseRandomizedTestOrderer>
58
</PropertyGroup>
69
<ItemGroup>
710
<Compile Include="BigInteger\absolutevalue.cs" />
@@ -51,4 +54,4 @@
5154
<Compile Include="BigInteger\TryWriteBytes.cs" />
5255
<Compile Include="ComplexTests.cs" />
5356
</ItemGroup>
54-
</Project>
57+
</Project>

0 commit comments

Comments
 (0)