[mono][wasm] System.Numerics.Vectors.Tests assert equal failure

[kotlarmilos]

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=237850&view=results
Build error leg or test failing: System.Numerics.Vectors.Tests
Pull request: #84304

Error log

ChromeDriver was started successfully.
[11:20:10] fail: [out of order message from the browser]: http://127.0.0.1:49236/runArgs.json - Failed to load resource: the server responded with a status of 404 (Not Found)
[11:20:10] info: could not load /runArgs.json: 404. Ignoring
[11:20:10] info: Incoming arguments: --run WasmTestRunner.dll System.Numerics.Vectors.Tests.dll -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing
[11:20:10] info: Application arguments: --run WasmTestRunner.dll System.Numerics.Vectors.Tests.dll -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing
[11:20:10] info: Silently starting separate runtime instance as another ES6 module to populate caches...
[11:20:11] info: Separate runtime instance finished loading.
[11:20:11] info: browser: [main] Console websocket connected.
[11:20:12] info: Initializing dotnet version 8.0.0-ci commit hash 4c2ed4d6642249ee0064572b9a06dc7091b9ae8a
[11:20:12] info: Discovering: System.Numerics.Vectors.Tests.dll (method display = ClassAndMethod, method display options = None)
[11:20:13] info: Discovered:  System.Numerics.Vectors.Tests.dll (found 1478 of 1479 test cases)
[11:20:13] info: Using random seed for test cases: 2085111559
[11:20:13] info: Using random seed for collections: 2085111559
[11:20:13] info: Starting:    System.Numerics.Vectors.Tests.dll
[11:20:13] fail: [FAIL] System.Numerics.Tests.GenericVectorTests.BitwiseOrOperatorInt64
[11:20:13] info: Assert.Equal() Failure
[11:20:13] info: Expected: <-1286865121, -808206021>
[11:20:13] info: Actual:   <-1286865121, -808206021>
[11:20:13] info:    at System.Reflection.MethodInvoker.Invoke(Object , IntPtr* , BindingFlags )
[11:20:13] fail: [FAIL] System.Numerics.Tests.GenericVectorTests.BitwiseAndOperatorUInt64
[11:20:13] info: Assert.Equal() Failure
[11:20:13] info: Expected: <1652044811, 18446744071795266485>
[11:20:13] info: Actual:   <1652044811, 18446744071795266485>
[11:20:13] info:    at System.Reflection.MethodInvoker.Invoke(Object , IntPtr* , BindingFlags )
[11:20:13] fail: [FAIL] System.Numerics.Tests.GenericVectorTests.BitwiseAndOperatorInt16
[11:20:13] info: Assert.Equal() Failure
[11:20:13] info: Expected: <-6458, -23036, 5574, -31615, -4495, -4379, 19540, 27750>
[11:20:13] info: Actual:   <-6458, -23036, 5574, -31615, -4495, -4379, 19540, 27750>
[11:20:13] info:    at System.Reflection.MethodInvoker.Invoke(Object , IntPtr* , BindingFlags )
[11:20:13] fail: [FAIL] System.Numerics.Tests.GenericVectorTests.BitwiseOrOperatorInt32
[11:20:13] info: Assert.Equal() Failure
[11:20:13] info: Expected: <-1986089271, 551676616, 604077146, -1410623614>
[11:20:13] info: Actual:   <-1986089271, 551676616, 604077146, -1410623614>
[11:20:13] info:    at System.Reflection.MethodInvoker.Invoke(Object , IntPtr* , BindingFlags )
[11:20:13] fail: [FAIL] System.Numerics.Tests.GenericVectorTests.BitwiseOrOperatorSByte
[11:20:13] info: Assert.Equal() Failure
[11:20:13] info: Expected: <-41, 31, -61, 78, -7, -35, -34, -90, 80, 85, -31, 77, 1, 30, -106, -113>
[11:20:13] info: Actual:   <-41, 31, -61, 78, -7, -35, -34, -90, 80, 85, -31, 77, 1, 30, -106, -113>
[11:20:13] info:    at System.Reflection.MethodInvoker.Invoke(Object , IntPtr* , BindingFlags )
[11:20:13] fail: [FAIL] System.Numerics.Tests.GenericVectorTests.BitwiseAndOperatorSByte
[11:20:13] info: Assert.Equal() Failure
[11:20:13] info: Expected: <86, -21, 36, -39, 76, 75, -124, 32, 23, 28, 120, -97, 86, -37, -47, 2>
[11:20:13] info: Actual:   <86, -21, 36, -39, 76, 75, -124, 32, 23, 28, 120, -97, 86, -37, -47, 2>
[11:20:13] info:    at System.Reflection.MethodInvoker.Invoke(Object , IntPtr* , BindingFlags )
[11:20:13] fail: [FAIL] System.Numerics.Tests.GenericVectorTests.BitwiseOrOperatorByte
[11:20:13] info: Assert.Equal() Failure
[11:20:13] info: Expected: <168, 50, 210, 249, 28, 148, 158, 17, 119, 87, 137, 156, 43, 230, 229, 120>
[11:20:13] info: Actual:   <168, 50, 210, 249, 28, 148, 158, 17, 119, 87, 137, 156, 43, 230, 229, 120>
[11:20:13] info:    at System.Reflection.MethodInvoker.Invoke(Object , IntPtr* , BindingFlags )
[11:20:14] fail: [FAIL] System.Numerics.Tests.GenericVectorTests.BitwiseOrOperatorInt16
[11:20:14] info: Assert.Equal() Failure
[11:20:14] info: Expected: <986, 24220, -32476, -1101, 11435, 15906, -6787, 6373>
[11:20:14] info: Actual:   <986, 24220, -32476, -1101, 11435, 15906, -6787, 6373>
[11:20:14] info:    at System.Reflection.MethodInvoker.Invoke(Object , IntPtr* , BindingFlags )
[11:20:14] fail: [FAIL] System.Numerics.Tests.GenericVectorTests.BitwiseAndOperatorUInt32
[11:20:14] info: Assert.Equal() Failure
[11:20:14] info: Expected: <3339095750, 1843094543, 3237790221, 678923124>
[11:20:14] info: Actual:   <3339095750, 1843094543, 3237790221, 678923124>
[11:20:14] info:    at System.Reflection.MethodInvoker.Invoke(Object , IntPtr* , BindingFlags )
[11:20:14] fail: [FAIL] System.Numerics.Tests.GenericVectorTests.BitwiseAndOperatorByte
[11:20:14] info: Assert.Equal() Failure
[11:20:14] info: Expected: <70, 99, 68, 164, 5, 75, 210, 111, 118, 244, 78, 197, 35, 217, 62, 216>
[11:20:14] info: Actual:   <70, 99, 68, 164, 5, 75, 210, 111, 118, 244, 78, 197, 35, 217, 62, 216>
[11:20:14] info:    at System.Reflection.MethodInvoker.Invoke(Object , IntPtr* , BindingFlags )
[11:20:14] fail: [FAIL] System.Numerics.Tests.GenericVectorTests.BitwiseAndOperatorUInt16
[11:20:14] info: Assert.Equal() Failure
[11:20:14] info: Expected: <38043, 46622, 39384, 50631, 44106, 26552, 8632, 34025>
[11:20:14] info: Actual:   <38043, 46622, 39384, 50631, 44106, 26552, 8632, 34025>
[11:20:14] info:    at System.Reflection.MethodInvoker.Invoke(Object , IntPtr* , BindingFlags )
[11:20:14] fail: [FAIL] System.Numerics.Tests.GenericVectorTests.BitwiseOrOperatorUInt32
[11:20:14] info: Assert.Equal() Failure
[11:20:14] info: Expected: <1581228415, 3572260203, 930171099, 614517734>
[11:20:14] info: Actual:   <1581228415, 3572260203, 930171099, 614517734>
[11:20:14] info:    at System.Reflection.MethodInvoker.Invoke(Object , IntPtr* , BindingFlags )
[11:20:14] fail: [FAIL] System.Numerics.Tests.GenericVectorTests.BitwiseOrOperatorUInt64
[11:20:14] info: Assert.Equal() Failure
[11:20:14] info: Expected: <18446744073678956429, 18446744073405669760>
[11:20:14] info: Actual:   <18446744073678956429, 18446744073405669760>
[11:20:14] info:    at System.Reflection.MethodInvoker.Invoke(Object , IntPtr* , BindingFlags )
[11:20:14] fail: [FAIL] System.Numerics.Tests.GenericVectorTests.BitwiseAndOperatorInt64
[11:20:14] info: Assert.Equal() Failure
[11:20:14] info: Expected: <-1769174059, -820037550>
[11:20:14] info: Actual:   <-1769174059, -820037550>
[11:20:14] info:    at System.Reflection.MethodInvoker.Invoke(Object , IntPtr* , BindingFlags )
[11:20:14] fail: [FAIL] System.Numerics.Tests.GenericVectorTests.BitwiseOrOperatorUInt16
[11:20:14] info: Assert.Equal() Failure
[11:20:14] info: Expected: <42419, 55258, 57723, 4772, 53371, 47362, 5935, 49970>
[11:20:14] info: Actual:   <42419, 55258, 57723, 4772, 53371, 47362, 5935, 49970>
[11:20:14] info:    at System.Reflection.MethodInvoker.Invoke(Object , IntPtr* , BindingFlags )
[11:20:14] fail: [FAIL] System.Numerics.Tests.GenericVectorTests.BitwiseAndOperatorSingle
[11:20:14] info: Assert.Equal() Failure
[11:20:14] info: Expected: <1.7061224E+09, 1.159023E+09, 2.13021E+09, -1.01173203E+09>
[11:20:14] info: Actual:   <1.7061224E+09, 1.159023E+09, 2.13021E+09, -1.01173203E+09>
[11:20:14] info:    at System.Reflection.MethodInvoker.Invoke(Object , IntPtr* , BindingFlags )
[11:20:14] fail: [FAIL] System.Numerics.Tests.GenericVectorTests.BitwiseAndOperatorDouble
[11:20:14] info: Assert.Equal() Failure
[11:20:14] info: Expected: <640540418, 2047298097>
[11:20:14] info: Actual:   <640540418, 2047298097>
[11:20:14] info:    at System.Reflection.MethodInvoker.Invoke(Object , IntPtr* , BindingFlags )
[11:20:14] fail: [FAIL] System.Numerics.Tests.GenericVectorTests.BitwiseAndOperatorInt32
[11:20:14] info: Assert.Equal() Failure
[11:20:14] info: Expected: <481920175, 1337538156, -2008151338, 1369391235>
[11:20:14] info: Actual:   <481920175, 1337538156, -2008151338, 1369391235>
[11:20:14] info:    at System.Reflection.MethodInvoker.Invoke(Object , IntPtr* , BindingFlags )
[11:20:17] info: Finished:    System.Numerics.Vectors.Tests.dll
[11:20:17] info: 
[11:20:17] info: === TEST EXECUTION SUMMARY ===
[11:20:17] info: Total: 1514, Errors: 0, Failed: 18, Skipped: 0, Time: 4.6570999s
[11:20:17] info: 
[11:20:17] info: Received expected 304956 of C:\helix\work\workitem\uploads\xharness-output\testResults.xml
[11:20:17] info: Finished writing 304956 bytes of RESULTXML
[11:20:17] info: Xml file was written to the provided writer.
[11:20:17] info: Tests run: 1514 Passed: 1496 Inconclusive: 0 Failed: 18 Ignored: 0 Skipped: 0
[11:20:17] fail: {"name":"ExitStatus","message":"Program terminated with exit(1)","status":1}
[11:20:17] info: WASM EXIT 1
[11:20:17] info: Waiting to flush log messages with a timeout of 120 secs ..
[11:20:18] fail: Application has finished with exit code TESTS_FAILED but 0 was expected
XHarness exit code: 71 (GENERAL_FAILURE)

Error Message

Fill the error message using known issues guidance.

{
  "ErrorMessage": "[FAIL] System.Numerics.Tests.GenericVectorTests.BitwiseOrOperatorInt64",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Report

Build	Definition	Step Name	Console log	Pull Request
279490	dotnet/runtime	Run Trimming Tests	Log	#85558
278937	dotnet/runtime	Run Trimming Tests	Log	#85558
278833	dotnet/runtime	Run Trimming Tests	Log	#85558
270017	dotnet/runtime	Run Trimming Tests	Log	#85373
269693	dotnet/runtime	Run Trimming Tests	Log	#85373
268564	dotnet/runtime	Run Trimming Tests	Log	#85373

Build	Definition	Test	Pull Request
279286	dotnet/runtime	WasmTestOnBrowser-Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.WorkItemExecution	#86485
279238	dotnet/runtime	WasmTestOnBrowser-Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.WorkItemExecution	#86485
278935	dotnet/runtime	Wasm.Build.Tests.IcuShardingTests.DefaultAvailableIcuShardsFromRuntimePack	#85558
277665	dotnet/runtime	WasmTestOnBrowser-Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.WorkItemExecution	#86408
277466	dotnet/runtime	WasmTestOnBrowser-System.Memory.Tests.WorkItemExecution	#86357
277000	dotnet/runtime	WasmTestOnBrowser-Microsoft.Extensions.Options.Tests.WorkItemExecution	#86394
276719	dotnet/runtime	normal-Hybrid.WASM.Tests.WorkItemExecution	#85254
276443	dotnet/runtime	DebuggerTests.EvaluateOnCallFrameTests.DebuggerTests.EvaluateOnCallFrameTests.EvaluateIndexingsByConstant	#84191
276439	dotnet/runtime	DebuggerTests.EvaluateOnCallFrameTests.DebuggerTests.EvaluateOnCallFrameTests.EvaluateSimpleMethodCallsWithVariableParms	#84191
276117	dotnet/runtime	normal-System.Memory.Tests.WorkItemExecution	#86357
275488	dotnet/runtime	Wasm.Build.NativeRebuild.Tests.ReferenceNewAssemblyRebuildTest.ReferenceNewAssembly	#85920
275263	dotnet/runtime	Wasm.Build.NativeRebuild.Tests.ReferenceNewAssemblyRebuildTest.ReferenceNewAssembly	#85920
275401	dotnet/runtime	Workloads-Webcil-Wasm.Build.Tests.BuildPublishTests.WorkItemExecution	#86035
275060	dotnet/runtime	normal-System.Linq.Tests.WorkItemExecution	#85811
275017	dotnet/runtime	normal-System.Globalization.Tests.WorkItemExecution	#86305
274980	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#81632
274652	dotnet/runtime	Wasm.Build.NativeRebuild.Tests.ReferenceNewAssemblyRebuildTest.ReferenceNewAssembly	#85920
274656	dotnet/runtime	normal-System.Runtime.Tests.WorkItemExecution	#86287
274345	dotnet/runtime	Interop.WorkItemExecution
273844	dotnet/runtime	Wasm.Build.NativeRebuild.Tests.ReferenceNewAssemblyRebuildTest.ReferenceNewAssembly	#85920
273579	dotnet/runtime	normal-System.Text.Json.Tests.WorkItemExecution	#84688
271862	dotnet/runtime	WasmTestOnBrowser-System.Reflection.MetadataLoadContext.Tests.WorkItemExecution	#86136
271291	dotnet/runtime	WasmTestOnBrowser-System.Reflection.MetadataLoadContext.Tests.WorkItemExecution	#86136
271126	dotnet/runtime	Interop.WorkItemExecution	#86127
271001	dotnet/runtime	WasmTestOnBrowser-System.IO.Compression.ZipFile.Tests.WorkItemExecution	#85491
269752	dotnet/runtime	WasmTestOnBrowser-System.Memory.Tests.WorkItemExecution	#85920
269233	dotnet/runtime	Workloads-Wasm.Build.NativeRebuild.Tests.SimpleSourceChangeRebuildTest.WorkItemExecution	#85730
269078	dotnet/runtime	WasmTestOnBrowser-System.Net.WebSockets.Client.Tests.WorkItemExecution	#85317
268941	dotnet/runtime	Workloads-Webcil-Wasm.Build.NativeRebuild.Tests.ReferenceNewAssemblyRebuildTest.WorkItemExecution	#85730
268723	dotnet/runtime	normal-System.Memory.Tests.WorkItemExecution	#85920
268118	dotnet/runtime	WasmTestOnBrowser-System.Net.Http.Functional.Tests.WorkItemExecution	#86007
268070	dotnet/runtime	normal-System.Memory.Tests.WorkItemExecution	#85920
267534	dotnet/runtime	Workloads-Wasm.Build.NativeRebuild.Tests.ReferenceNewAssemblyRebuildTest.WorkItemExecution	#85730
266815	dotnet/runtime	normal-System.Xml.Linq.SDMSample.Tests.WorkItemExecution	#85858
266368	dotnet/runtime	normal-System.Runtime.Extensions.Tests.WorkItemExecution	#85920
266269	dotnet/runtime	Workloads-Webcil-Wasm.Build.Tests.WasmTemplateTests.WorkItemExecution	#85932
265911	dotnet/runtime	System.Buffers.Text.Tests.Base64EncoderUnitTests.BasicEncodingWithFinalBlockFalse	#85920

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	43

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=237850&view=results
Build error leg or test failing: System.Numerics.Vectors.Tests
Pull request: #84304

Error Message

Fill the error message using known issues guidance.

{
  "ErrorMessage": "[FAIL] System.Numerics.Tests.GenericVectorTests.BitwiseAndOperatorUInt64",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Author:	kotlarmilos
Assignees:	-
Labels:	blocking-clean-ci, area-Infrastructure-mono, needs-area-label
Milestone:	8.0.0

[jandupej]

Looking at the log, the tests actually seem to generate the right values. Could the equality operator be broken? I had a similar problem in https://github.com/dotnet/runtime/pull/83515/files where True was generated as any non-zero value by the intrinsic, while the tests were expecting a hard 1 for True.

[vargaz]

Is this a random failure ?

[lewing]

cc @kg @BrzVlad @radekdoulik

[kg]

As far as I know simd operations on WASM are supposed to produce all 1-bits for true and all 0-bits for false, anything else is incorrect: https://github.com/WebAssembly/simd/blob/master/proposals/simd/SIMD.md#operations
I thought this was how it worked on x86/x64 as well but I've never looked into it. I can't tell from looking at the existing failures in the history whether they were running in AOT mode (where wasm SIMD is enabled) or interp mode (where it's not).

[kg]

Also I should note any time NaNs come into play the actual bits of those floats are unspecified in wasm, so doing bitwise comparisons on float vectors is incorrect on WASM and you need to specifically do f32/f64 compares. But it doesn't look like that would be happening here.

[vargaz]

Testcase:

using System;
using System.Runtime.InteropServices;
using System.Reflection;
using System.Numerics;
using System.Runtime.CompilerServices;
using Xunit;

public class Tests
{
        private static void TestBitwiseOrOperator<T>(Vector<T> v) where T : struct
        {
            Assert.Equal(v, v);
        }

    public static void Main () {
        TestBitwiseOrOperator<int>(new Vector<int> (new int [] { 1, 2, 3, 4, 5, 6, 7, 8 }));
    }
}

Add to csproj:

     <ItemGroup>
       <PackageReference Include="xunit.assert" Version="2.4.2"/>
	 </ItemGroup>

This seems to only happen in mixed mode, TestBitwiseOrOperator is AOT-ed but Assert.Equal is not.

[vargaz]

The problem seems to occur when the interpreter makes a call to the AOT-ed Vector.Equals (Vector) method, the argument offset seems to be off, i.e. it doesn't point to the <1, 2, 3, 4> data on the stack. Maybe because Vector has 16 byte alignment, and an interpreter stack val is only 8 bytes ?
@BrzVlad