Test failure: Mismatch in align instruction

[BruceForstall]

runtime-coreclr libraries-pgo
System.Linq.Tests
net8.0-windows-Release-x64-CoreCLR_checked-fullpgo_random_gdv_edge

https://dev.azure.com/dnceng-public/public/_build/results?buildId=263808&view=ms.vss-test-web.build-test-results-tab&runId=5188634&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab&resultId=199194

C:\h\w\BEBA0A63\w\AD3A0987\e>set DOTNET 
DOTNET_JitRandomEdgeCounts=1
DOTNET_JitRandomGuardedDevirtualization=1
DOTNET_JitRandomlyCollect64BitCounts=1
DOTNET_ReadyToRun=0
DOTNET_TC_QuickJitForLoops=1
DOTNET_TieredCompilation=1
DOTNET_TieredPGO=1

C:\h\w\BEBA0A63\w\AD3A0987\e>call RunTests.cmd --runtime-path C:\h\w\BEBA0A63\p 
----- start Fri 05/05/2023 19:11:17.35 ===============  To repro directly: ===================================================== 
pushd C:\h\w\BEBA0A63\w\AD3A0987\e\
"C:\h\w\BEBA0A63\p\dotnet.exe" exec --runtimeconfig System.Linq.Tests.runtimeconfig.json --depsfile System.Linq.Tests.deps.json xunit.console.dll System.Linq.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================

C:\h\w\BEBA0A63\w\AD3A0987\e>"C:\h\w\BEBA0A63\p\dotnet.exe" exec --runtimeconfig System.Linq.Tests.runtimeconfig.json --depsfile System.Linq.Tests.deps.json xunit.console.dll System.Linq.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Linq.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Linq.Tests (found 1728 of 1732 test cases)
  Starting:    System.Linq.Tests (parallel test collections = on, max threads = 4)
    System.Linq.Tests.ToArrayTests.ToArray_FailOnExtremelyLargeCollection [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Linq.Tests.WhereTests.IndexOverflows [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Linq.Tests.SelectManyTests.IndexOverflow [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Linq.Tests.SkipWhileTests.IndexSkipWhileOverflowBeyondIntMaxValueElements [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Linq.Tests.TakeWhileTests.IndexTakeWhileOverflowBeyondIntMaxValueElements [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
Mismatch in align instruction.
Containing IG: IG98
loopHeadPredIG: IG113
loopHeadIG: IG114
igInLoop: IG116
igInLoop->backEdge: IG116
igInLoop has align instruction for : IG117
Loop:
	IG114
	IG115
	IG116
	IG117
	IG118
	IG119
	IG120
	IG121
	IG122
	IG123
	IG124
	IG125
	IG126
	IG127
	IG128
	IG129
	IG130
	IG131
	IG132
	IG133
	IG134
	IG135
	IG136
	IG137
	IG138
	IG139
	IG140
	IG141
	IG142
	IG143
	IG144
	IG145
	IG146
	IG147
	IG148
	IG149
	IG150
	IG151
	IG152
	IG153
	IG154
	IG155
	IG156
	IG157
	IG158
	IG159
	IG160
	IG161
	IG162
	IG163
	IG164
	IG165
	IG166
	IG167
	IG168
	IG169
	IG170
	IG171
	IG172
	IG173
	IG174
	IG175
	IG176
	IG177
	IG178
	IG179
	IG180
	IG181
	IG182
	IG183
	IG184
	IG185
	IG186
	IG187
	IG188
	IG189
	IG190
	IG191
	IG192
	IG193
	IG194
	IG195
	IG196
	IG197
	IG198
	IG199
	IG200
	IG201
	IG202
	IG203
	IG204
	IG205
	IG206
	IG207
	IG208
	IG209
	IG210
	IG211
	IG212
	IG213
	IG214
	IG215
	IG216
	IG217
	IG218
	IG219
	IG220
	IG221
	IG222
	IG223
	IG224
	IG225
	IG226
	IG227
	IG228
	IG229
	IG230
	IG231
	IG232
	IG233
	IG234
	IG235
	IG236
	IG237
	IG238
	IG239
	IG240
	IG241
	IG242
	IG243
	IG244

Assert failure(PID 4576 [0x000011e0], Thread: 4208 [0x1070]): Assertion failed 'false' in 'System.Number:Dragon4(ulong,int,uint,bool,int,bool,System.Span`1[ubyte],byref):uint' during 'Generate code' (IL size 1229; hash 0x413f9f9f; Tier1)

    File: D:\a\_work\1\s\src\coreclr\jit\emit.cpp Line: 5785
    Image: C:\h\w\BEBA0A63\p\dotnet.exe

----- end Fri 05/05/2023 19:12:34.26 ----- exit code -1073740286 ----------------------------------------------------------

@AndyAyersMS @EgorBo @kunalspathak

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

---

runtime-coreclr libraries-pgo
System.Linq.Tests
net8.0-windows-Release-x64-CoreCLR_checked-fullpgo_random_gdv_edge

https://dev.azure.com/dnceng-public/public/_build/results?buildId=263808&view=ms.vss-test-web.build-test-results-tab&runId=5188634&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab&resultId=199194

C:\h\w\BEBA0A63\w\AD3A0987\e>set DOTNET 
DOTNET_JitRandomEdgeCounts=1
DOTNET_JitRandomGuardedDevirtualization=1
DOTNET_JitRandomlyCollect64BitCounts=1
DOTNET_ReadyToRun=0
DOTNET_TC_QuickJitForLoops=1
DOTNET_TieredCompilation=1
DOTNET_TieredPGO=1

C:\h\w\BEBA0A63\w\AD3A0987\e>call RunTests.cmd --runtime-path C:\h\w\BEBA0A63\p 
----- start Fri 05/05/2023 19:11:17.35 ===============  To repro directly: ===================================================== 
pushd C:\h\w\BEBA0A63\w\AD3A0987\e\
"C:\h\w\BEBA0A63\p\dotnet.exe" exec --runtimeconfig System.Linq.Tests.runtimeconfig.json --depsfile System.Linq.Tests.deps.json xunit.console.dll System.Linq.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================

C:\h\w\BEBA0A63\w\AD3A0987\e>"C:\h\w\BEBA0A63\p\dotnet.exe" exec --runtimeconfig System.Linq.Tests.runtimeconfig.json --depsfile System.Linq.Tests.deps.json xunit.console.dll System.Linq.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Linq.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Linq.Tests (found 1728 of 1732 test cases)
  Starting:    System.Linq.Tests (parallel test collections = on, max threads = 4)
    System.Linq.Tests.ToArrayTests.ToArray_FailOnExtremelyLargeCollection [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Linq.Tests.WhereTests.IndexOverflows [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Linq.Tests.SelectManyTests.IndexOverflow [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Linq.Tests.SkipWhileTests.IndexSkipWhileOverflowBeyondIntMaxValueElements [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
    System.Linq.Tests.TakeWhileTests.IndexTakeWhileOverflowBeyondIntMaxValueElements [SKIP]
      Condition(s) not met: "IsStressModeEnabled"
Mismatch in align instruction.
Containing IG: IG98
loopHeadPredIG: IG113
loopHeadIG: IG114
igInLoop: IG116
igInLoop->backEdge: IG116
igInLoop has align instruction for : IG117
Loop:
	IG114
	IG115
	IG116
	IG117
	IG118
	IG119
	IG120
	IG121
	IG122
	IG123
	IG124
	IG125
	IG126
	IG127
	IG128
	IG129
	IG130
	IG131
	IG132
	IG133
	IG134
	IG135
	IG136
	IG137
	IG138
	IG139
	IG140
	IG141
	IG142
	IG143
	IG144
	IG145
	IG146
	IG147
	IG148
	IG149
	IG150
	IG151
	IG152
	IG153
	IG154
	IG155
	IG156
	IG157
	IG158
	IG159
	IG160
	IG161
	IG162
	IG163
	IG164
	IG165
	IG166
	IG167
	IG168
	IG169
	IG170
	IG171
	IG172
	IG173
	IG174
	IG175
	IG176
	IG177
	IG178
	IG179
	IG180
	IG181
	IG182
	IG183
	IG184
	IG185
	IG186
	IG187
	IG188
	IG189
	IG190
	IG191
	IG192
	IG193
	IG194
	IG195
	IG196
	IG197
	IG198
	IG199
	IG200
	IG201
	IG202
	IG203
	IG204
	IG205
	IG206
	IG207
	IG208
	IG209
	IG210
	IG211
	IG212
	IG213
	IG214
	IG215
	IG216
	IG217
	IG218
	IG219
	IG220
	IG221
	IG222
	IG223
	IG224
	IG225
	IG226
	IG227
	IG228
	IG229
	IG230
	IG231
	IG232
	IG233
	IG234
	IG235
	IG236
	IG237
	IG238
	IG239
	IG240
	IG241
	IG242
	IG243
	IG244

Assert failure(PID 4576 [0x000011e0], Thread: 4208 [0x1070]): Assertion failed 'false' in 'System.Number:Dragon4(ulong,int,uint,bool,int,bool,System.Span`1[ubyte],byref):uint' during 'Generate code' (IL size 1229; hash 0x413f9f9f; Tier1)

    File: D:\a\_work\1\s\src\coreclr\jit\emit.cpp Line: 5785
    Image: C:\h\w\BEBA0A63\p\dotnet.exe

----- end Fri 05/05/2023 19:12:34.26 ----- exit code -1073740286 ----------------------------------------------------------

@AndyAyersMS @EgorBo @kunalspathak

Author:	BruceForstall
Assignees:	-
Labels:	JitStress, area-CodeGen-coreclr, blocking-clean-ci-optional
Milestone:	8.0.0

[kunalspathak]

This might have exposed with recent changes in #85265

[BruceForstall]

Also 2 failures in net8.0-linux-Release-x64-CoreCLR_checked-fullpgo_random_gdv_edge-Ubuntu.1804.Amd64.Open

[BruceForstall]

fyi, more failures in weekend run:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=265188&view=ms.vss-test-web.build-test-results-tab

[kunalspathak]

From the message printed by JIT, we are inside a loop (IG114) that needs alignment and while trying to find the backedge IG to the top of the loop IG114, we discover another loop IG116 -> IG116 and this should not happen.

I tried to repro this locally but was not able to and I am guessing it could be because there are 3 random switches needed to repro.

DOTNET_JitRandomGuardedDevirtualization=1
DOTNET_JitRandomEdgeCounts=1
DOTNET_JitRandomlyCollect64BitCounts=1

@AndyAyersMS - I am wondering if there is a way to make it deterministic so I can repro this scenario?

[AndyAyersMS]

AFAIK The randomness here is deterministic. But the test execution order and interleaving ar not. You can try limiting the number of CPUs you use to match what the CI machines have

(parallel test collections = on, max threads = 4)

via something like -maxthreads 4 passed to xunit.

[AndyAyersMS]

If you want I can try and repro this...

[kunalspathak]

If you want I can try and repro this...

Sure, if it is easy for you. Previously too we have seen these errors and it was only through dumps that I had to add the printing #83286 to understand the state of blocks. It would be good to live debug this to find the actual cause.

[AndyAyersMS]

I haven't been able to repro yet. Have you looked at the dump? Is this failure while running a test or part of the xunit infrastructure? I kind of suspect the latter; in fact I wonder if xunit is trying to print one of those "long running test" warnings.

Assert failure(PID 4397 [0x0000112d], Thread: 4657 [0x1231]): Assertion failed 'false' in 'System.Number:Dragon4(ulong,int,uint,bool,int,bool,System.Span`1[ubyte],byref):uint' during 'Generate code' (IL size 1229; hash 0x413f9f9f; Tier1

I am going to try running this with my machine loaded down with other stuff and see if that jars anything loose.

[kunalspathak]

Is this failure while running a test or part of the xunit infrastructure?

It happens inside the test System.Number:Dragon4 which could be called from xunit. How do you find out that? !clrstack on the dump is not working. It seems that it is going through tiering. Here is the full stack:

00 KERNELBASE!RaiseFailFastException
01 coreclr!_DbgBreakCheck
02 coreclr!CEEInfo::doAssert
03 clrjit!assertAbort
04 clrjit!emitter::getLoopSize
05 clrjit!emitter::emitCalculatePaddingForLoopAlignment
06 clrjit!emitter::emitLoopAlignAdjustments
07 clrjit!CodeGenPhase::DoPhase
08 clrjit!Phase::Run
09 clrjit!DoPhase
0a clrjit!CodeGen::genGenerateCode
0b clrjit!Compiler::compCompile
0c clrjit!Compiler::compCompileHelper
0d clrjit!Compiler::compCompile::__l170::__Body::Run
0e clrjit!Compiler::compCompile
0f clrjit!jitNativeCode::__l8::__Body::Run::__l6::__Body::Run
10 clrjit!`jitNativeCode'::`8'::__Body::Run
11 clrjit!jitNativeCode
12 clrjit!CILJit::compileMethod
13 coreclr!invokeCompileMethodHelper
14 coreclr!invokeCompileMethod
15 coreclr!UnsafeJitFunction
16 coreclr!MethodDesc::JitCompileCodeLocked
17 coreclr!MethodDesc::JitCompileCodeLockedEventWrapper
18 coreclr!MethodDesc::JitCompileCode
19 coreclr!MethodDesc::PrepareILBasedCode
1a coreclr!MethodDesc::PrepareCode
1b coreclr!TieredCompilationManager::CompileCodeVersion
1c coreclr!TieredCompilationManager::OptimizeMethod
1d coreclr!TieredCompilationManager::DoBackgroundWork
1e coreclr!TieredCompilationManager::BackgroundWorkerStart
1f coreclr!TieredCompilationManager::BackgroundWorkerBootstrapper1
20 coreclr!ManagedThreadBase_DispatchInner
21 coreclr!ManagedThreadBase_DispatchMiddle
22 coreclr!``ManagedThreadBase_DispatchOuter'::`11'::__Body::Run'::`5'::__Body::Run
23 coreclr!`ManagedThreadBase_DispatchOuter'::`11'::__Body::Run
24 coreclr!ManagedThreadBase_DispatchOuter
25 coreclr!ManagedThreadBase_FullTransition
26 coreclr!TieredCompilationManager::BackgroundWorkerBootstrapper0