Test crossgen-comparison Linux arm checked failing

[AntonLapounov]

Failed in #32379 and other recent PRs.
https://helix.dot.net/api/2019-06-17/jobs/50fa1ddf-092f-4146-81e7-2fd73a587551/workitems/WorkItem/console

Comparing crossgen results in "/root/helix/work/workitem/log/Linux.x64_arm.Checked" and "/root/helix/work/workitem/log/Linux.arm.Checked" directories for files of type "NativeOrReadyToRunImage":
File hash sum mismatch for "Microsoft.CodeAnalysis" assembly for files of type "NativeOrReadyToRunImage":
 - "/root/helix/work/workitem/log/Linux.x64_arm.Checked" has "8cdf4235c28b3da04429ca5a762cb8d1f83278cfb1b364c25663a109683a9a2b"
 - "/root/helix/work/workitem/log/Linux.arm.Checked" has "242689aaf24704b75ee64ff34174f7870984773a3173b2c690453b2cdf44670e"
File hash sum mismatch for "Microsoft.Diagnostics.Tracing.TraceEvent" assembly for files of type "NativeOrReadyToRunImage":
 - "/root/helix/work/workitem/log/Linux.x64_arm.Checked" has "92daecb8a3dd0e1a258bc4820162d494eaabfcf048b60de2a0f1f87e51a2d881"
 - "/root/helix/work/workitem/log/Linux.arm.Checked" has "5efc4130f37218df48b2b28422a610b1401b3500883c60502b4b9cc355aa1410"
File hash sum mismatch for "System.Private.CoreLib" assembly for files of type "NativeOrReadyToRunImage":
 - "/root/helix/work/workitem/log/Linux.x64_arm.Checked" has "14dbdbc7d46ba0ac54faecbf4254bae28ebef65f7ad35e315d6876c32711d478"
 - "/root/helix/work/workitem/log/Linux.arm.Checked" has "109f3de73f1bda8db534e628eaa599ee9732b24c08d074a9989302accc76af3b"
File hash sum mismatch for "System.Private.Xml" assembly for files of type "NativeOrReadyToRunImage":
 - "/root/helix/work/workitem/log/Linux.x64_arm.Checked" has "33e549d979e1f7717ee36fbe2bcbca04e0cf54023842da29bf203ad70be786a7"
 - "/root/helix/work/workitem/log/Linux.arm.Checked" has "b1b6bfb9b1db5cea9c53e95f5aa7bf7b7d05c1de89da792e598c4fdcb09fa5ca"
File hash sum mismatch for "System.Runtime.Numerics" assembly for files of type "NativeOrReadyToRunImage":
 - "/root/helix/work/workitem/log/Linux.x64_arm.Checked" has "4fc6759797e9025c59e8c9d50c5f0cb119d4031cfd73f967960687abdc81a520"
 - "/root/helix/work/workitem/log/Linux.arm.Checked" has "6531fd88be53ec650857b2afa43e65142a51d94f233f7fa38cdd25d22c73ed31"
Number of omitted results: 0
Number of mismatched results: 5

According to @echesakovMSFT the last success was https://dev.azure.com/dnceng/public/_build/results?buildId=537160

[echesakov]

I am bisecting the regression

[echesakov]

Tracked down to 017a64a - #2107

I will see tomorrow where these difference in the images occur

cc @BrianBohe

[echesakov]

There is an actual difference in reported debug info on cross-achitecture compiler (x64_arm) v. native compiler (arm) for System.Reflection.Emit.TypeBuilder:DefineGenericParameters(System.String]):System.Reflection.Emit.GenericTypeParameterBuilder[]:this method in System.Private.CoreLib.dll

*************** Variable debug info
-7 live ranges
+6 live ranges
  0(   UNKNOWN) : From 00000000h to 00000012h, in r0
  0(   UNKNOWN) : From 00000012h to 000001B0h, in r5
  1(   UNKNOWN) : From 00000000h to 00000012h, in r1
  1(   UNKNOWN) : From 00000012h to 000001B0h, in r6
  2(   UNKNOWN) : From 00000022h to 00000038h, in r0
-  3(   UNKNOWN) : From 00000076h to 000001A6h, in r8
+  3(   UNKNOWN) : From 00000076h to 000001B0h, in r8
-  3(   UNKNOWN) : From 000001AAh to 000001B0h, in r8
IP mapping count : 3
IL offs PROLOG : 0x00000000 ( STACK_EMPTY )
IL offs 0x0000 : 0x0000000C ( STACK_EMPTY )
IL offs EPILOG : 0x00000046 ( STACK_EMPTY )

And in System.Globalization.CalendricalCalculationsHelper:SumLongSequenceOfPeriodicTerms(double):double

*************** Variable debug info
-3 live ranges
+4 live ranges
  0(   UNKNOWN) : From 00000000h to 00000010h, in f0
  0(   UNKNOWN) : From 00000010h to 0000194Eh, in ST(L-32)
-  1(   UNKNOWN) : From 000000A0h to 000019BEh, in ST(L-34)
+  1(   UNKNOWN) : From 000000A0h to 0000147Eh, in ST(L-34)
+  1(   UNKNOWN) : From 00001482h to 000019BEh, in ST(L-34)
IP mapping count : 2
IL offs PROLOG : 0x00000000 ( STACK_EMPTY )
IL offs EPILOG : 0x000000A4 ( STACK_EMPTY )

I am going to take a look into other assemblies' differences

cc @dotnet/jit-contrib

[BruceForstall]

It looks like these are differences in the recently-added coalescing code:

    if (!m_VariableLiveRanges->empty() && m_VariableLiveRanges->back().m_EndEmitLocation.IsPreviousInsNum(emit) &&
        siVarLoc::Equals(&varLocation, &(m_VariableLiveRanges->back().m_VarLocation)))
    {
        // The variable is being born just after the instruction at which it died.
        // In this case, i.e. an update of the variable's value, we coalesce the live ranges.
        m_VariableLiveRanges->back().m_EndEmitLocation.Init();
    }

from #2107

[echesakov]

Similar differences in Microsoft.CodeAnalysis.dll

*************** Variable debug info
-31 live ranges
+30 live ranges
  0(   UNKNOWN) : From 00000000h to 00000010h, in r0
  0(   UNKNOWN) : From 00000010h to 000011F8h, in r5
  0(   UNKNOWN) : From 00001204h to 00001274h, in r5
  1(   UNKNOWN) : From 00000000h to 00000010h, in r1
  2(   UNKNOWN) : From 00000000h to 00000010h, in r2
  3(   UNKNOWN) : From 00000000h to 00000010h, in r3
  3(   UNKNOWN) : From 00000010h to 00000040h, in r2
  4(   UNKNOWN) : From 00000000h to 00000010h, in sp[12] (1 slot)
  4(   UNKNOWN) : From 00000010h to 00000044h, in r11[16] (1 slot)
  5(   UNKNOWN) : From 00000000h to 00000010h, in sp[16] (1 slot)
  5(   UNKNOWN) : From 00000010h to 0000003Eh, in r11[20] (1 slot)
  6(   UNKNOWN) : From 00000000h to 00000010h, in sp[20] (1 slot)
  6(   UNKNOWN) : From 00000010h to 00000048h, in r11[24] (1 slot)
  7(   UNKNOWN) : From 00000000h to 00000010h, in sp[24] (1 slot)
  7(   UNKNOWN) : From 00000010h to 0000004Eh, in r4
  8(   UNKNOWN) : From 00000000h to 00000010h, in sp[28] (1 slot)
  8(   UNKNOWN) : From 00000010h to 000000DEh, in r11[32] (1 slot)
  8(   UNKNOWN) : From 0000014Ah to 0000014Eh, in r11[32] (1 slot)
  8(   UNKNOWN) : From 00000182h to 00000186h, in r11[32] (1 slot)
  8(   UNKNOWN) : From 000001ACh to 000001B0h, in r11[32] (1 slot)
  9(   UNKNOWN) : From 00000000h to 00000010h, in sp[32] (1 slot)
  9(   UNKNOWN) : From 00000010h to 000011E6h, in r11[36] (1 slot)
  9(   UNKNOWN) : From 00001204h to 00001206h, in r11[36] (1 slot)
  9(   UNKNOWN) : From 00001206h to 00001230h, in r4
  9(   UNKNOWN) : From 00001230h to 00001234h, in r11[36] (1 slot)
  9(   UNKNOWN) : From 00001234h to 00001252h, in r4
  9(   UNKNOWN) : From 00001252h to 00001256h, in r11[36] (1 slot)
- 11(   UNKNOWN) : From 00000A4Eh to 00000B84h, in r8
+ 11(   UNKNOWN) : From 00000A4Eh to 00001160h, in r8
- 11(   UNKNOWN) : From 00000B86h to 00001160h, in r8
 11(   UNKNOWN) : From 00001176h to 000011F8h, in r8
 11(   UNKNOWN) : From 00001256h to 00001274h, in r8
IP mapping count : 8
IL offs PROLOG : 0x00000000 ( STACK_EMPTY )
IL offs 0x0000 : 0x0000000A ( STACK_EMPTY )
IL offs 0x0003 : 0x0000000E ( STACK_EMPTY )
IL offs 0x001C : 0x0000002C ( STACK_EMPTY )
IL offs 0x002A : 0x00000036 ( STACK_EMPTY )
IL offs EPILOG : 0x0000003A ( STACK_EMPTY )
IL offs 0x0012 : 0x00000042 ( STACK_EMPTY )
IL offs NO_MAP : 0x00000066 ( STACK_EMPTY )

[CarolEidt]

It hadn't occurred to me that the crossgen-comparison diff would catch the live range coalescing. That's kind of cool!

[echesakov]

@BruceForstall These are two disassembles from the native compiler

G_M47947_IG10:
00019A  F2C0 0C00      movt    r12, HIGH RELOC 0xB3F15AB0
00019E  44FC           add     r12, pc
0001A0  F8DC C000      ldr     r12, [r12]
0001A4  47E0           blx     r12        // CORINFO_HELP_ASSIGN_REF
0001A6  F108 0801      add     r8, r8, 1
0001AA  4547           cmp     r7, r8
0001AC  F73F AF68      bgt     G_M47947_IG09

and the cross-architectural compiler

00019E  44FC           add     r12, pc
0001A0  F8DC C000      ldr     r12, [r12]
0001A4  47E0           blx     r12        // CORINFO_HELP_ASSIGN_REF
G_M47947_IG10:
0001A6  F108 0801      add     r8, r8, 1
0001AA  4547           cmp     r7, r8
0001AC  F73F AF68      bgt     G_M47947_IG09