Cannot print exception string because Exception.ToString() failed

[adityamandaleeka]

This occurred on an Ubuntu CI job while running Base02d_seq_ser. The job appears to have been kicked off for this PR but I don't have any reason to believe it's related to the changes in that PR.

Dump and binaries: http://dotnetrp.azurewebsites.net/dumpling/download/1504

[adityamandaleeka]

Another example of this: http://dotnet-ci.cloudapp.net/job/dotnet_coreclr/job/master/job/checked_ubuntu_tst_prtest/3179

This time in Loader/classloader/generics/Instantiation/Negative/abstract08

Dump and binaries: http://dotnetrp.azurewebsites.net/dumpling/download/1507

[janvorli]

As for the first dump (I haven't looked at the other yet), I can see that the formatting of the stack trace has failed due to the null reference exception:

(lldb) pe
Exception object: 00007f74a0017728
Exception type:   System.NullReferenceException
Message:          Object reference not set to an instance of an object.
InnerException:   <none>
StackTrace (generated):
    SP               IP               Function
    00007FFFF222B6D0 00007F78F00434D5 System.Private.CoreLib.dll!System.Text.StringBuilder.AppendFormatHelper(System.IFormatProvider, System.String, System.ParamsArray)+0x905
    00007FFFF222B760 00007F78F02CBF46 System.Private.CoreLib.dll!System.Diagnostics.StackTrace.ToString(TraceFormat)+0x266
    00007FFFF222B890 00007F78F0055A88 System.Private.CoreLib.dll!System.Exception.GetStackTrace(Boolean)+0x48
    00007FFFF222B8B0 00007F78F0055919 System.Private.CoreLib.dll!System.Exception.ToString(Boolean, Boolean)+0x189
    00007FFFF222B8F0 00007F78F0054DD8 System.Private.CoreLib.dll!System.Exception.InternalToString()+0x68

StackTraceString: <none>
HResult: 80004003

Disassembling code around the failure point shows something strange:

00007f78f00434c6 488b7db0             mov     rdi, qword ptr [rbp - 0x50]
00007f78f00434ca 4885ff               test    rdi, rdi
00007f78f00434cd 0f84aa000000         je      0x7f78f004357d
00007f78f00434d3 33ff                 xor     edi, edi
>>> 00007f78f00434d5 488b07               mov     rax, qword ptr [rdi]

See the xor edi, edi? That means that the RDI would always be zero there! The function is quite long, but trying to match the source and the diassembly, it looks like the test rdi, rdi is the else if (arg != null) below:

                if (s == null) {
                    IFormattable formattableArg = arg as IFormattable;

                    if (formattableArg != null) {
                        if (sFmt == null && fmt != null) {
                            sFmt = fmt.ToString();
                        }

                        s = formattableArg.ToString(sFmt, provider);
                    } else if (arg != null) {
                        s = arg.ToString();
                    }
                }

[adityamandaleeka]

The second dump has exactly the same issue. I wonder if this is related to any recent JIT changes.

[adityamandaleeka]

I was curious about this so I dumped the disassembly for System.Private.CoreLib.dll!System.Text.StringBuilder.AppendFormatHelper from an older dump (from some unrelated run last week) and noticed that it's different (excerpts below):

OLDER

    0x7ff23025fe50: movabsq $0x7ff230145118, %r11     ; imm = 0x7FF230145118 
    0x7ff23025fe5a: cmpl   %edi, (%rdi)
    0x7ff23025fe5c: callq  *(%r11)
    0x7ff23025fe5f: movq   %rax, -0x68(%rbp)
    0x7ff23025fe63: movl   -0x34(%rbp), %r8d
    0x7ff23025fe67: movl   -0x38(%rbp), %r9d
    0x7ff23025fe6b: jmp    0x7ff23025fe9b
    0x7ff23025fe6d: movq   %r10, -0x58(%rbp)
    0x7ff23025fe71: jmp    0x7ff23025fdaf
    0x7ff23025fe76: movq   -0x50(%rbp), %rcx
    0x7ff23025fe7a: testq  %rcx, %rcx
    0x7ff23025fe7d: je     0x7ff23025ff2e
    0x7ff23025fe83: movq   %rcx, %rdi
    0x7ff23025fe86: movq   (%rcx), %rax
    0x7ff23025fe89: movq   0x48(%rax), %rax
    0x7ff23025fe8d: callq  *(%rax)

NEWER

    0x7ff635b9b4b0: movabsq $0x7ff635a8f0c0, %r11     ; imm = 0x7FF635A8F0C0 
    0x7ff635b9b4ba: cmpl   %edi, (%rdi)
    0x7ff635b9b4bc: callq  *(%r11)
    0x7ff635b9b4bf: movq   %rax, -0x68(%rbp)
    0x7ff635b9b4c3: movl   -0x34(%rbp), %r8d
    0x7ff635b9b4c7: movl   -0x38(%rbp), %r9d
    0x7ff635b9b4cb: jmp    0x7ff635b9b4fa
    0x7ff635b9b4cd: movq   %r10, -0x58(%rbp)
    0x7ff635b9b4d1: jmp    0x7ff635b9b40f
    0x7ff635b9b4d6: movq   -0x50(%rbp), %rdi
    0x7ff635b9b4da: testq  %rdi, %rdi
    0x7ff635b9b4dd: je     0x7ff635b9b58d
    0x7ff635b9b4e3: xorl   %edi, %edi
    0x7ff635b9b4e5: movq   (%rdi), %rax
    0x7ff635b9b4e8: movq   0x48(%rax), %rax
    0x7ff635b9b4ec: callq  *(%rax)

[parjong]

Is it related with #6283? The error log looks very similar.

[CarolEidt]

I'm looking at this, while in parallel attempting to run the failing arm tests from #6283 on the emulator. Using the corerun from the 2nd dump location above, with a version of the test built locally, I would expect the code generated to be the same, since it is exactly the same jit.

[CarolEidt]

I am unable to repro this with either test case. What's even stranger than the fact that the "this" pointer is being set to zero is that just above, it has been tested against zero. It's almost as if assertion prop has gotten it exactly backward (i.e. propagating null to this block instead of non-null), or that value numbering is incorrectly propagating the results of the IsInstanceOf call. If my dotnet/coreclr#6227 PR is indeed responsible for the arm failures in #6283 (which I'm still working on attempting to repro), it would be very surprising if it is the same issue as this, since this deals with object refs and my PR should affect only structs (unless, of course, there's some interaction that I haven't been able to see). @dotnet/jit-contrib - can anyone else think of something in the JIT that might determine that a ref's value is zero?

[swgillespie]

Here's another CI failure that looks like this, but on ARM: http://dotnet-ci.cloudapp.net/job/dotnet_coreclr/job/master/job/arm_emulator_cross_release_ubuntu_prtest/522/console

[adityamandaleeka]

This just repro'd again on the Ubuntu job for PR dotnet/coreclr#6297

[adityamandaleeka]

Here are a few more dumps for occurrences of this issue in CoreCLR CI runs:

http://dotnetrp.azurewebsites.net/dumpling/download/1579
http://dotnetrp.azurewebsites.net/dumpling/download/1580
http://dotnetrp.azurewebsites.net/dumpling/download/1581
http://dotnetrp.azurewebsites.net/dumpling/download/1582
http://dotnetrp.azurewebsites.net/dumpling/download/1598
http://dotnetrp.azurewebsites.net/dumpling/download/1601
http://dotnetrp.azurewebsites.net/dumpling/download/1604
http://dotnetrp.azurewebsites.net/dumpling/download/1606

[adityamandaleeka]

It just happened again: http://dotnet-ci.cloudapp.net/job/dotnet_coreclr/job/master/job/checked_ubuntu_tst_prtest/3577/consoleFull#2096352037c7af6a29-b465-404a-b249-90a3911e4354

http://dotnetrp.azurewebsites.net/dumpling/download/1610

[adityamandaleeka]

And another: http://dotnet-ci.cloudapp.net/job/dotnet_coreclr/job/master/job/jitstress/job/x64_checked_ubuntu_jitstressregs0x10_tst/33/

http://dotnetrp.azurewebsites.net/dumpling/download/1611

[RussKeldorph]

So presumably confirmed that this is indeed not due to dotnet/coreclr#6021 since that was backed out. The symptom became basically 100% repro on ARM after dotnet/coreclr#6021, though, so perhaps it would be better to investigate the problem there in case both the ARM failure and the x64 failure happen to be the same issue.

[CarolEidt]

@papaslavik has identified the root cause of at least one of the arm failures (#6459) that seems to be similar to this. We have both been looking into potential solutions. Of course, there's no guarantee that it is the same root cause as this, but since it is an issue of using uninitialized data in assertion prop (and the bad code generated here has the appearance of being bad assertion prop), it could very well be.

[CarolEidt]

@adityamandaleeka - I just merged a PR (#6484) that fixes a problem with assertion prop using uninitialized data. I am hoping these intermittent failures are addressed by that as well. Please let me know if you encounter more of these failures.