Dart VM crashes on Windows

[efortuna]

The Dart VM crashes when compiling browser tests. Unfortunately, the crash is not related to a particular test, although it regularly seems to crash on a particular test when all the tests are run in order0. If you run that test in isolation, it passes. If you run the particular test suite it may also pass (or a different test fails). But if you run the entire set of tests like the buildbot, the test will fail. The current one failing is NumberSyntaxTest on the web-ff-win7 bot, but this behavior started happening March 9th in the afternoon.

Some examples of the errors:
Kind: 21886600
http://build.chromium.org/p/client.dart/builders/web-chrome-win7/builds/797/steps/chrome%20tests%3A%20webdriver%20--checked%20--browser%3Dchrome/logs/stdio

and
VM exited with signal 1073741819
http://build.chromium.org/p/client.dart/builders/web-ff-win7/builds/803/steps/ff%20tests%3A%20webdriver%20--checked%20--browser%3Dff/logs/stdio

The second error is reported by frog, which passes on the error signal (-1073741819) sent from the VM, but I believe these different errors are indicative of the same underlying problem. Some web searches suggest this error might be an EXCEPTION_ACCESS_VIOLATION. I looked in the Windows event logs, and got the following message:

Faulting application name: dart.exe
Exception code: 0xc0000005

The browser buildbots will continue to be unstable until we can fix this.
Thanks.

[efortuna]

The first appearance of this error in the Windows log is 3/7/2012 at 11:45:58 PM

[iposva-google]

cc @a-siva.

[a-siva]

Could we run a debug version of the VM instead of the release version. That might trip over some assert and give us better diagnostics.

[efortuna]

Tried running the debug version. The bug didn't repro. We ca see if it's possible to attach a debugger just before the particular test runs, though.

[efortuna]

I can attach Visual Studio 2008 to the Dart VM just before the crash. Will this be enough to help you guys start investigating the error?
I get the message: "Unhandled exception at 0x013419a8 in dart.exe: 0xC0000005: Access violation reading location 0x00000478.

Debugging in VS can probably provide additional information surrounding the condition causing the crash.

[a-siva]

Sure a stack trace of all the threads when you are attached with visual studio would be helpful.

[iposva-google]

Set owner to @a-siva.
Added Accepted label.

[efortuna]

On crash in release mode, but with some debug symbols, here's the call stack I get:

dart.exe!dart::Object::Handler() + 0x36 bytes <-- yellow arrow here
dart.exe!dart::Array::MakeArray() + 0x17f bytes
dart.exe!dart::Function::BuildSignature() + 0x903 bytes
dart.exe!dart::Parser::ParseFunctionStatement() + 0x396 bytes
dart.exe!dart::Parser::ParsePrimary() + 0x52 bytes
dart.exe!dart::Parser::ParsePostfixExptr() + 0x15 bytes
dart.exe!dart::ICData::ZoneHandler() + 0x15 bytes
dart.exe!dart::GrowableObjectArray::New() + 0xa bytes
dart.exe!dart::AbstractTypeArguments::IsWitinBoundsOf() + 0x7c5 bytes
dart.exe!dart::Handles<2,64,6>::SetupNextScopeBlock() + 0x1c bytes
dart.exe!dart::Class::NumTypeArguments() + 0xe9 bytes
dart.exe!dart::DN_String_hashCode() + 0xbf bytes
dart.exe!dart::DartEntry::InvokeStatic() + 0x102 bytes
dart.exe!_Dart_InvokeStatic + 0x3c6 bytes
dart.exe!_main() + 0x325 bytes
dart.exe!dart::__tmainCRTStartup() Line 266 + 0x12 bytes
kernel32.dll!7697339a()

VS pulled up crt0.c which doesn't look like your code (most likely the first bit of code in the stack that has full debug symbols), but for completeness I've replicated the info here:
#ifdef WINMAIN
  (some stuff)
#else
    _tinitenv = _tenviron;
    mainret = _tmain(__argc, __targv, _tenviron); <-- arrow here, corresponds with the line dart.exe!dart::__tmainCRTStartup() Line 266 + 0x12 bytes in the stack trace above

in the watch window mainret = 0 and __argc = 12, and &__ImageBase (some _IMAGE_DOS_HEADER * type) = 0x011d0000

Going up to the code for Object::Handle() where the problem occurred, we have following extremely legible assembly code:
011DEF72 ret
011DEF73 mov edx,dword ptr [ecx-1]
011DEF76 mov ecx,dword ptr [edx+43h] <-- arrow pointing at this instruction
011DEF79 mov dword ptr [eax],ecx
011DEF7B red
and so on

[efortuna]

(the last line of assembly should read "ret" instead of "red")

[a-siva]

Submitted https://chromiumcodereview.appspot.com//9700079

[efortuna]

Hi Siva,
Unfortunately it looks like there may be more to this bug than we originally thought. Tests are still failing in the compile step on Windows on some machines. I've enabled debugging again so we can get a stack trace again when the bug occurs next.

[efortuna]

Latest crash on web-ie-win7:
dart.exe!dart::Instance::CheckedHandle() + 0x58 bytes
dart.exe!dart::DRT_BreakpointStaticHandler() + 0x3d3 bytes
dart.exe!dart::DRT_InlineCacheMissHandlerOneArg() + 0xab bytes
dart.exe!dart::DartEntry::InvokeStatic() + 0x102 bytes
dart.exe!_Dart_InvokeStatic() + 0x3c6 bytes
dart.exe!_main() + 0x325 bytes
dart.exe!__tmainCRTStartup() Line 266 + 0x12 bytes

the assembly:
001A1A7B jmp dart::Instance::CheckedHandle+56h (1A1A86h)
001A1A7D mov dword ptr [esi+4],ecx
001A1A80 mov dword ptr [esi],offset dart::Instance::`vftable' (379F00h)
001A1A86 mov eax, dword ptr [esi]
001A1A88 mov edx,dword ptr [eax+7Ch]
001A1A8B mov ecx,esi
001A1A8F call edx

[efortuna]

the web-chrome-win7 is also failing with an identical stack trace even though it's failing on a different test FutureTest on the chrome bot, and ListInsertRangeTest on the IE bot.

[efortuna]

I have now been able to reproduce this bug on a local Windows machine as well (looks like a similar stack trace).

Here's the stack trace:
ntdll.dll!77cc15de()
ntdll.dll!7cc15de()
ntdll.dll!77db014e()
dart.exe!dart::DRT_InlineCache­MissHandlerOneArg() + 0xab bytes
dart.exe!dart::DartEntry::Invo­keStatic() + 0x102 bytes
dart.exe!_Dart_InvokeStatic() + 0x3c6 bytes
dart.exe!_main() + 0x325 bytes
dart.exe!__tmainCRTStartup() Line 266 + 0x12 bytes
ntdll.dll!77cc15de()
ntdll.dll!7cc15de()
ntdll.dll!77db014e()
dart.exe!dart::DRT_InlineCacheMissHandlerOneArg() + 0xab bytes
dart.exe!dart::DartEntry::InvokeStatic() + 0x102 bytes
dart.exe!_Dart_InvokeStatic() + 0x3c6 bytes
dart.exe!_main() + 0x325 bytes
dart.exe!__tmainCRTStartup() Line 266 + 0x12 bytes

[efortuna]

Hello all,
Quick update, this issue just occurred on the VM Windows release bot as well: http://chromegw.corp.google.com/i/client.dart/builders/vm-win32-debug/builds/1094/steps/tests/logs/stdio

[sigmundch]

This error seems related, but I'm not sure:

http://chromegw.corp.google.com/i/client.dart/builders/web-chrome-win7/builds/1538/steps/chrome%20frog%20tests%3A%20chrome%20--checked/logs/stdio

FAILED: frog-chrome release_ia32 dom/DOMConstructorsTest
Expected: pass
Actual: crash

stdout:
Kind: 6029329

stderr:
test.dart: Compilation failed (step 1 of 2), exit code 3

Compilation command: Release_ia32/frog/bin/frog.bat --ignore-unrecognized-flags --enable_asserts --enable_type_checks --libdir=e:/b/build/slave/web-chrome-win7/build/dart/frog/lib --compile-only --out=e:/b/build/slave/web-chrome-win7/build/dart/Release_ia32/generated_tests/frog-chrome/client_tests_client_dom_DOMConstructorsTest-/test.js e:/b/build/slave/web-chrome-win7/build/dart/client/tests/client/dom/DOMConstructorsTest.dart

Command line: python e:/b/build/slave/web-chrome-win7/build/dart/tools/testing/run_selenium.py --browser=chrome --timeout=238 --out=e:/b/build/slave/web-chrome-win7/build/dart/Release_ia32/generated_tests/frog-chrome/client_tests_client_dom_DOMConstructorsTest-/../e__b_build_slave_web-chrome-win7_build_dart_client_tests_client_dom_DOMConstructorsTest.dartfrog-chrome.html

[sigmundch]

Here is a similar one with a different kind (I think the most common one is the one earlier in this bug thread, but Emily suppressed those from the bots so we could continue tracking other tests):

FAILED: frog-ff release_ia32 dom/TypedArrays3Test
Expected: pass
Actual: crash

stdout:
Kind: 1572881

stderr:
test.dart: Compilation failed (step 1 of 2), exit code 3

Compilation command: Release_ia32/frog/bin/frog.bat --ignore-unrecognized-flags --enable_asserts --enable_type_checks --libdir=e:/b/build/slave/web-ff-win7/build/dart/frog/lib --compile-only --out=e:/b/build/slave/web-ff-win7/build/dart/Release_ia32/generated_tests/frog-ff/client_tests_client_dom_TypedArrays3Test-/test.js e:/b/build/slave/web-ff-win7/build/dart/client/tests/client/dom/TypedArrays3Test.dart

Command line: python e:/b/build/slave/web-ff-win7/build/dart/tools/testing/run_selenium.py --browser=ff --timeout=238 --out=e:/b/build/slave/web-ff-win7/build/dart/Release_ia32/generated_tests/frog-ff/client_tests_client_dom_TypedArrays3Test-/../e__b_build_slave_web-ff-win7_build_dart_client_tests_client_dom_TypedArrays3Test.dartfrog-ff.html

http://chromegw.corp.google.com/i/client.dart/builders/web-ff-win7/builds/1513/steps/ff%20frog%20tests%3A%20ff%20--checked/logs/stdio

[efortuna]

Removed Priority-High label.
Added Priority-Critical label.

[lexprfuncall]

We have been able to reduce this down to the StringInterpolate2Test. Running the attached "crash.bat" will run this lone test case. It fails every time on our windows machine.

Interestingly, one can comment out all lines in the main function except the first two. Even more interestingly, changing the last assignment of false to true makes the bug go away!

---

Attachment:
crash.bat (431 Bytes)