full_coverage_test is flaky on Windows and Linux

[whesse]

standalone/full_coverage_test is failing flakily on win32 release since 868d2c6 Mon Jul 13 17:49:49 2015 -0700
It also fails on win64 release.

Marking it as flaky in tests/standalone/standalone.status.

This test is also using package:unittest, which the core tests in tests/core, tests/standalone/, tests/language, tests/compiler, and tests/lib (except async?) should not be dependent on.
It should be rewritten not to use it - not many changes are needed.

FAILED: none-vm-checked release_x64 standalone/full_coverage_test
Expected: Pass
Actual: RuntimeError
CommandOutput[vm]:

stdout:
unittest-suite-wait-for-done
FAIL: CoverageTests
Expected: <0>
Actual: <3>

package:unittest/src/simple_configuration.dart 128:34 SimpleConfiguration.onExpectFailure
package:unittest/src/simple_configuration.dart 24:13 _ExpectFailureHandler.fail
package:unittest/src/matcher/expect.dart 121:5 DefaultFailureHandler.failMatch
package:unittest/src/matcher/expect.dart 95:20 expect
tests\standalone\full_coverage_test.dart 96:5 generateCoverage
tests\standalone\full_coverage_test.dart 219:5 main.
package:unittest/src/internal_test_case.dart 120:37 InternalTestCase.run.
....

@fsc8000 @mlippautz @kodandersson

[kodandersson]

I'll attempt to get a minidump to debug this (cannot repro on my own Windows VM).

[kodandersson]

Recent crash in Linux:
http://build.chromium.org/p/client.dart/builders/vm-linux-release-optcounter-threshold-be/builds/6427/steps/tests/logs/stdio

NOTE: This is reported as a runtime error, but the child VM process is actually crashing (exit status -6 on Linux, 3 on Windows).

Seems we still have a non-Windows-specific issue with shutdown vs. concurrent sweep.

Here are the stack traces:
(gdb) t 1
[Switching to thread 1 (Thread 0x7f3f17912740 (LWP 29993))]
#0 0x00007f3f1654e0d5 in __GI_raise (sig=) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64 in ../nptl/sysdeps/unix/sysv/linux/raise.c
(gdb) bt
#0 0x00007f3f1654e0d5 in __GI_raise (sig=) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x00007f3f1655183b in __GI_abort () at abort.c:91
#2 0x0000000000586fef in dart::DynamicAssertionHelper::Fail (this=, format=) at runtime/platform/assert.cc:41
#3 0x000000000069b354 in thread_registry (this=, this=, isolate=, this=)
at runtime/vm/thread_registry.h:71
#4 dart::Isolate::Shutdown (this=0x2b08c00) at runtime/vm/isolate.cc:1503
#5 0x00000000005bb161 in dart::Dart::ShutdownIsolate () at runtime/vm/dart.cc:352
#6 0x0000000000552e8a in Dart_ShutdownIsolate () at runtime/vm/dart_api_impl.cc:1400
#7 0x00000000005498b1 in dart::bin::main (argc=5, argv=0x7fffbf459ab8) at runtime/bin/main.cc:1095
#8 0x000000000054a289 in main (warning: Unmapped DWARF Register #-2 encountered.
argc=<error reading variable: Unable to access DWARF register number 0xfffffffe>, argv=0x7529)
at runtime/bin/main.cc:1124
(gdb) t 2
[Switching to thread 2 (Thread 0x7f3f14d49700 (LWP 29997))]
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
215 ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S: No such file or directory.
(gdb) bt
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1 0x00000000007111bb in dart::Monitor::WaitMicros (this=0x7f3f1001ee30, micros=) at runtime/vm/os_thread_linux.cc:334
#2 0x00000000007c07dc in dart::ThreadPool::Worker::Loop() ()
#3 0x00000000007c06b7 in dart::ThreadPool::Worker::Main(unsigned long) ()
#4 0x000000000071089e in dart::ThreadStart (data_ptr=) at runtime/vm/os_thread_linux.cc:94
#5 0x00007f3f174fbe9a in start_thread (arg=0x7f3f14d49700) at pthread_create.c:308
#6 0x00007f3f1660b38d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7 0x0000000000000000 in ?? ()
(gdb) t 3
[Switching to thread 3 (Thread 0x7f3f17910700 (LWP 29994))]
#0 0x00007f3f1660ba23 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
82 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0 0x00007f3f1660ba23 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1 0x0000000000589d25 in dart::bin::EventHandlerImplementation::Poll (args=44530592) at runtime/bin/eventhandler_linux.cc:372
#2 0x0000000000585dde in dart::bin::ThreadStart (data_ptr=) at runtime/bin/thread_linux.cc:87
#3 0x00007f3f174fbe9a in start_thread (arg=0x7f3f17910700) at pthread_create.c:308
#4 0x00007f3f1660b38d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()
(gdb) t 4
[Switching to thread 4 (Thread 0x7f3f1514a700 (LWP 29996))]
#0 decode (value=, value=) at runtime/vm/bitfield.h:54
54 return static_cast((value >> position) & ((kUwordOne << size) - 1));
(gdb) bt
#0 decode (value=, value=) at runtime/vm/bitfield.h:54
#1 IsMarked (this=) at runtime/vm/raw_object.h:334
#2 dart::GCSweeper::SweepPage (this=, page=, freelist=0x2b0fa10, locked=false) at runtime/vm/gc_sweeper.cc:28
#3 0x00000000008009dc in dart::SweeperTask::Run (this=0x2e3b9f0) at runtime/vm/gc_sweeper.cc:125
#4 0x00000000007c0746 in dart::ThreadPool::Worker::Loop() ()
#5 0x00000000007c06b7 in dart::ThreadPool::Worker::Main(unsigned long) ()
#6 0x000000000071089e in dart::ThreadStart (data_ptr=) at runtime/vm/os_thread_linux.cc:94
#7 0x00007f3f174fbe9a in start_thread (arg=0x7f3f1514a700) at pthread_create.c:308
#8 0x00007f3f1660b38d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#9 0x0000000000000000 in ?? ()
(gdb) t 5
[Switching to thread 5 (Thread 0x7f3f153b1700 (LWP 29995))]
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
215 ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S: No such file or directory.
(gdb) bt
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:215
#1 0x00000000007111bb in dart::Monitor::WaitMicros (this=0x2a78090, micros=) at runtime/vm/os_thread_linux.cc:334
#2 0x00000000007bfabf in WaitMicros (micros=, this=, micros=) at runtime/vm/lockers.h:54
#3 dart::ThreadInterrupter::ThreadMain (parameters=) at runtime/vm/thread_interrupter.cc:293
#4 0x000000000071089e in dart::ThreadStart (data_ptr=) at runtime/vm/os_thread_linux.cc:94
#5 0x00007f3f174fbe9a in start_thread (arg=0x7f3f153b1700) at pthread_create.c:308
#6 0x00007f3f1660b38d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#7 0x0000000000000000 in ?? ()

[kodandersson]

Hypothesis: the sweeper is re-awakened by some allocation late in the shutdown process. Will track it down and move the waiting and/or add NoSafepointScopes. (Longer-term, we need to revamp the "tasks" counter method to synchronize these things.)

[kodandersson]

... and here's the allocation (why are we compiling classes during shutdown?):

(gdb) bt
#0 0xf7723440 in __kernel_vsyscall ()
#1 0xf7400607 in raise () from /lib/i386-linux-gnu/libc.so.6
#2 0xf7403a33 in abort () from /lib/i386-linux-gnu/libc.so.6
#3 0x081e7ac5 in dart::DynamicAssertionHelper::Fail (this=this@entry=0xffdf9ee4, format=format@entry=0x882cdc8 "expected: %s")
at runtime/platform/assert.cc:41
#4 0x08304650 in dart::Heap::AllocateNew (this=this@entry=0xa87b438, size=size@entry=32) at runtime/vm/heap.cc:81
#5 0x083ef01f in dart::Heap::Allocate (this=0xa87b438, size=32, space=dart::Heap::kNew) at runtime/vm/heap.h:82
#6 0x083acd7f in dart::Object::Allocate (cls_id=cls_id@entry=62, size=size@entry=32, space=space@entry=dart::Heap::kNew)
at runtime/vm/object.cc:1729
#7 0x083bf41c in dart::Array::New (class_id=class_id@entry=62, len=len@entry=4, space=space@entry=dart::Heap::kNew)
at runtime/vm/object.cc:19632
#8 0x083bf634 in dart::Array::New (len=len@entry=4, space=space@entry=dart::Heap::kNew) at runtime/vm/object.cc:19619
#9 0x083c89ab in dart::GrowableObjectArray::New (capacity=capacity@entry=4, space=space@entry=dart::Heap::kNew)
at runtime/vm/object.cc:19891
#10 0x0821b56f in dart::Compiler::CompileClass (cls=...) at runtime/vm/compiler.cc:298
#11 0x0838ca0a in dart::Class::EnsureIsFinalized (this=this@entry=0xffdfaeac, isolate=isolate@entry=0xa8abce0) at runtime/vm/object.cc:2938
#12 0x0855a86d in dart::CodeCoverage::PrintClass (lib=..., cls=..., jsarr=..., filter=filter@entry=0xffdfa914,
as_call_sites=as_call_sites@entry=false) at runtime/vm/coverage.cc:145
#13 0x0855d98a in dart::CodeCoverage::PrintJSON (isolate=isolate@entry=0xa8abce0, stream=stream@entry=0xffdfa99c, filter=0xffdfa914,
filter@entry=0x0, as_call_sites=as_call_sites@entry=false) at runtime/vm/coverage.cc:290
#14 0x0855e5cb in dart::CodeCoverage::Write (isolate=isolate@entry=0xa8abce0) at runtime/vm/coverage.cc:249
#15 0x08375b82 in dart::Isolate::Shutdown (this=this@entry=0xa8abce0) at runtime/vm/isolate.cc:1478
#16 0x0822d05b in dart::Dart::ShutdownIsolate () at runtime/vm/dart.cc:352
#17 0x081a2943 in dart::Dart_ShutdownIsolate () at runtime/vm/dart_api_impl.cc:1400
#18 0x081877da in dart::bin::main (argc=argc@entry=5, argv=argv@entry=0xffdfb8e4) at runtime/bin/main.cc:1095
#19 0x08187d3b in main (argc=5, argv=0xffdfb8e4) at runtime/bin/main.cc:1124

[fsc8000]

maybe s/EnsureIsFinalized/is_finalized/?

[kodandersson]

I don't think that's sufficient, it seems to rely on actually compiling things. I'll move it to an earlier point in the shutdown process; CL is in the works (although the issue is low-ish prio since it only happens with the flag).

[kodandersson]

Fixed ee49289