Repository navigation
investigate flaky test_make_callback/test-async-hooks-gcable in CI #30648
Description
Activity
again:
11:26:27 not ok 2592 node-api/test_make_callback/test-async-hooks-gcable 11:26:27 --- 11:26:27 duration_ms: 0.264 11:26:27 severity: crashed 11:26:27 exitcode: -11 11:26:27 stack: |- 11:26:27 ...Yeah, this one is out of control today. Always in the containered builds.
Reacted by Gireesh Punathilmore instances: https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_shared_x64/16224/consoleFull
I guess most of the recent CI runs have this affected. Pinning to catch attention.
- pinned this issue
on Nov 26, 2019 cc @nodejs/v8 @nodejs/async_hooks @nodejs/n-api
@gireeshpunathil @Trott This is likely impossible to debug without core dumps and without access to the machine in question? I’ll open an access request on nodejs/build.
Reacted by Gireesh Punathil(gdb) bt #0 0x000056076b17e854 in v8::internal::ConcurrentMarkingVisitor::VisitPointers(v8::internal::HeapObject, v8::internal::FullObjectSlot, v8::internal::FullObjectSlot) () #1 0x000056076b18894c in v8::internal::ConcurrentMarking::Run(int, v8::internal::ConcurrentMarking::TaskState*) () #2 0x000056076b0f3b86 in non-virtual thunk to v8::internal::CancelableTask::Run() () #3 0x000056076aec9e15 in node::(anonymous namespace)::PlatformWorkerThread(void*) () #4 0x00007f8c559dd6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007f8c5570688f in clone () from /lib/x86_64-linux-gnu/libc.so.6This might be related/the same bug as #30498?
Okay, results from debugging so far:
- This is weirdly related to ESM in some way: It always crashes when a major GC happens during our
Module::CreateSyntheticModule()calls during bootstrap. I think 796f3d0 might actually be the commit that started these issues. - This makes it reproduce somewhat frequently (~ 50 % of the time) locally for me:
diff --git a/deps/v8/src/objects/objects.cc b/deps/v8/src/objects/objects.cc index 227cff8da47a..c1d344c304a0 100644 --- a/deps/v8/src/objects/objects.cc +++ b/deps/v8/src/objects/objects.cc @@ -6458,6 +6458,10 @@ Handle<Derived> HashTable<Derived, Shape>::NewInternal( Factory* factory = isolate->factory(); int length = EntryToIndex(capacity); RootIndex map_root_index = Shape::GetMapRootIndex(); + if (Shape::kEntrySize == 2) { + isolate->heap()->CollectAllGarbage( + Heap::kNoGCFlags, GarbageCollectionReason::kFullHashtable); + } Handle<FixedArray> array = factory->NewFixedArrayWithMap(map_root_index, length, allocation); Handle<Derived> table = Handle<Derived>::cast(array);
That might be a good starting point for debugging in V8 here, and/or bisecting V8. (But that’s for tomorrow rather than today :))
@nodejs/v8
This might be related/the same bug as #30498?
I’m very confident that this is indeed the case now.
Reacted by Jiawen GengReacted by Rich Trott- This is weirdly related to ESM in some way: It always crashes when a major GC happens during our
Here’s a V8 CL that resolves this issue both locally and on the container host: https://chromium-review.googlesource.com/c/v8/v8/+/1939752
Reacted by Michaël Zasso, Gireesh Punathil, Richard Lau, Rich Trott, Denys Otrishko and Jiawen Geng- added a commit that references this issue
on Nov 28, 2019 - unpinned this issue
on Nov 29, 2019 - added a commit that references this issue
on Nov 29, 2019 - added a commit that references this issue
on Nov 30, 2019 - added a commit that references this issue
on Jan 12, 2020 - added a commit that references this issue
on Feb 6, 2020
ref: https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_zlib_x64/16212/consoleFull