Assertion `(source_it) != (source_.end())' failed while doing make

[gireeshpunathil]

Version

master

Platform

Linux 5.4.0-77-generic #86-Ubuntu SMP Thu Jun 17 02:35:03 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Subsystem

build

What steps will reproduce the bug?

$./configure
$make -j4

ACTION _home_gireesh_node_node_gyp_node_target_node_mksnapshot /home/gireesh/node/out/Release/obj/gen/node_snapshot.cc
/home/gireesh/node/out/Release/node_mksnapshot[660863]: ../src/node_native_module.cc:226:v8::MaybeLocal<v8::String> node::native_module::NativeModuleLoader::LoadBuiltinModuleSource(v8::Isolate*, const char*): Assertion `(source_it) != (source_.end())' failed.
 1: 0x55d5331d5ee4 node::Abort() [/home/gireesh/node/out/Release/node_mksnapshot]
 2: 0x55d5331d5f78  [/home/gireesh/node/out/Release/node_mksnapshot]
 3: 0x55d5333f8fdd node::native_module::NativeModuleLoader::LoadBuiltinModuleSource(v8::Isolate*, char const*) [/home/gireesh/node/out/Release/node_mksnapshot]
 4: 0x55d5333f95d1 node::native_module::NativeModuleLoader::LookupAndCompile(v8::Local<v8::Context>, char const*, std::vector<v8::Local<v8::String>, std::allocator<v8::Local<v8::String> > >*, node::native_module::NativeModuleLoader::Result*) [/home/gireesh/node/out/Release/node_mksnapshot]
 5: 0x55d533234649 node::native_module::NativeModuleEnv::LookupAndCompile(v8::Local<v8::Context>, char const*, std::vector<v8::Local<v8::String>, std::allocator<v8::Local<v8::String> > >*, node::Environment*) [/home/gireesh/node/out/Release/node_mksnapshot]
 6: 0x55d53336f36e node::InitializePrimordials(v8::Local<v8::Context>) [/home/gireesh/node/out/Release/node_mksnapshot]
 7: 0x55d53336f560 node::GetPerContextExports(v8::Local<v8::Context>) [/home/gireesh/node/out/Release/node_mksnapshot]
 8: 0x55d53336f2a0 node::InitializePrimordials(v8::Local<v8::Context>) [/home/gireesh/node/out/Release/node_mksnapshot]
 9: 0x55d53336f695 node::NewContext(v8::Isolate*, v8::Local<v8::ObjectTemplate>) [/home/gireesh/node/out/Release/node_mksnapshot]
10: 0x55d53327a4a9 node::SnapshotBuilder::Generate(node::SnapshotData*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >) [/home/gireesh/node/out/Release/node_mksnapshot]
11: 0x55d53327b014 node::SnapshotBuilder::Generate(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >) [/home/gireesh/node/out/Release/node_mksnapshot]
12: 0x55d532e05d7d main [/home/gireesh/node/out/Release/node_mksnapshot]
13: 0x7f89037610b3 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
14: 0x55d532e0e46e _start [/home/gireesh/node/out/Release/node_mksnapshot]
Aborted (core dumped)
make: *** [node.target.mk:26: /home/gireesh/node/out/Release/obj/gen/node_snapshot.cc] Error 134
rm 2561388a0ca6c2168de218f405f8f7b697c77862.intermediate 9d285d3140c975d071f47f872a2170cedfc46bc5.intermediate aaef1ed75da9c226afcc645ea47e3916fde6ec36.intermediate

How often does it reproduce? Is there a required condition?

everytime

What is the expected behavior?

node builds successfully

What do you see instead?

build crashes

Additional information

first time I am building in this system, so suspect something specific to build toolchain versions, resource limit etc. but cannot figure out what.

[legendecas]

This is typically for built in module not been found. Can you re-configure to generate refresh new build scripts and try again?

Edit: just found that one of the re-production steps is ./configure. There is a change on master branch recently that all builtin module files were globbed to be bundled into node binary. You may check on the variable node_library_files in config.gypi to see if it is present and find out if there are any files missing from the list.

[legendecas]

@gireeshpunathil does that help?

[gireeshpunathil]

@legendecas - thanks for the pointers provided, unfortunately, I did not have time to look at that yet. I will try to squeeze in sometime this week. Meanwhile, does it help if I recreate this in a docker image and pass it to you for your inspection?

[gireeshpunathil]

@legendecas - this was super-quick and super-useful!!!

Cannot find native builtin. "internal/per_context/primordials" 1: 0x564f109d3ee4 node::Abort() 

this is what I get with your patch. How do I fix this now? Any idea?

[legendecas]

That sounds like a problem on the config.gypi, i.e. no files are included in the build -- internal/per_context/primordials is the first file been evaluated during the snapshot process.

Can you ./configure again?

[gireeshpunathil]

In one system:

gireesh@s1:~/node$ grep "primordials" config.gypi
                                         'lib/internal/per_context/primordials.js',
gireesh@s1:~/node$ ./configure
Node.js configure: Found Python 3.8.10...
INFO: configure completed successfully
gireesh@s1:~/node$ grep "primordials" config.gypi
gireesh@s1:~/node$ 

In another system:

root@828e127b5089:/node# grep "primordials" config.gypi
                                         'lib/internal/per_context/primordials.js',
root@828e127b5089:/node# ./configure
Node.js configure: Found Python 3.8.10...
INFO: configure completed successfully
root@828e127b5089:/node# grep "primordials" config.gypi
                                         'lib/internal/per_context/primordials.js',
root@828e127b5089:/node# 

I am confused. :)

[legendecas]

@gireeshpunathil if the first output is the problem behaving one, can you try this command python3 -c 'import sys;sys.path.insert(0, \'tools\');from utils import SearchFiles;print(SearchFiles(\'lib\', \'js\'))' on the system that has the problem?

[gireeshpunathil]

@legendecas - here it is:

['lib/assert/strict.js', 'lib/dns/promises.js', 'lib/fs/promises.js', 'lib/internal/abort_controller.js', 'lib/internal/assert.js', 'lib/internal/async_hooks.js', 'lib/internal/blob.js', 'lib/internal/blocklist.js', 'lib/internal/buffer.js', 'lib/internal/child_process.js', 'lib/internal/cli_table.js', 'lib/internal/constants.js', 'lib/internal/dgram.js', 'lib/internal/dtrace.js', 'lib/internal/encoding.js', 'lib/internal/error_serdes.js', 'lib/internal/errors.js', 'lib/internal/event_target.js', 'lib/internal/fixed_queue.js', 'lib/internal/freelist.js', 'lib/internal/freeze_intrinsics.js', 'lib/internal/heap_utils.js', 'lib/internal/histogram.js', 'lib/internal/http.js', 'lib/internal/idna.js', 'lib/internal/inspector_async_hook.js', 'lib/internal/js_stream_socket.js', 'lib/internal/linkedlist.js', 'lib/internal/net.js', 'lib/internal/options.js', 'lib/internal/priority_queue.js', 'lib/internal/querystring.js', 'lib/internal/repl.js', 'lib/internal/socket_list.js', 'lib/internal/socketaddress.js', 'lib/internal/stream_base_commons.js', 'lib/internal/timers.js', 'lib/internal/trace_events_async_hooks.js', 'lib/internal/tty.js', 'lib/internal/url.js', 'lib/internal/util.js', 'lib/internal/v8_prof_polyfill.js', 'lib/internal/v8_prof_processor.js', 'lib/internal/validators.js', 'lib/internal/watchdog.js', 'lib/internal/worker.js', 'lib/path/posix.js', 'lib/path/win32.js', 'lib/stream/promises.js', 'lib/stream/web.js', 'lib/timers/promises.js', 'lib/util/types.js']

[gireeshpunathil]

@legendecas - I think at this point I should mention this - now that you are bringing python into the issue:

diff --git a/tools/utils.py b/tools/utils.py
index 9d53bd56bd..35a0bd22a0 100644
--- a/tools/utils.py
+++ b/tools/utils.py
@@ -109,7 +109,7 @@ def IsWindows():
 
 
 def SearchFiles(dir, ext):
-  list = glob.glob(dir+ '/**/*.' + ext, recursive=True)
+  list = glob.glob(dir+ '/**/*.' + ext)
   if sys.platform == 'win32':
     list = [ x.replace('\\', '/')for x in list]
   return list

I had to make this change in utils.py, as without that it was giving an error that the the function does not take the second parameter. Do you think this could have any effect on the failure that we are seeing?

[legendecas]

@gireeshpunathil I believe that option is exactly the issue here. Can you share the error when the function takes the second parameter?

[gireeshpunathil]

  ACTION Generating node protocol sources from protocol json aaef1ed75da9c226afcc645ea47e3916fde6ec36.intermediate
Traceback (most recent call last):
  File "tools/js2c.py", line 234, in <module>
    main()
  File "tools/js2c.py", line 219, in main
    js_files = utils.SearchFiles(options.directory, 'js')
  File "/home/gireesh/node/tools/utils.py", line 112, in SearchFiles
    list = glob.glob(dir+ '/**/*.' + ext, recursive=True)
TypeError: glob() got an unexpected keyword argument 'recursive'
  LINK(target) /home/gireesh/node/out/Release/gen-regexp-special-case
make: *** [libnode.target.mk:13: /home/gireesh/node/out/Release/obj/gen/node_javascript.cc] Error 1

[targos]

This should have been fixed by #39293

[gireeshpunathil]

This should have been fixed by #39293

@targos - unfortunately, does not look like it fixed it, I am still getting the same error.

[legendecas]

I was wondering if the ./configure used a different version (as printed above as 3.8.10) with the build step of tools/js2c.py (probably 2.x or 3.5 lower).