Use J2V8 with Node.js v8.x

[drywolf]

@matiwinnetou @aschrijver @irbull

I create this issue to move the discussion about building J2V8 with Node.js v8.x over from #327
(and to avoid potential duplicate efforts)

With the tests in v8/debug/ disabled and the additions from @matiwinnetou I was now able to build & run tests successfully for Node.js v8.2.1 on the marked platforms below

By looking into android-configure and the Android NDK docs I hope it will be not too hard to get Android working also 🤞

(Android needs fixing, see this)

Platform	NJS Version	Build	Tests
Android x86	8.2.1	fails	-
	8.4.0	✓	some tests ok, some crash (same behavior as in 7.4.0)
Android ARM	8.2.1	TBD	TBD
	8.4.0	TBD	TBD
Alpine Linux	8.2.1	✓	all ok
	8.4.0	✓	some tests ok some crash the JVM
Linux	8.2.1	✓	all ok
	8.4.0	✓	some tests ok some crash the JVM
Mac OS	8.2.1	✓	all ok
	8.4.0	✓	some tests ok some crash the JVM
Windows	8.2.1	✓	all ok
	8.4.0	✓	all ok

[drywolf]

I spent quite some time yesterday and today the last couple of hours to figure out the right way to build a(ny) Node 8.x version with the latest Android NDK (nightly).

Just some minutes ago I've had the big breakthrough, and at least I got a Node.js build of a static lib that ran through till the end. Now the next step is to try and get CMake to correctly link to what was produced there ... so we are not done just yet. But at least for Node.js I think we might have something workable. I will keep posting updates as they roll in.

PS: I am building for x86/ia32 Android

[matiwinnetou]

Wow, didn't know it was that hard to build android with NodeJS 8.2.1. Great job!

I can add you as a collaborator to my branch:
https://github.com/eclipsesource/J2V8/compare/master...matiwinnetou:nodejs8?expand=1

or how do we work together on this?

[drywolf]

I think I will push my current progress as a new branch to my fork tomorrow.
I have to clean everything up a bit before I do that, because I had to try out so many different combinations of build configurations, right now it's just a huge pile of commented code with the few lines that finally worked somewhere in between 😆

At the moment I am sorting out the CMake build ... I have already made some progress, but now I am stuck with some stdlib header problems.

To elaborate ... the whole build is now using Clang and libc++ as the stdlib (this seems to be recommended for the newer NDK versions). If you try to build the latest Node.js with NDK+GCC you pretty early do hit a brick wall from what I (and others) experienced.

The error that I am now getting after CMake has generated the makefiles and Clang starts compiling the J2V8 project are the following:

Scanning dependencies of target j2v8
[ 50%] Building CXX object CMakeFiles/j2v8.dir/jni/com_eclipsesource_v8_V8Impl.cpp.o
In file included from /j2v8/jni/com_eclipsesource_v8_V8Impl.cpp:12:
In file included from /j2v8/node/deps/v8/include/libplatform/libplatform.h:9:
In file included from /j2v8/node/deps/v8/include/libplatform/v8-tracing.h:8:
In file included from /tmp/android-toolchain/bin/../include/c++/v1/fstream:169:
In file included from /tmp/android-toolchain/bin/../include/c++/v1/ostream:138:
In file included from /tmp/android-toolchain/bin/../include/c++/v1/ios:216:
In file included from /tmp/android-toolchain/bin/../include/c++/v1/__locale:15:
In file included from /tmp/android-toolchain/bin/../include/c++/v1/string:470:
In file included from /tmp/android-toolchain/bin/../include/c++/v1/string_view:171:
In file included from /tmp/android-toolchain/bin/../include/c++/v1/__string:56:
In file included from /tmp/android-toolchain/bin/../include/c++/v1/algorithm:640:
In file included from /tmp/android-toolchain/bin/../include/c++/v1/memory:629:
In file included from /tmp/android-toolchain/bin/../include/c++/v1/typeinfo:61:
In file included from /tmp/android-toolchain/bin/../include/c++/v1/exception:82:
/tmp/android-toolchain/bin/../include/c++/v1/cstdlib:114:9: error: no member named 'strtold' in the global namespace
using ::strtold;
      ~~^
/tmp/android-toolchain/bin/../include/c++/v1/cstdlib:132:9: error: no member named '_Exit' in the global namespace
using ::_Exit;
      ~~^
/tmp/android-toolchain/bin/../include/c++/v1/cstdlib:153:9: error: no member named 'at_quick_exit' in the global namespace
using ::at_quick_exit;
      ~~^
/tmp/android-toolchain/bin/../include/c++/v1/cstdlib:154:9: error: no member named 'quick_exit' in the global namespace
using ::quick_exit;
      ~~^
In file included from /j2v8/jni/com_eclipsesource_v8_V8Impl.cpp:12:
In file included from /j2v8/node/deps/v8/include/libplatform/libplatform.h:9:
In file included from /j2v8/node/deps/v8/include/libplatform/v8-tracing.h:8:
In file included from /tmp/android-toolchain/bin/../include/c++/v1/fstream:169:
In file included from /tmp/android-toolchain/bin/../include/c++/v1/ostream:138:
In file included from /tmp/android-toolchain/bin/../include/c++/v1/ios:216:
In file included from /tmp/android-toolchain/bin/../include/c++/v1/__locale:15:
In file included from /tmp/android-toolchain/bin/../include/c++/v1/string:474:
In file included from /tmp/android-toolchain/bin/../include/c++/v1/cwchar:107:
/tmp/android-toolchain/bin/../include/c++/v1/cwctype:68:9: error: no member named 'iswblank' in the global namespace; did you mean 'isblank'?
using ::iswblank;
      ~~^
/tmp/android-toolchain/sysroot/usr/include/ctype.h:66:5: note: 'isblank' declared here
int isblank(int);
    ^
In file included from /j2v8/jni/com_eclipsesource_v8_V8Impl.cpp:12:
In file included from /j2v8/node/deps/v8/include/libplatform/libplatform.h:9:
In file included from /j2v8/node/deps/v8/include/libplatform/v8-tracing.h:8:
In file included from /tmp/android-toolchain/bin/../include/c++/v1/fstream:169:
In file included from /tmp/android-toolchain/bin/../include/c++/v1/ostream:138:
In file included from /tmp/android-toolchain/bin/../include/c++/v1/ios:216:
In file included from /tmp/android-toolchain/bin/../include/c++/v1/__locale:15:
In file included from /tmp/android-toolchain/bin/../include/c++/v1/string:474:
/tmp/android-toolchain/bin/../include/c++/v1/cwchar:127:9: error: no member named 'vfwscanf' in the global namespace
using ::vfwscanf;
      ~~^
/tmp/android-toolchain/bin/../include/c++/v1/cwchar:128:9: error: no member named 'vswscanf' in the global namespace
using ::vswscanf;
      ~~^
/tmp/android-toolchain/bin/../include/c++/v1/cwchar:138:9: error: no member named 'wcstof' in the global namespace
using ::wcstof;
      ~~^

@DanAlbert @matiwinnetou @aschrijver any ideas 🤔 ?

PS: might be the same problem as in android/ndk#452

[DanAlbert]

You're missing the libandroid_support headers. I'd recommend using the cmake toolchain file that's in the NDK at $NDK/build/cmake/android.toolchain.cmake.

libc++ as the stdlib (this seems to be recommended for the newer NDK versions)

Starting with the not yet released r16, yes, but if you need a full C++11 (or newer) STL then it's your only option.

[matiwinnetou]

@drywolf I wish I could help, this is too much outside of my area of expertise. Not only I don't have extensive c++ experience delicately saying but also no Android experience.

[aschrijver]

@drywolf same here, though I am learning a lot. I have updated nodejs/node#14783 with latest findings, and I can do some test builds this weekend.

[drywolf]

You're missing the libandroid_support headers.

@DanAlbert Thanks for the hint, I already dug through the mentioned issue to see that this is the problem.
What I can not figure out though is how to correctly include them. I am using a standalone toolchain as described by the NDK docs to build Node.js and now also the j2v8 lib.
(the Node.js build already finished ok with that)

The r16 NDK I downloaded from the nightly builds (4280707), as mentioned here.
And from that I created a standalone toolchain:

RUN $NDK/build/tools/make_standalone_toolchain.py \
    --arch x86 --api 17 --stl=libc++ --install-dir /tmp/android-toolchain

My own CMake toolchain file looks like this:

set(CMAKE_SYSTEM_NAME Android)
set(CMAKE_SYSTEM_VERSION 17) # API level

set(CMAKE_ANDROID_ARCH x86)
set(CMAKE_ANDROID_ARCH_ABI x86)
set(CMAKE_ANDROID_STANDALONE_TOOLCHAIN /tmp/android-toolchain)

# set(CMAKE_ANDROID_STL_TYPE c++_shared)
set(CMAKE_ANDROID_STL_TYPE c++_static)

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -stdlib=libc++ -std=c++11 -fPIC")

When I now point CMake to use the android.toolchain.cmake file that is in the NDK directory (not the generated toolchain directory that I currently use) then CMake tries to locate the compiler binaries also in the NDK directory (among maybe also some other things)

-- Android: Targeting API '14' with architecture 'arm', ABI 'armeabi-v7a', and processor 'armv7-a'
-- Android: Selected Clang toolchain 'arm-linux-androideabi-clang' with GCC toolchain 'arm-linux-androideabi-4.9'
CMake Error at CMakeLists.txt:16 (project):
  The CMAKE_C_COMPILER:

    /temp/docker/android/android-ndk-r16-beta1-canary/toolchains//prebuilt/linux-x86_64/bin/clang

  is not a full path to an existing compiler tool.

(manually redirecting CMAKE_C_COMPILER & CXX compiler back to the standalone paths has no effect)

Also the first two lines about API 14 and armeabi-v7a seem entirely alien to me, because I am targeting API 17 / x86 with my build.

@DanAlbert I would not expect to point back to the NDK resources after I have run make_standalone_toolchain.py ... but if I really have to use the cmake toolchain file from the NDK directory, then is there something else that I need to set to make it use the right environment ? (API 17 and x86 in my case) ... I already tried to set the usual CMake variables for that, but nothing changed.

Thanks

[drywolf]

I have now made it to a point where the android support headers seem to be included property (I took the relevant parts from the NDK cmake files), but now there is another error about the typedef for size_t missing:

In file included from /j2v8/jni/com_eclipsesource_v8_V8Impl.cpp:12:
In file included from /j2v8/node/deps/v8/include/libplatform/libplatform.h:9:
In file included from /j2v8/node/deps/v8/include/libplatform/v8-tracing.h:8:
In file included from /temp/docker/android/android-ndk-r16-beta1-canary/sources/cxx-stl/llvm-libc++/include/fstream:169:
In file included from /temp/docker/android/android-ndk-r16-beta1-canary/sources/cxx-stl/llvm-libc++/include/ostream:138:
In file included from /temp/docker/android/android-ndk-r16-beta1-canary/sources/cxx-stl/llvm-libc++/include/ios:215:
In file included from /temp/docker/android/android-ndk-r16-beta1-canary/sources/cxx-stl/llvm-libc++/include/iosfwd:90:
In file included from /temp/docker/android/android-ndk-r16-beta1-canary/sources/cxx-stl/llvm-libc++/include/wchar.h:119:
In file included from /temp/docker/android/android-ndk-r16-beta1-canary/sources/android/support/include/wchar.h:32:
In file included from /tmp/android-toolchain/sysroot/usr/include/wchar.h:32:
In file included from /temp/docker/android/android-ndk-r16-beta1-canary/sources/cxx-stl/llvm-libc++/include/stdio.h:108:
In file included from /temp/docker/android/android-ndk-r16-beta1-canary/sources/android/support/include/stdio.h:31:
In file included from /tmp/android-toolchain/sysroot/usr/include/stdio.h:42:
In file included from /tmp/android-toolchain/sysroot/usr/include/sys/types.h:39:
/tmp/android-toolchain/sysroot/usr/include/bits/pthread_types.h:38:3: error: unknown type name 'size_t'
  size_t stack_size;

[drywolf]

I have managed to sort this out and also some linker errors after that, meaning I got a native j2v8 binary using Node.js 8.x for the first time ... but now comes the interesting part to see if this native library was correctly compiled so it can be loaded by the J2V8 AAR 🤞

[drywolf]

Not quite yet, but I think I'm still on the right track:

D/dalvikvm( 2306): Trying to load lib /data/app-lib/com.eclipsesource.v8.test-2/libj2v8.so 0xb1f707a0
E/dalvikvm( 2306): dlopen("/data/app-lib/com.eclipsesource.v8.test-2/libj2v8.so") failed: dlopen failed: cannot locate symbol "htons" referenced by "libj2v8.so"...
I/TestRunner( 2306): failed: testGetBooleanBiggerArray(com.eclipsesource.v8.V8ArrayTest)

@DanAlbert any idea about this one ? Where should this function come from when linking with the NDK ?

[DanAlbert]

Our toolchain file is not meant to work alongside a standalone toolchain. It's supposed to be used directly. Standalone toolchains are intended for interfacing with arbitrary build systems that are difficult to use with the NDK (autoconf, mostly). We support CMake, and that should Just Work (aside from any possible source compatibility issues with Android).

Can you share your build script? Just following along here it sounds like you're headed down the wrong track and having to put in a lot of effort just to handle the NDK. That shouldn't be needed for a CMake project.

E/dalvikvm( 2306): dlopen("/data/app-lib/com.eclipsesource.v8.test-2/libj2v8.so") failed: dlopen failed: cannot locate symbol "htons" referenced by "libj2v8.so"...

https://android.googlesource.com/platform/bionic/+/master/libc/libc.map.txt

htons wasn't in bionic (not as a real symbol, anyway, it may have been an inline because for some reason bionic was full of inlines prior to L) until L. If you have a result to this symbol from the build you were targeting at least L, but since that's an error from dalvik you're running on K at the latest. https://android.googlesource.com/platform/ndk/+/master/docs/user/common_problems.md#target-api-set-higher-than-device-api

[drywolf]

I pushed my progress to an experimental branch, with the following additions for building with Node 8.x and the r16 NDK beta:

• creating the standalone-toolchain from the NDK
• the CMake toolchain file
• build-script that compiles the J2V8 shared native library (using the CMake toolchain file from above)

The htons is used by Node.js, which is statically linked into the j2v8 shared library.

If you have a result to this symbol from the build you were targeting at least L, but since that's an error from dalvik you're running on K at the latest.

I think I get what you are saying there, but could you give me a hint how I could find out if that is the case. When I look at the build-scripts & configurations that I am using I can't see any origin for such a problem (mainly I'm looking at the API 17 level, which is below the platform level that I am creating the emulator from (android-19).
Maybe I'm getting it wrong, but I thought I just had to compare the two numbers to know if code is going to work or not.

@DanAlbert Thanks for helping. Up until now I could figure out most things myself. But for this specific error I might be out of ideas.

[drywolf]

Now I am trying to just use the NDK + toolchain.cmake file (based on your suggestion from above & this blog post)

When I do that I get the following output, which is similar to what I already had seen during my previous attempts, but by setting the right variables at the CMake CLI at least now the platform and api-level seem to be alright.

SHELL building android@x86 => j2v8cmake
-- Android: Targeting API '17' with architecture 'x86', ABI 'x86', and processor 'i686'
-- Android: Selected Clang toolchain 'x86-clang' with GCC toolchain 'x86-4.9'
CMake Error at CMakeLists.txt:16 (project):
  The CMAKE_C_COMPILER:

    /temp/docker/android/android-ndk-r16-beta1-canary/toolchains//prebuilt/linux-x86_64/bin/clang

  is not a full path to an existing compiler tool.

What is strange about this error though (and I saw this before, but I thought it was just a irrelevant glitch in the path) is the double forward-slash between "toolchains" and "prebuilt".
I checked the directories & files in the NDK and it looks like the actually expected right path should be:
/temp/docker/android/android-ndk-r16-beta1-canary/toolchains/llvm/prebuilt/linux-x86_64

The llvm is apparently missing, now I am trying to figure out if there is another CMake Android/NDK parameter that I am missing.

PS: the point of the problem might be along these lines of the CMake scripts: https://github.com/Kitware/CMake/blob/0e9634d2c90fd748774f9238219cb1661550f233/Modules/Platform/Android/Determine-Compiler-NDK.cmake#L230-L232