Requiring rugged gem causes segfault

[liamwhite]

This took me hours to find.

$ pry
[1] pry(main)> require 'rugged'
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fa42f10a02a, pid=9844, tid=0x00007fa43029e700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_151-b12) (build 1.8.0_151-b12)
# Java VM: GraalVM 0.31 (25.71-b01-internal-jvmci-0.40 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0xade02a]  Unsafe_SetNativeAddress+0xca
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/backups/truffletest/hs_err_pid9844.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
Aborted

Stack: [0x00007fa43019e000,0x00007fa43029f000],  sp=0x00007fa43028a240,  free space=944k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xade02a]  Unsafe_SetNativeAddress+0xca
j  sun.misc.Unsafe.putAddress(JJ)V+0
j  com.oracle.truffle.llvm.runtime.memory.LLVMMemory.putAddress(JJ)V+25
j  com.oracle.truffle.llvm.runtime.memory.LLVMMemory.putAddress(JLcom/oracle/truffle/llvm/runtime/LLVMAddress;)V+6
j  com.oracle.truffle.llvm.runtime.global.LLVMGlobalWriteNode$WriteObjectNode.doNative(Lcom/oracle/truffle/api/frame/VirtualFrame;Lcom/oracle/truffle/llvm/runtime/global/LLVMGlobal;Ljava/lang/Object;Lcom/oracle/truffle/llvm/runtime/global/LLVMGlobal$GetNativePointer;Lcom/oracle/truffle/llvm/runtime/nodes/api/LLVMToNativeNode;)Ljava/lang/Object;+21
j  com.oracle.truffle.llvm.runtime.global.LLVMGlobalWriteNodeFactory$WriteObjectNodeGen.executeAndSpecialize(Lcom/oracle/truffle/api/frame/VirtualFrame;Lcom/oracle/truffle/llvm/runtime/global/LLVMGlobal;Ljava/lang/Object;)Ljava/lang/Object;+182
j  com.oracle.truffle.llvm.runtime.global.LLVMGlobalWriteNodeFactory$WriteObjectNodeGen.execute(Lcom/oracle/truffle/api/frame/VirtualFrame;Lcom/oracle/truffle/llvm/runtime/global/LLVMGlobal;Ljava/lang/Object;)Ljava/lang/Object;+82
j  com.oracle.truffle.llvm.nodes.memory.store.LLVMGlobalVariableStoreNode.doNative(Lcom/oracle/truffle/api/frame/VirtualFrame;Lcom/oracle/truffle/llvm/runtime/LLVMAddress;Lcom/oracle/truffle/llvm/runtime/global/LLVMGlobalWriteNode$WriteObjectNode;)Ljava/lang/Object;+7
j  com.oracle.truffle.llvm.nodes.memory.store.LLVMGlobalVariableStoreNodeGen.executeAndSpecialize(Lcom/oracle/truffle/api/frame/VirtualFrame;Ljava/lang/Object;)Ljava/lang/Object;+152
j  com.oracle.truffle.llvm.nodes.memory.store.LLVMGlobalVariableStoreNodeGen.executeGeneric(Lcom/oracle/truffle/api/frame/VirtualFrame;)Ljava/lang/Object;+243
j  com.oracle.truffle.llvm.nodes.others.LLVMStaticInitsBlockNode.execute(Lcom/oracle/truffle/api/frame/VirtualFrame;)Ljava/lang/Object;+44
J 5618 C1 org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callProxy(Lcom/oracle/truffle/api/frame/VirtualFrame;)Ljava/lang/Object; (138 bytes) @ 0x00007fa414bc7e24 [0x00007fa414bc7dc0+0x64]
J 5617 C1 org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callRoot([Ljava/lang/Object;)Ljava/lang/Object; (53 bytes) @ 0x00007fa414bc795c [0x00007fa414bc78a0+0xbc]
J 11153 JVMCI org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callBoundary([Ljava/lang/Object;)Ljava/lang/Object; (33 bytes) @ 0x00007fa41490328c [0x00007fa4149031e0+0xac]
J 8922 C1 org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.call([Ljava/lang/Object;)Ljava/lang/Object; (19 bytes) @ 0x00007fa4153da95c [0x00007fa4153da720+0x23c]
j  com.oracle.truffle.llvm.Runner.handleParserResult(Lcom/oracle/truffle/llvm/runtime/LLVMContext;Lcom/oracle/truffle/llvm/parser/LLVMParserResult;)V+95
j  com.oracle.truffle.llvm.Runner.parse(Lcom/oracle/truffle/llvm/runtime/LLVMLanguage;Lcom/oracle/truffle/llvm/runtime/LLVMContext;Lcom/oracle/truffle/api/source/Source;Lcom/oracle/truffle/llvm/runtime/LLVMContext$ExternalLibrary;Ljava/nio/ByteBuffer;)Lcom/oracle/truffle/api/CallTarget;+204
j  com.oracle.truffle.llvm.Runner.parse(Lcom/oracle/truffle/llvm/runtime/LLVMLanguage;Lcom/oracle/truffle/llvm/runtime/LLVMContext;Lcom/oracle/truffle/api/source/Source;)Lcom/oracle/truffle/api/CallTarget;+169
j  com.oracle.truffle.llvm.Sulong.parse(Lcom/oracle/truffle/api/TruffleLanguage$ParsingRequest;)Lcom/oracle/truffle/api/CallTarget;+22
j  com.oracle.truffle.api.TruffleLanguage$ParsingRequest.parse(Lcom/oracle/truffle/api/TruffleLanguage;)Lcom/oracle/truffle/api/CallTarget;+2
j  com.oracle.truffle.api.TruffleLanguage.parse(Lcom/oracle/truffle/api/source/Source;Lcom/oracle/truffle/api/nodes/Node;Lcom/oracle/truffle/api/frame/MaterializedFrame;[Ljava/lang/String;)Lcom/oracle/truffle/api/CallTarget;+17
j  com.oracle.truffle.api.TruffleLanguage$Env.parse(Lcom/oracle/truffle/api/source/Source;[Ljava/lang/String;)Lcom/oracle/truffle/api/CallTarget;+32
j  org.truffleruby.language.loader.FeatureLoader.parseSource(Lcom/oracle/truffle/api/source/Source;)Lcom/oracle/truffle/api/CallTarget;+12
j  org.truffleruby.language.loader.FeatureLoader$$Lambda$828.accept(Ljava/lang/Object;)V+8
j  org.truffleruby.cext.Linker.loadLibrary(Ljava/io/File;Ljava/util/function/Consumer;Ljava/util/function/Consumer;)V+327
j  org.truffleruby.language.loader.FeatureLoader.loadCExtLibrary(Ljava/lang/String;)V+76
j  org.truffleruby.language.loader.RequireNode.requireCExtension(Ljava/lang/String;Ljava/lang/String;)V+52
j  org.truffleruby.language.loader.RequireNode.require(Ljava/lang/String;Lcom/oracle/truffle/api/profiles/BranchProfile;Lcom/oracle/truffle/api/profiles/BranchProfile;Lcom/oracle/truffle/api/profiles/ConditionProfile;Lorg/truffleruby/core/string/StringNodes$MakeStringNode;)Z+284

[...]
open("/opt/graalvm-0.31/jre/languages/ruby/lib/ruby/gems/2.3.0/gems/rugged-0.26.0/ext/rugged/rugged_tag.c", O_RDONLY) = 56
fstat(56, {st_mode=S_IFREG|0644, st_size=6147, ...}) = 0
read(56, "/*\n * Copyright (C) the Rugged c"..., 6147) = 6147
read(56, "", 1)                         = 0
close(56)                               = 0
stat("/opt/graalvm-0.31/jre/languages/ruby/lib/cext/ruby/ruby.h", {st_mode=S_IFREG|0664, st_size=61304, ...}) = 0
access("/opt/graalvm-0.31/jre/languages/ruby/lib/cext/ruby/ruby.h", R_OK) = 0
open("/opt/graalvm-0.31/jre/languages/ruby/lib/cext/ruby/ruby.h", O_RDONLY) = 56
fstat(56, {st_mode=S_IFREG|0664, st_size=61304, ...}) = 0
read(56, "/*******************************"..., 61304) = 61304
read(56, "", 1)                         = 0
close(56)                               = 0
stat("/opt/graalvm-0.31/jre/languages/ruby/lib/ruby/gems/2.3.0/gems/rugged-0.26.0/ext/rugged/rugged.h", {st_mode=S_IFREG|0644, st_size=5554, ...}) = 0
access("/opt/graalvm-0.31/jre/languages/ruby/lib/ruby/gems/2.3.0/gems/rugged-0.26.0/ext/rugged/rugged.h", R_OK) = 0
open("/opt/graalvm-0.31/jre/languages/ruby/lib/ruby/gems/2.3.0/gems/rugged-0.26.0/ext/rugged/rugged.h", O_RDONLY) = 56
fstat(56, {st_mode=S_IFREG|0644, st_size=5554, ...}) = 0
read(56, "/*\n * Copyright (C) the Rugged c"..., 5554) = 5554
read(56, "", 1)                         = 0
close(56)                               = 0
stat("/opt/graalvm-0.31/jre/languages/ruby/lib/ruby/gems/2.3.0/gems/rugged-0.26.0/vendor/libgit2/include/git2/types.h", {st_mode=S_IFREG|0644, st_size=14245, ...}) = 0
access("/opt/graalvm-0.31/jre/languages/ruby/lib/ruby/gems/2.3.0/gems/rugged-0.26.0/vendor/libgit2/include/git2/types.h", R_OK) = 0
open("/opt/graalvm-0.31/jre/languages/ruby/lib/ruby/gems/2.3.0/gems/rugged-0.26.0/vendor/libgit2/include/git2/types.h", O_RDONLY) = 56
fstat(56, {st_mode=S_IFREG|0644, st_size=14245, ...}) = 0
read(56, "/*\n * Copyright (C) the libgit2 "..., 14245) = 14245
read(56, "", 1)                         = 0
close(56)                               = 0
stat("/opt/graalvm-0.31/jre/languages/ruby/lib/ruby/gems/2.3.0/gems/rugged-0.26.0/vendor/libgit2/include/git2/oid.h", {st_mode=S_IFREG|0644, st_size=8277, ...}) = 0
access("/opt/graalvm-0.31/jre/languages/ruby/lib/ruby/gems/2.3.0/gems/rugged-0.26.0/vendor/libgit2/include/git2/oid.h", R_OK) = 0
open("/opt/graalvm-0.31/jre/languages/ruby/lib/ruby/gems/2.3.0/gems/rugged-0.26.0/vendor/libgit2/include/git2/oid.h", O_RDONLY) = 56
fstat(56, {st_mode=S_IFREG|0644, st_size=8277, ...}) = 0
read(56, "/*\n * Copyright (C) the libgit2 "..., 8277) = 8277
read(56, "", 1)                         = 0
close(56)                               = 0
stat("/usr/include/x86_64-linux-gnu/sys/types.h", {st_mode=S_IFREG|0644, st_size=6705, ...}) = 0
access("/usr/include/x86_64-linux-gnu/sys/types.h", R_OK) = 0
open("/usr/include/x86_64-linux-gnu/sys/types.h", O_RDONLY) = 56
fstat(56, {st_mode=S_IFREG|0644, st_size=6705, ...}) = 0
read(56, "/* Copyright (C) 1991-2014 Free "..., 6705) = 6705
read(56, "", 1)                         = 0
close(56)                               = 0
lseek(26, 1574643, SEEK_SET)            = 1574643
read(26, "PK\3\4\24\0\0\0\0\0\253LFL)DA\241\22\30\0\0\22\30\0\0V\0\0\0", 30) = 30
lseek(26, 1574759, SEEK_SET)            = 1574759
read(26, "\312\376\272\276\0\0\0004\0\270\n\0!\0l\t\0\37\0m\t\0\37\0n\t\0\26\0o\n\0"..., 1024) = 1024
lseek(26, 1575783, SEEK_SET)            = 1575783
read(26, "/global/LLVMGlobal;\1\0\5state\1\0\rSt"..., 2048) = 2048
lseek(26, 1577831, SEEK_SET)            = 1577831
read(26, "com/oracle/truffle/api/frame/Mat"..., 3090) = 3090
futex(0x7fc2c4081354, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fc2c4081350, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x7fc2c400b254, FUTEX_WAIT_PRIVATE, 8079, NULL) = 0
futex(0x7fc2c400b228, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7fc2c40c5154, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fc2c40c5150, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
lseek(26, 1502925, SEEK_SET)            = 1502925
read(26, "PK\3\4\24\0\0\0\0\0\253LFL\372\367\21wL\6\0\0L\6\0\0m\0\0\0", 30) = 30
lseek(26, 1503064, SEEK_SET)            = 1503064
read(26, "\312\376\272\276\0\0\0004\0001\n\0\4\0#\t\0\3\0$\7\0%\7\0&\1\0\5nex"..., 1024) = 1024
lseek(26, 1504088, SEEK_SET)            = 1504088
read(26, "GlobalFactory$GetNativePointerNo"..., 588) = 588
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0} ---

[eregon]

This took me hours to find.

What took hours to find?

In general, our C-ext support is still in an early stage, where we support the bundled extensions and a few gems we tried but C extensions in the wild don't necessarily build or work out of the box.

How did you install rugged?
It seems to fail at me by not finding the bundled libgit2 (maybe due to a missing -L flag, but I also don't see any .so in lib/ruby/gems/2.3.0/gems/rugged-0.26.0).

[liamwhite]

What took hours to find?

It took me a long time to pinpoint that the bug came from Rugged (it was only the strace that clued me in to it.) This is just me venting my frustrations about the debugging process here, especially as it relates to this segfault.

@eregon To build rugged, you need to have cmake and the libgit2 libary+development headers installed.

$ apt-cache policy libgit2-dev
libgit2-dev:
  Installed: 0.21.1-3
  Candidate: 0.21.1-3
  Version table:
 *** 0.21.1-3 0
        500 http://http.debian.net/debian/ jessie/main amd64 Packages
        100 /var/lib/dpkg/status

$ apt-cache policy cmake
cmake:
  Installed: 3.0.2-1+deb8u1
  Candidate: 3.0.2-1+deb8u1
  Version table:
 *** 3.0.2-1+deb8u1 0
        500 http://http.debian.net/debian/ jessie/main amd64 Packages
        100 /var/lib/dpkg/status

Just to be sure, I rebuilt it:

$ gem pristine rugged
Restoring gems to pristine condition...
Building native extensions.  This could take a while...
Restored rugged-0.26.0

$ pry
[1] pry(main)> require 'rugged'
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fd1f105802a, pid=16502, tid=0x00007fd1f21ec700
# (...)

[eregon]

I see, thanks for the report.
Feel free to report the issue as soon as you get a SEGV, we might be able to guess where it comes from.

Normally we have nice stacktraces going through C and Ruby for errors happening in C extensions.
However, this doesn't work with SEGV (which are most often caused by native extensions or libraries).

Rugged should normally use its vendored copy of libgit2, according to its README. But that doesn't seem to work yet with TruffleRuby.

Running gem install rugged -- --use-system-libraries on MRI 2.4.2 shows:

Fetching: rugged-0.26.0.gem (100%)
Building native extensions with: '--use-system-libraries'
This could take a while...
ERROR:  Error installing rugged:
	ERROR: Failed to build gem native extension.

    current directory: /home/eregon/.gem/ruby/2.4.2/gems/rugged-0.26.0/ext/rugged
/home/eregon/.rubies/ruby-2.4.2/bin/ruby -r ./siteconf20180208-32584-1hblogq.rb extconf.rb --use-system-libraries
checking for gmake... yes
Building Rugged using system libraries.
libgit2 version is not compatible, expected ~> 0.26.0
*** extconf.rb failed ***

It seems rugged is expecting libgit2 0.26.x, which both of us do not have.
This sounds like it could be the cause of the SEGV, an incompatible libgit2.
Not sure why TruffleRuby doesn't hit that version check.

It seems rugged builds vendor/libgit2/build/libgit2.a and statically link it in lib/rugged/rugged.so.
This seems a bit tricky and we'll have to investigate how to do it in TruffleRuby.

So, in conclusion, we need to figure out why the version check doesn't work for us and why we link against the system libgit2 even though we should link against the vendored libgit2.

Investigating the SEGV on a wrong version of libgit2 seems less relevant as it might just be that older libgit2 versions have some bug or the extension is incompatible with them.

[liamwhite]

@eregon
By patching lib/mri/mkmf.rb and adding this line at the end:

MakeMakefile::TRY_LINK = "clang -o conftest $(CPPFLAGS) /opt/graalvm-0.31/jre/languages/ruby/lib/cext/ruby.bc /opt/graalvm-0.31/jre/languages/ruby/lib/cext/trufflemock.bc $(src) $(INCFLAGS) -g -Wimplicit-function-declaration -Wno-unknown-warning-option -Wno-int-conversion -Wno-int-to-pointer-cast -Wno-unused-value -Wno-incompatible-pointer-types -ferror-limit=500 $(LIBPATH) $(LIBS)"

I was able to get the rugged gem to successfully compile without a system libgit2. (The omission of LIBPATH seems like an oversight that can be fixed here).

However, this output appeared in the compile log:

compiling rugged_tree.c
linking shared-object rugged/rugged.su
Could not find a search path for git2 -- passing libgit2.so on to Sulong
current directory: /opt/graalvm-0.31/jre/languages/ruby/lib/ruby/gems/2.3.0/gems/rugged-0.26.0/ext/rugged
make "DESTDIR=" install
install -c -p -m 0755 rugged.su ./.gem.20180209-23654-k1iodk/rugged
To see why this extension failed to compile, please check the mkmf.log which can be found here:

  /opt/graalvm-0.31/jre/languages/ruby/lib/ruby/gems/2.3.0/extensions/x86_64-linux/2.3.0/rugged-0.26.0/mkmf.log
Successfully installed rugged-0.26.0
Parsing documentation for rugged-0.26.0
Parsing sources...
100% [87/87]  lib/rugged/walker.rb                     
Installing ri documentation for rugged-0.26.0
Done installing documentation for rugged after 18 seconds
1 gem installed

and when I try to require the gem, there is an error:

[2] pry(main)> require 'rugged'
com.oracle.truffle.llvm.runtime.LLVMContext$ExternalLibrary@ac6a78e0 not found!
libgit2.so: cannot open shared object file: No such file or directory
RuntimeError: libgit2.so: cannot open shared object file: No such file or directory
from /opt/graalvm-0.31/jre/languages/ruby/lib/mri/rubygems/core_ext/kernel_require.rb:55:in `gem_original_require'

[liamwhite]

linker.rb is called with

-L/opt/graalvm-0.31/jre/languages/ruby/lib/ruby/gems/2.3.0/gems/rugged-0.26.0/ext/rugged/../../vendor/libgit2/build -L. -L/opt/graalvm-0.31/jre/languages/ruby/lib -lgit2

; at line 48 it passes libraries=["libgit2.so"] (does not exist) to java, which just ignores the file (because it does not exist, the try block does not execute).

linker.rb needs to have a facility for handling .a static libraries (in this case libgit2.a, which is in LIBPATH) as well as .so dynamic libraries. This is far beyond my area of expertise, so I will stop here.

[eregon]

@liamwhite Thank you for investigating further and finding some bugs & fixes!
We'll integrate your findings and look at how to handle these static .a libraries.

[liamwhite]

@eregon It should just be a matter of unarchiving the .a file and directly including all its object files into the .su archive.

[chrisseaton]

But do the .a files contain machine code or LLVM bitcode? Sulong cannot interpret machine code (it can, but not in this context so we'll ignore that for now). It can dynamically link to machine code, but not statically. If the .a file contains static machine code libraries, then we need to do something else, like link it into a dynamic library.

[eregon]

I think we need a way to either compile libgit to bitcode or to a .so.
It might also be possible to include the bitcode alongside a native .o (need to check with the Sulong team how and if that's portable).