code_native broken on official macOS binaries

[distractedlambda]

This issue stems from a discussion I started on the Julia Discourse forum.

In short, the code_native() introspection function seems to work inconsistently (and sometimes break completely) on both Linux and macOS, based on results from several macOS and Linux setups with several Julia versions. I'll be giving the versioninfo() for the two setups I used (macOS 10.13.5 on Apple hardware, Fedora 28 inside a Docker container), along with the results of several code_native() calls to demonstrate the issue. My macOS setup is running the just-released Julia 0.6.4 as of writing this issue, but I experienced the exact same behavior on macOS with Julia 0.6.3.

macOS versioninfo()

Julia Version 0.6.4
Commit 9d11f62bcb (2018-07-09 19:09 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin14.5.0)
  CPU: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell MAX_THREADS=16)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, haswell)

Fedora versioninfo()

Julia Version 0.6.3
Commit d55cadc (2018-05-28 20:20 UTC)
Platform Info:
  OS: Linux (x86_64-redhat-linux)
  CPU: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
  WORD_SIZE: 64
  BLAS: libopenblas (DYNAMIC_ARCH NO_AFFINITY Haswell MAX_THREADS=128)
  LAPACK: libopenblasp.so.0
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, haswell)

Fedora Behavior

As a very simple test case, let's dump the assembly for +(::Int, ::Int):

julia> code_native(+, (Int, Int), :intel)
WARNING: Could not determine size of symbol

That's strange. If we try wrapping it in another function:

julia> f(a, b) = a + b
f (generic function with 1 method)

julia> code_native(f, (Int, Int), :intel)
        .text
Filename: REPL[3]
        push    rbp
        mov     rbp, rsp
Source line: 1
        lea     rax, [rdi + rsi]
        pop     rbp
        ret
        nop     word ptr [rax + rax]

That works, but I see no reason why dumping code for + should fail. I also wonder if the WARNING being displayed in place of an assembly printout is text emitted from external code (i.e. not Julia), since I can't think of what the "Could not determine size of symbol" would mean in Julia.

macOS Behavior

Repeating the same initial test as before:

julia> code_native(+, (Int, Int), :intel)
        .section        __TEXT,__text,regular,pure_instructions
Filename: int.jl
        push    ebp
        dec     eax
        mov     ebp, esp
Source line: 32
        dec     eax
        lea     eax, [edi + esi]
        pop     ebp
        ret
Source line: 32
        nop
        nop
        nop
        nop
        nop
        nop

This time we get an assembly printout, but it's completely wrong. We're seeing only 32-bit instructions, though the actual code is of course 64-bit. Additionally, we've got needless dec eax instructions interspersed, which I suspect aren't actually in the generated code. We can try wrapping + in a function again, but we still get the broken ASM.

---

To show that this problem is limited to code_native, we can try code_llvm, which always gives identical (and correct) output:

julia> code_llvm(+, (Int, Int))

define i64 @"jlsys_+_60291"(i64, i64) #0 !dbg !5 {
top:
  %2 = add i64 %1, %0
  ret i64 %2
}

While I used the + case as a minimal example, no code_native call I've tried has produced correct (i.e. 64-bit) assembly on my macOS setup. Another user (by the username Per in the discussion linked above) was able to get code_native working correctly with a source-built Julia 0.7.0-beta.214 on macOS High Sierra (10.13.x), but experienced the same macOS behavior I did with a downloaded 0.7.0-beta.0 and a downloaded 0.6.3 on macOS Sierra (10.12.x). Finally, with a locally-rebuilt system image in Julia 0.6.3 and macOS Sierra, Per was able to reproduce the WARNING output I experienced on Fedora.

[nalimilan]

I also wonder if the WARNING being displayed in place of an assembly printout is text emitted from external code (i.e. not Julia), since I can't think of what the "Could not determine size of symbol" would mean in Julia.

FWIW, it's printed by Julia:

julia/src/disasm.cpp

Line 409 in 3791357

jl_printf(JL_STDERR, "WARNING: Could not determine size of symbol\n");

[distractedlambda]

@nalimilan Interesting; can't believe I didn't think to just grep through the Julia sources for that message. But this does confirm a couple things for me:

1. code_native works by disassembling in-memory binaries, rather than by making some separate LLVM call to generate assembly source or something (which makes sense).
2. The "symbol" the WARNING refers to is not a Julia symbol, but rather a binary-code-level symbol.

Looking through the source file you linked, it appears that the warning refers to not being able to either look up or determine the size of the requested function. I'm somewhat confused as to why this is a WARNING instead of an ERROR (or better yet a Julia exception), since jl_dump_fptr_asm immediately returns after printing that message without doing its only job. Regardless, it's interesting to me that getting this message seems to (at least to some degree) be related to how the system image was built.

As for the issue being experienced on macOS, it looks like that would have to be occurring in jl_dump_asm_internal, but I would have barely any idea where to start with that. It occurred to me to check something basic like the return value of jl_get_llvm_disasm_target, but I need to get a Julia setup with debug symbols first...

[distractedlambda]

It just occurred to me to try disassembling +(::Int, ::Int) on Fedora with the prebuilt system image disabled, and sure enough, it works. So it looks like the WARNING issue isn't really specific to Linux, but rather to how the system image is built. Since a user-defined function isn't part of the system image, it disassembles just fine.

[distractedlambda]

Also, perhaps this deserves a separate issue, but on either of my setups, running julia -C help prints the machine and feature list 3 times, followed by:

ERROR: Julia and the system image were compiled for different architectures.
Please delete or regenerate sys.{so,dll,dylib}.

And if I try to set a CPU with a misspelled name, I get 7 identical error messages:

'hawell' is not a recognized processor for this target (ignoring processor)
'hawell' is not a recognized processor for this target (ignoring processor)
'hawell' is not a recognized processor for this target (ignoring processor)
'hawell' is not a recognized processor for this target (ignoring processor)
'hawell' is not a recognized processor for this target (ignoring processor)
'hawell' is not a recognized processor for this target (ignoring processor)
'hawell' is not a recognized processor for this target (ignoring processor)

[distractedlambda]

Ignore that; thought the "Close and comment" button was a "Close comment" button.

[yuyichao]

So it looks like the WARNING issue isn't really specific to Linux, but rather to how the system image is built

It usually means that the debug info is stripped or not installed.

And if I try to set a CPU with a misspelled name, I get 7 identical error messages:

Fixed on master. Not going to be fixed on 0.6.

[yuyichao]

running julia -C help prints the machine and feature list 3 times, followed by:

Also fixed on master and not backportable.

[distractedlambda]

@yuyichao Okay, that's good to know.

It usually means that the debug info is stripped or not installed.

So then does that mean that most of the official Linux binaries don't include debug symbols in the system image? I have both julia and julia-devel installed under a Fedora 28 VM, and I still get the WARNING if I try to disassemble a specialization from the system image.

[yuyichao]

So then does that mean that most of the official Linux binaries don't include debug symbols in the system image? I have both julia and julia-devel installed under a Fedora 28 VM

Judging from the package name these are NOT official Linux binaries. The official ones are the ones you can download from julialang.org. -devel package does not have anything to do with debugging. If I read this correctly, you need the -debuginfo package.

[distractedlambda]

@yuyichao Okay, that actually makes a lot more sense then. The julia-devel package described itself as "Julia development, debugging and testing files", which gave me hope it might include debug symbols for the system image, somehow.

So that demystifies the WARNING issue, but still leaves the incorrect-disasm-on-macos issue.

[distractedlambda]

If I build Julia 0.6.4 from source on macOS, I get a system image with debug symbols and 64-bit assembly outputs. This solves the issue for me personally, since I'm okay with building from source, but it's strange that I get such different results from the official binary. I'm changing the name of the issue to reflect what it now seems to be.