Bootstrap failure on ARMv7

[nalimilan]

When trying to build a nightly RPM on Fedora 22 armv7hl, I get failures during the bootstrap step. Depending on the value of JULIA_TARGET_CPU I use, the error is different:

• With native (JULIA_TARGET_CPU not passed):

osutils.jl
error during bootstrap:
LoadError(at "sysimg.jl" line 82: LoadError(at "osutils.jl" line 38: BoundsError(a=Array{Any, 1}[(), ()], i=(0,))))
Makefile:168: recipe for target '/builddir/build/BUILD/julia/build/usr/lib/julia/sys0.o' failed

https://kojipkgs.fedoraproject.org//work/tasks/26/9290026/build.log

• With cortex-a8:

osutils.jl
error during bootstrap:
LoadError(at "sysimg.jl" line 82: LoadError(at "osutils.jl" line 38: Base.TypeError(func=:getfield, context="", expected=DataType, got=<?::false>)))
signal (11): Segmentation fault
/bin/sh: line 1:  7563 Segmentation fault      (core dumped) /builddir/build/BUILD/julia/build/usr/bin/julia -C cortex-a8 --build /builddir/build/BUILD/julia/build/usr/lib/julia/sys0 sysimg.jl
Makefile:168: recipe for target '/builddir/build/BUILD/julia/build/usr/lib/julia/sys0.o' failed

https://kojipkgs.fedoraproject.org//work/tasks/1460/9291460/build.log

I'm not familiar with ARM at all, so I'm not sure which one is the more reasonable (maybe none). Note that I'm using USE_SYSTEM_LLVM=1, which is LLVM 3.5. A difficulty is that gcc, which is used in the build, uses -march=armv7-a on Fedora armv7hl, but LLVM does not accept this as a target. This is why I tried JULIA_CPU_TARGET=cortex-a8. Fedora's LLVM is built using these settings:

  --with-cpu=cortex-a8 \
  --with-tune=cortex-a8 \
  --with-arch=armv7-a \
  --with-float=hard \
  --with-fpu=vfpv3-d16 \
  --with-abi=aapcs-vfp

http://pkgs.fedoraproject.org/cgit/llvm.git/tree/llvm.spec?h=f22&id=5aea06bdf020fd2fc750286d397e51e01a94a765#n404

I see that ARM.inc advises --with-cpu=cortex-a9 --with-fpu=neon instead. Do you think that can be an issue?

BTW, note JCFLAGS includes -fsigned-char.

[vtjnash]

you must match the --with-float and --with-abi flags exactly between the compilers (system libraries, gcc, llvm, and julia) or you will get random data corruption and segfaults. the rest (cpu, tune, arch) generally don't matter and simply describe the available instruction set.

[nalimilan]

@vtjnash That's the case AFAICT, as the values of these two flags passed to gcc when building Julia are exactly the same as those used to build LLVM (that's the point of a distribution, after all). But I'm not sure how code generation from Julia works: what flags are used in that case? Those used to build LLVM? Some other default? Can that be the problem?

Another surprising fact is that changing the value of JULIA_TARGET_CPU changes the error, which IIUC should not happen if as you say it doesn't matter. Or maybe it just changes the symptom of an underlying bug due to something else.

[nalimilan]

I get yet another error on Fedora rawhide (gcc 5 with _GLIBCXX_USE_CXX11_ABI=1):

osutils.jl
error during bootstrap:
LoadError(at "sysimg.jl" line 82: LoadError(at "osutils.jl" line 38: Base.ArgumentError(msg="set must be non-empty")))
Makefile:168: recipe for target '/builddir/build/BUILD/julia/build/usr/lib/julia/sys0.o' failed

https://kojipkgs.fedoraproject.org//work/tasks/7279/9357279/build.log
https://kojipkgs.fedoraproject.org//work/tasks/7338/9357338/build.log

@ViralBShah Any suggestions?

EDIT: JULIA_CPU_TARGET=cortex-a9 does not make any difference.

[ViralBShah]

I suspect it is a julia bug, and we should run it through valgrind or something to figure things out. It is quite common for the build to die in the system image phase in different places on different ARM architectures.

Cc: @ihnorton

[ViralBShah]

I wonder if Jameson's patches for ppc will help - especially the bit relaying to cpuid stuff.

[vtjnash]

i didn't patch any cpuid stuff, just deleted some dead code to make it easier to build.

fwiw, a little known fact is that the sys.ji file is actually fairly portable. if you have one from a similar build SHA1 on a similar machine (same OS and WORD_SIZE), you can just copy it over. For example, I'm currently using a sys.ji from my normal x86_64 ubuntu build to debug and test the PPC64le build.

[ViralBShah]

Yes, I did mean disabling it. The contents suggest hard coded values for armv7, which probably don't work out well on other arm processors. I will try using SYS.ji from elsewhere.

[vtjnash]

i don't think those values were actually read anywhere that mattered.

you might also try enabling MEMDEBUG (in options.h) and see if that helps

[nalimilan]

I ran the bootstrap inside Valgrind, with MEMDEBUG enabled. It's been running for almost a day now, and it's still not finished. So far Valgrind has uncovered a few things, but the traces are very poor. I've tried running julia-debug instead of julia, and still the same. What should I do to fix this? Is there any value in this kind of information?

 cd base && valgrind --smc-check=all-non-file --suppressions=../contrib/valgrind-julia.supp /builddir/build/BUILD/julia/build/usr/bin/julia-debug -C cortex-a8 --build /builddir/build/BUILD/julia/build/usr/lib/julia/sys0 sysimg.jl
==3822== Memcheck, a memory error detector
==3822== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==3822== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==3822== Command: /builddir/build/BUILD/julia/build/usr/bin/julia-debug -C cortex-a8 --build /builddir/build/BUILD/julia/build/usr/lib/julia/sys0 sysimg.jl
==3822== 
==3822== Conditional jump or move depends on uninitialised value(s)
==3822==    at 0x401B984: index (in /usr/lib/ld-2.21.90.so)
==3822== 
==3822== Conditional jump or move depends on uninitialised value(s)
==3822==    at 0x401B988: index (in /usr/lib/ld-2.21.90.so)
==3822== 
==3822== Conditional jump or move depends on uninitialised value(s)
==3822==    at 0x4008A5C: fillin_rpath (in /usr/lib/ld-2.21.90.so)
==3822==    by 0x4009283: _dl_init_paths (in /usr/lib/ld-2.21.90.so)
==3822== 
==3822== Conditional jump or move depends on uninitialised value(s)
==3822==    at 0x401BE40: strcpy (in /usr/lib/ld-2.21.90.so)
==3822== 
==3822== Use of uninitialised value of size 4
==3822==    at 0x401BD78: strcpy (in /usr/lib/ld-2.21.90.so)
==3822== 
==3822== Use of uninitialised value of size 4
==3822==    at 0x401BD84: strcpy (in /usr/lib/ld-2.21.90.so)
==3822== 
==3822== Invalid read of size 4
==3822==    at 0x401BD20: strcpy (in /usr/lib/ld-2.21.90.so)
==3822==  Address 0x7777ac0 is 16 bytes inside a block of size 19 alloc'd
==3822==    at 0x4848AAC: operator new(unsigned int) (in /usr/lib/valgrind/vgpreload_memcheck-arm-linux.so)
==3822== 
exports.jl
base.jl
reflection.jl
build_h.jl
version_git.jl
c.jl
options.jl
promotion.jl
tuple.jl
range.jl
expr.jl
error.jl
bool.jl
number.jl
int.jl
operators.jl
pointer.jl
refpointer.jl
rounding.jl
float.jl
==3822== Invalid read of size 4
==3822==    at 0x401BE6C: strcpy (in /usr/lib/ld-2.21.90.so)
==3822==  Address 0x7e45a80 is 24 bytes inside a block of size 25 alloc'd
==3822==    at 0x4848AAC: operator new(unsigned int) (in /usr/lib/valgrind/vgpreload_memcheck-arm-linux.so)
==3822== 

https://kojipkgs.fedoraproject.org//work/tasks/4521/9414521/build.log

[ViralBShah]

I am working on setting up a machine at scaleway.com. That should hopefully allow people to get in and fix things quickly.

[nalimilan]

Cool. Though the Fedora build system is not that bad, I can easily make some local changes and send them for build. (Not sure about the speed of the build machine compared with Scaleway's.)

[nalimilan]

Crap, the build timed out after 24h. I tried using gdb instead of Valgrind, but a bug prevents it from working on Fedora arm at the moment.

So if you have a Fedora/RHEL image on Scaleway, I'd be happy to try there.

[sbromberger]

I'll repeat the offer I made on julia-users: if someone wants a shell account on a Raspberry Pi 2 to test builds, I'm happy to provide one (or more). Just let me know. I have been unsuccessful in getting Julia to build (see #10235) and would welcome someone with some expertise here.

[ViralBShah]

My chromebook environment got wiped away due to an error on my part during reboot. It will be a while for me to set it up again. This may not be sooner than next week.