Illegal instruction (core dumped) on armv6l

[skj-dev]

When trying to run nomad (v0.5.6) on an armv6l Raspberry Pi (Gen 1 and PiZero), the executable crashes. It does, however run on armv7l Raspberry Pi (Gen 2 & Gen 3). I'm running Arch Linux on the Pis in question, and using the Linux Arm nomad binary downloaded directly from nomadproject.io.

The following is the last bit of output from strace:

set_tls(0x76fe71e0, 0x76fe78d8, 0x76fe9000, 0x76fe71e0, 0x76fe9000) = 0
mprotect(0x76f79000, 8192, PROT_READ)   = 0
mprotect(0x76fa6000, 4096, PROT_READ)   = 0
mprotect(0x10f0000, 4096, PROT_READ)    = 0
mprotect(0x76fe8000, 4096, PROT_READ)   = 0
munmap(0x76faa000, 59413)               = 0
set_tid_address(0x76fe6d88)             = 290
set_robust_list(0x76fe6d90, 12)         = 0
rt_sigaction(SIGRTMIN, {sa_handler=0x76f83300, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x76e61460}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {sa_handler=0x76f8341c, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0x76e61460}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
ugetrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
--- SIGILL {si_signo=SIGILL, si_code=ILL_ILLOPC, si_addr=0x5db82} ---
+++ killed by SIGILL (core dumped) +++
Illegal instruction (core dumped)

[schmichael]

Our 32-bit ARM build is currently for armv6h or armhf or however you want to say "hard float" :)

We don't have any armv6l boxes handy and the various ARM flavors have proven exceedingly difficult targets to hit -- especially with CGO enabled. We'd like to produce builds without CGO which would make supporting various ARM flavors far far easier. I'm not sure we'll have any resources to invest in fixing this until we removed the CGO dependency.

[skj-dev]

I inadvertently muddled things by running from the uname -m output. The systems in question are hard float, but the difference is in the FPU. The Pi Gen 1 and PiZero systems have VFPv2, while the Pi Gen2 and Gen3 have VFPv3-D16. The nomad binary shows VFPv3-D16, which explains why it doesn't run on the older Pis.

I definitely realize that supporting the smaller Raspberry Pis might not be worth the effort, and just wanted to clarify. :)

[sean@rpi ~]$ readelf -A /usr/bin/ls
Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "6"
  Tag_CPU_arch: v6
  Tag_ARM_ISA_use: Yes
  Tag_THUMB_ISA_use: Thumb-1
  Tag_FP_arch: VFPv2
  Tag_ABI_PCS_wchar_t: 4
  Tag_ABI_FP_rounding: Needed
  Tag_ABI_FP_denormal: Needed
  Tag_ABI_FP_exceptions: Needed
  Tag_ABI_FP_number_model: IEEE 754
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_align_preserved: 8-byte, except leaf SP
  Tag_ABI_enum_size: int
  Tag_ABI_VFP_args: VFP registers
  Tag_CPU_unaligned_access: v6


[sean@rpi ~]$ readelf -A ./nomad
Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "7-A"
  Tag_CPU_arch: v7
  Tag_CPU_arch_profile: Application
  Tag_ARM_ISA_use: Yes
  Tag_THUMB_ISA_use: Thumb-2
  Tag_FP_arch: VFPv3-D16
  Tag_ABI_PCS_wchar_t: 4
  Tag_ABI_FP_rounding: Needed
  Tag_ABI_FP_denormal: Needed
  Tag_ABI_FP_exceptions: Needed
  Tag_ABI_FP_number_model: IEEE 754
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_enum_size: int
  Tag_ABI_VFP_args: VFP registers
  Tag_CPU_unaligned_access: v6

[muellerkyle]

I ran into this same issue attempting to run nomad on a Raspberry Pi zero and I was able to get nomad to compile to a state where it runs on the ARMv6l. The caveat is that I haven't fully tested nomad on the zero yet. But, I did want to document how I was able to get it to run.

I have an ARMv7 board (I think it's a Raspberry Pi 2) which has more memory than the Zero too.

I cloned this repo and checking out the latest tag on the ARMv7 box. I then modified the GNUmakefile file and added the GOARM=6 flag for the Go compiler to this section:

pkg/linux_arm/nomad: $(SOURCE_FILES) ## Build Nomad for linux/arm
        @echo "==> Building $@ with tags $(GO_TAGS)..."
        @CGO_ENABLED=1 GOOS=linux GOARCH=arm CC=arm-linux-gnueabihf-gcc-5 \
                go build \
                -ldflags $(GO_LDFLAGS) \
                -tags "$(GO_TAGS)" \
                -o "$@"

I then finished with a make bootstrap and a make pkg/linux_arm/nomad and used scp to get the binary from the pkg/linux_arm directory to my Pi Zero. And I was able to run it without the Illegal instruction that I get from the pre-built binary.

I'll note if I run into any further issues here on this ticket, if anyone wants. But, I did get it to build and thought I would note how I did it. For anyone else following this path, you need GO 1.9 and at least gcc 5 on your build Pi.

Thanks!

Update: I was able to get redis running on the Pi Zero using the example job. It spun up without issue.

[schmichael]

@muellerkyle Thanks for the fix! Would you, @ttyS0, and anyone else interested in the 32bit ARM build mind testing the binary I attached to #3823?

[muellerkyle]

I'll pull a copy of your build and try to get it tested tonight.
Thank you!

Should there be a seperate GNUMake profile for the ARMv6 vs v7?

[preetapan]

@muellerkyle #3823 (comment)

[skj-dev]

It still crashes for me with Illegal instruction on a fresh install of Raspbian on a Pi Zero W.

The binary is showing up as a arch v7 for me.

root@raspberrypi:~# wget -q https://github.com/hashicorp/nomad/files/1683761/nomad.gz
root@raspberrypi:~# gunzip nomad.gz
root@raspberrypi:~# chmod a+x nomad
root@raspberrypi:~# ls -l nomad
-rwxr-xr-x 1 root root 35782112 Feb  1 00:17 nomad
root@raspberrypi:~# sha256sum nomad
8e9d527c0d501e41ded3b134f833ba2bb0f62220f8579ad3a410de76329116b7  nomad
root@raspberrypi:~# ./nomad
Illegal instruction
root@raspberrypi:~# uname -a
Linux raspberrypi 4.9.59+ #1047 Sun Oct 29 11:47:10 GMT 2017 armv6l GNU/Linux
root@raspberrypi:~# readelf -A ./nomad
Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "7-A"
  Tag_CPU_arch: v7
  Tag_CPU_arch_profile: Application
  Tag_ARM_ISA_use: Yes
  Tag_THUMB_ISA_use: Thumb-2
  Tag_FP_arch: VFPv3-D16
  Tag_ABI_PCS_wchar_t: 4
  Tag_ABI_FP_rounding: Needed
  Tag_ABI_FP_denormal: Needed
  Tag_ABI_FP_exceptions: Needed
  Tag_ABI_FP_number_model: IEEE 754
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_enum_size: int
  Tag_ABI_VFP_args: VFP registers
  Tag_CPU_unaligned_access: v6
root@raspberrypi:~# readelf -A /bin/ls
Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "6"
  Tag_CPU_arch: v6
  Tag_ARM_ISA_use: Yes
  Tag_THUMB_ISA_use: Thumb-1
  Tag_FP_arch: VFPv2
  Tag_ABI_PCS_wchar_t: 4
  Tag_ABI_FP_rounding: Needed
  Tag_ABI_FP_denormal: Needed
  Tag_ABI_FP_exceptions: Needed
  Tag_ABI_FP_number_model: IEEE 754
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_align_preserved: 8-byte, except leaf SP
  Tag_ABI_enum_size: int
  Tag_ABI_VFP_args: VFP registers
  Tag_CPU_unaligned_access: v6
root@raspberrypi:~# 

:(

[muellerkyle]

I had the same result (Illegal Instruction). I was unable to get that binary to work.
I noticed that the linked ticket did have another user, @nicholasjackson , confirming a working executable built on Pi 3

Ah, nevermind. I see that he did not get it to run.

I can post my binary for testing. I believe that I compiled in on a Raspberry Pi 2.

I've uploaded my binary here: https://github.com/muellerkyle/nomadpizero/releases/download/v0.7.1/nomad.tar.gz

[schmichael]

Yeah, so far I don't think anyone has figured out how to cross-compile for ARMv6 using our root Vagrant box which cuts our releases.

Might have to leave ARMv6 up to community support for the time being... If nothing else I'd be happy to add a note to our downloads page about minimum ARM requirements if someone can concisely describe them.

[nicholasjackson]

@schmichael for the time being I am happy to build on my Pi 3, the releases for 0.7.0 and 0.7.1 are currently there, I will build a 0.8.0 dev release today.

https://github.com/nicholasjackson/nomad-arm6l/releases