[Question] Cross Compiling for ARMv6 {glibc/musl}

[flotter]

package main
import "fmt"
func main() {
fmt.Println("hello world")
}

I attempted to cross compile this example application for (1) glibc sysroot and (2) musl libc sysroot.

I got a segfault in the first case, and loader issues in the second case. I did not realise tinygo was not looking at CC and picked up my Ubuntu installed GCC crosscompiler "arm-linux-gnueabihf-gcc" which was for ARMv7.

tinygo version 0.20.0 linux/amd64 (using go version go1.16.2 and LLVM version 11.0.0)

~/go-example$ export CC=/home/flotter/xxx/build/rpiv6/glibc/gcc/bin/armv6-rpi-linux-gnueabihf-gcc
OR
~/go-example$ export CC=/home/flotter/xxx/build/rpiv6/musl/gcc/bin/armv6-rpi-linux-musleabihf-gcc
~/go-example$ export CGO_ENABLED=1
~/go-example$ export GOOS=linux
~/go-example$ export GOARCH=arm
~/go-example$ export GOARM=6

~/go-example$ tinygo env
GOOS="linux"
GOARCH="arm"
GOROOT="/usr/lib/go-1.16"
GOPATH="/home/flotter/work"
GOCACHE="/home/flotter/.cache/tinygo"
CGO_ENABLED="1"
TINYGOROOT="/usr/local/lib/tinygo"

/go-example$ CGO_ENABLE=1 tinygo build -o test_cross test.go
error: failed to link /tmp/tinygo2943599410/main: exec: "arm-linux-gnueabihf-gcc": executable file not found in $PATH

(1) Question: It looks like tinygo does not look at CC. Is it possible to tell Tinygo which compiler to use?

(2) Question: It does not look like tinygo looks at GOARM. Can I use it to build for an ARM v6 (i.e. Raspberry Pi Zero/1) ?

[aykevl]

This behavior has changed in the dev branch (which will be the coming release). Previously, we would use the cross compiler installed on the host. In the dev branch (and in the coming release) we won't do that. Instead, TinyGo will bundle musl. This means that cross compiling just works, to any supported Linux architecture (386/amd64/arm/arm64) from any host. Including CGo support.

~/go-example$ export CGO_ENABLED=1

Note that TinyGo does not support disabling CGo, it is always enabled. This is possible because it bundles a C compiler.

(1) Question: It looks like tinygo does not look at CC. Is it possible to tell Tinygo which compiler to use?

No. With the current release, we make a hardcoded guess. With the next release, it will not be necessary anymore because we don't use the system compiler/linker.

(2) Question: It does not look like tinygo looks at GOARM. Can I use it to build for an ARM v6 (i.e. Raspberry Pi Zero/1) ?

Hmm, right now we don't. The dev branch currently hardcodes armv7. I think we should change this to also support GOARM.
Thanks for pointing this out!

[aykevl]

I've made a PR to add support for the GOARM environment variable: #2260. The PR defaults to ARMv5 right now. Perhaps a later version (such as ARMv6) would be a better default, I'm interested in hearing your opinion on this.

[flotter]

Thank you for your help. Much appreciated.

Just to double-check my understanding: you are saying that I would soon be able to cross-compile (in other words, tinygo build for another architecture such as ARMv7 from my AMD64 desktop), without providing TinyGo with any external cross compiler, and that the internal version will be use musl libc?

Perhaps a later version (such as ARMv6) would be a better default, I'm interested in hearing your opinion on this.

I see the default for Go appears to be ARMv5 (GOARM defaults to armv5, but not sure what will happen if the supplied cross-compiler does not support ARMv5!). However, it's explicitly said to always supply GOARM when you set GOARCH=arm (https://zchee.github.io/golang-wiki/GoArm/).

I do not have a too strong opinion here. I do agree making the default ARMv5 (no floating point) does not feel like defaulting to the most common case. My personal focus is a wider spectrum of devices, and I would want to make sure I set it explicitly in order to not yield a build which under performs or breaks.

Two questions:

(1) Is the TinyGo build always statically linked against musl? I would guess not.

(2) If I cross compile TinyGo for a target device, and the target device and it's sysroot is produced by a separate cross compiler and musl libc build, would not sharing the same cross-compiler and musl libc version not be unsafe (or perhaps the target system has uClibc or something else)? My instinct is I would want to use one cross-compiler and libc/sysroot to build all my components against, to make sure everything plays together nicely.

[aykevl]

Just to double-check my understanding: you are saying that I would soon be able to cross-compile (in other words, tinygo build for another architecture such as ARMv7 from my AMD64 desktop), without providing TinyGo with any external cross compiler, and that the internal version will be use musl libc?

When targeting Linux (GOOS=linux), yes.

(1) Is the TinyGo build always statically linked against musl? I would guess not.

You mean the tinygo binary itself?
The compiler itself is at the moment built on an older Debian version and thus depends on glibc. I intend to change to a fully static build but haven't done that yet.

(2) If I cross compile TinyGo for a target device, and the target device and it's sysroot is produced by a separate cross compiler and musl libc build, would not sharing the same cross-compiler and musl libc version not be unsafe (or perhaps the target system has uClibc or something else)? My instinct is I would want to use one cross-compiler and libc/sysroot to build all my components against, to make sure everything plays together nicely.

Sorry, I don't follow.
You are saying you want to cross compile TinyGo (the compiler binary, tinygo) for an ARM target device?

[flotter]

Sorry my explanation was not clear. My use case is always that I just use TinyGo on my desktop machine, but target a variety of platforms when building Go applications.

(2) "If I cross compile TinyGo" => "If I cross compile with TinyGo ..."

In other words, if I do an "ldd " today I will see that the TinyGo application depends on libraries. Today I believe this is glibc as the CGO side will use the hardcoded "system supplied" compiler (in my case for ARM). This is the case I first described above.

From your explanation I understood that it will not be possible to supply TinyGo with a cross compiler in the future (i.e. as in CC today with Go), and that magically TinyGo will have an internal compiler which will use musl for CGO.

If my understanding is correct, then, without supplying a compiler to TinyGo (for cross compiling), the CGO part will be built using internal compiler/musl assuming a version of that library (which assumes a kernel syscall set). It feels like I am missing something, because the assumptions made here may not work with the target platrform I am cross compiling this TinyGo application for, which could have an older version of the Linux kernel, and older version of musl libc.

I am relatively new to Go, so if I am getting something fundamentally wrong in the TinyGo ecosystem, I apologise for wasting your time.

Here is an example development flow for a new minimal Linux userspace on Raspberry Pi Zero:

(1) Build a cross compiler for ARMv6 with musl targetting kernel v5.10 => armv6-rpi-linux-musleabihf
(2) Create a root filesystem for this device and copy the musl library (and dynamic loader)
(3) Build busybox using this compiler

(4) Now I want to use TinyGo to build a Go application targeting this platform.

If the output of TinyGo build produces a dynamic application, it will need to know what kind of system it will run on (glibc, musl, uClibc and the correct version). If it produces a static application, then it would be less of an issue, as library code is embedded and ultimately as long as the syscall into the kernel is compatible, all is OK.

[aykevl]

From your explanation I understood that it will not be possible to supply TinyGo with a cross compiler in the future (i.e. as in CC today with Go), and that magically TinyGo will have an internal compiler which will use musl for CGO.

Correct.

If my understanding is correct, then, without supplying a compiler to TinyGo (for cross compiling), the CGO part will be built using internal compiler/musl assuming a version of that library (which assumes a kernel syscall set).

Correct.

It feels like I am missing something, because the assumptions made here may not work with the target platrform I am cross compiling this TinyGo application for, which could have an older version of the Linux kernel,

As long as the Linux kernel is at least 2.6.39 (released over 10 years ago), everything should be fine. For more information: https://wiki.musl-libc.org/supported-platforms.html

and older version of musl libc.

This doesn't matter because musl is statically linked.
This does assume you're not linking against any system libraries (using cgo). If you're compiling pure Go code or only very simple CGo that doesn't specifically link against any shared library (#cgo LDFLAGS: -lsomelibrary) then you should be fine.

[flotter]

Sounds promising. Thank you for all your time in helping me. I will do some testing and report back.

[aykevl]

Looks like the question is answered, so closing.