[darwin aarch64 cgo] regression in 0.8.0-1366-gfc302f00a

[motiejus]

Zig Version

0.8.0-1921-g83a668195

Steps to Reproduce

Identical to #10297, but substitute:

• x86_64 with aarch64.
• amd64 with arm64.

Step-by-step:

hello.go

package main

// #include <stdio.h>
// void helloworld() { printf("hello, world\n"); }
import "C"

func main() {
	C.helloworld()
}

~/test

#!/bin/bash
set -xe

pushd $HOME/dev/zig/build
cmake ..
make -j$(nproc)
popd
GOOS=darwin GOARCH=arm64 CC=$HOME/zcc CGO_ENABLED=1 go build -a -buildmode=pie -ldflags "-s -w" $HOME/hello.go

~/zcc

user@motiejus:~/dev/zig$ cat ~/zcc 
#!/bin/bash
zig=$HOME/dev/zig/build/zig
exec "$zig" cc -target aarch64-macos-gnu "$@"

Bisect Result

fc302f0 is the first bad commit, which doesn't revert cleanly.

$ git bisect good dde0adcb363f3a3f306c0fc9eaec511cc3b74965
$ git bisect bad master
$ git bisect run ~/test
<...>
fc302f00a9de5de0490f4a66720e75946763c695 is the first bad commit
commit fc302f00a9de5de0490f4a66720e75946763c695
Author: Jakub Konka <kubkon@jakubkonka.com>
Date:   Sun Oct 10 10:33:15 2021 +0200

    macho: redo relocation handling and lazy bind globals

    * apply late symbol resolution for globals - instead of resolving
      the exact location of a symbol in locals, globals or undefs,
      we postpone the exact resolution until we have a full picture
      for relocation resolution.
    * fixup stubs to defined symbols - this is currently a hack rather
      than a final solution. I'll need to work out the details to make
      it more approachable. Currently, we preemptively create a stub
      for a lazy bound global and fix up stub offsets in stub helper
      routine if the global turns out to be undefined only. This is quite
      wasteful in terms of space as we create stub, stub helper and lazy ptr
      atoms but don't use them for defined globals.
    * change log scope to .link for macho.
    * remove redundant code paths from Object and Atom.
    * drastically simplify the contents of Relocation struct (i.e., it is
      now a simple superset of macho.relocation_info), clean up relocation
      parsing and resolution logic.

 src/codegen.zig            |   45 +-
 src/link/MachO.zig         |  478 +++++++-------
 src/link/MachO/Archive.zig |    2 +-
 src/link/MachO/Atom.zig    | 1538 +++++++++++++++++---------------------------
 src/link/MachO/Dylib.zig   |    2 +-
 src/link/MachO/Object.zig  |   11 +-
 6 files changed, 857 insertions(+), 1219 deletions(-)
bisect run success

Expected Behavior

~/test should produce a working darwin aarch64 executable.

Actual Behavior

Running ~/test on master fails to produce an executable:

$ GOOS=darwin GOARCH=arm64 CC=$HOME/zcc CGO_ENABLED=1 go build -a -buildmode=pie -ldflags "-s -w" hello.go
# runtime/cgo
warning(link): framework not found for '-framework CoreFoundation'
warning(link): Framework search paths:
# runtime/cgo
/tmp/go-build2151189485/b003/_cgo_import.go:2:3: usage: //go:cgo_import_dynamic local [remote ["library"]]
/tmp/go-build2151189485/b003/_cgo_import.go:4:3: usage: //go:cgo_import_dynamic local [remote ["library"]]
/tmp/go-build2151189485/b003/_cgo_import.go:6:3: usage: //go:cgo_import_dynamic local [remote ["library"]]

cc @kubkon

[kubkon]

@motiejus thanks! Tracking this down took a little longer than expected, but got the fix in #10301.

[kubkon]

Oh and FYI, I'm not sure if you noticed but that linker is unable to find CoreFoundation framework. This is an intended change in the way zig automatically detects an SDK if available. Currently, when explicitly specifying the compilation target, here aarch64-macos-gnu, means zig enters cross-compilation mode and it is the user's responsibility to specify the path to the sysroot via zig cc --sysroot <path>. However, if you leave out -target aarch64-macos-gnu and build natively on an arm64 macOS, zig will autodetect the SDK (if available).

[motiejus]

Oh and FYI, I'm not sure if you noticed but that linker is unable to find CoreFoundation framework. This is an intended change in the way zig automatically detects an SDK if available. Currently, when explicitly specifying the compilation target, here aarch64-macos-gnu, means zig enters cross-compilation mode and it is the user's responsibility to specify the path to the sysroot via zig cc --sysroot <path>. However, if you leave out -target aarch64-macos-gnu and build natively on an arm64 macOS, zig will autodetect the SDK (if available).

I did notice the warning, but it cross-compiled the executable successfully (i.e. produced a binary that file thinks is an arm64 macho executable, weighing about 1MB) without the sysroot. I don't have an M1 to test this with (this should come in 2022), so I assumed it "would work".

A few follow-up questions:

• What is the status of the executable if it's not compiled with a --sysroot? Would it even run?
• How do we acquire a sysroot? Is there "a tarball somewhere at apple.com", or should I copy from an existing host? Any pointers to legal implications, if you know?
• [curiosity] what is the sysroot used for during the compilation phase?

[kubkon]

Oh and FYI, I'm not sure if you noticed but that linker is unable to find CoreFoundation framework. This is an intended change in the way zig automatically detects an SDK if available. Currently, when explicitly specifying the compilation target, here aarch64-macos-gnu, means zig enters cross-compilation mode and it is the user's responsibility to specify the path to the sysroot via zig cc --sysroot <path>. However, if you leave out -target aarch64-macos-gnu and build natively on an arm64 macOS, zig will autodetect the SDK (if available).

I did notice the warning, but it cross-compiled the executable successfully (i.e. produced a binary that file thinks is an arm64 macho executable, weighing about 1MB) without the sysroot. I don't have an M1 to test this with (this should come in 2022), so I assumed it "would work".

A few follow-up questions:

* What is the status of the executable if it's not compiled with a `--sysroot`? Would it even run?

This depends. By default, on macOS the linker is supposed to always link every shared object from the linker line. While it may not be strictly necessary since for instance you don't resolve any symbols in the said shared object, some runtime mechanism might depend on it, for instance you could use ObjC runtime to send a message as a string to poll the runtime address of the loaded Foundation framework. In this case, if you didn't link any symbols from Foundation and your linker stripped this dylib from the binary (treating it as unused), ObjC will return address 0x0 of Foundation since it wasn't loaded at all.

This situation reminds of this in that while this particular Hello World example didn't require the CoreFoundation framework, some inner Go mechanism might which will manifest is a really nasty and difficult to track runtime exception. The question now is: should zig warn the user or perhaps throw an error instead? Initially I assumed the former is enough, but if zig is used as a drop-in C/C++ compiler replacement and is used inside another build tool (like go build) then perhaps an error would be a better choice?

More on this topic: #10192

* How do we acquire a sysroot? Is there "a tarball somewhere at apple.com", or should I copy from an existing host? Any pointers to legal implications, if you know?

That's an interesting one, and I don't really have an answer to that. Some folks have packaged a sysroot for a version of macOS already, for instance https://github.com/hexops/sdk-macos-11.3. Legal implications - I have no clue, however, I'd recommend packaging only tbd files which are text-based descriptors for actual shared libraries (in fact, there might not even be any other option these days).

* [curiosity] what is the sysroot used for during the compilation phase?

We use the sysroot in a few places mainly when building natively (we then pull the libc headers from there together with framework stubs), but it's also required when linking in frameworks on macOS (or targeting macOS). When linking in dylibs we do a recursive linking of any dylib that the original dylib re-exported, and to resolve that we often require a sysroot to prepend to the path. Otherwise the resolution might fail.

[motiejus]

• What is the status of the executable if it's not compiled with a --sysroot? Would it even run?

This depends. By default, on macOS the linker is supposed to always link every shared object from the linker line. While it may not be strictly necessary since for instance you don't resolve any symbols in the said shared object, some runtime mechanism might depend on it, for instance you could use ObjC runtime to send a message as a string to poll the runtime address of the loaded Foundation framework. In this case, if you didn't link any symbols from Foundation and your linker stripped this dylib from the binary (treating it as unused), ObjC will return address 0x0 of Foundation since it wasn't loaded at all.

This situation reminds of this in that while this particular Hello World example didn't require the CoreFoundation framework, some inner Go mechanism might which will manifest is a really nasty and difficult to track runtime exception.

Let me try to re-phrase to see if I understood this correctly: Foundation needs to be linked in a Mach-O binary; but, if SDK is not specified, zld will set it's address to 0x0, which is problematic as described in #9542. And there is no way to detect whether that Foundation will ever be resolved (and thus cause a mysterious crash).

The question now is: should zig warn the user or perhaps throw an error instead? Initially I assumed the former is enough, but if zig is used as a drop-in C/C++ compiler replacement and is used inside another build tool (like go build) then perhaps an error would be a better choice?

If my re-phrasing above is correct, then I definitely agree we should be more vocal:

1. Always require sysroot when compiling for darwin (or darwin/aarch64? see below). If sysroot is not specified, bail.
2. Add an extra --darwin-no-foundation, which keeps the current behavior: i.e. link Foundation to 0x0. The dangerous/possibly incomplete binary should be opt-in.

In our case, zig cc is even deeper: bazel -> rules_go -> go build -> $CXX -> bazel-zig-cc -> zig cc. The user normally interacts with the first two only; the rest are very much behind the scenes. Failing early is much more preferred if we are doing something wrong.

* How do we acquire a sysroot? Is there "a tarball somewhere at apple.com", or should I copy from an existing host? Any pointers to legal implications, if you know?

That's an interesting one, and I don't really have an answer to that. Some folks have packaged a sysroot for a version of macOS already, for instance https://github.com/hexops/sdk-macos-11.3. Legal implications - I have no clue, however, I'd recommend packaging only tbd files which are text-based descriptors for actual shared libraries (in fact, there might not even be any other option these days).

To be clear: is this required for x86_64 too, not only aarch64? I am clarifying, because compiling x86_64 does not produce warning(link): framework not found for '-framework CoreFoundation', only aarch64 does.

Only the *.tlbs weigh ~1.3MB, which is good. I will figure out the legal/licensing part.

* [curiosity] what is the sysroot used for during the compilation phase?

We use the sysroot in a few places mainly when building natively (we then pull the libc headers from there together with framework stubs)

For the record, our use case (bazel-zig-cc) is always "cross-compilation" mode, that's how we get hermetic builds. So our builds don't have to rely on anything in the system (and, by extension, produced binaries are the same, regardless where they are compiled).

[uhthomas]

Hey @motiejus it looks like I'm seeing some similar messages for macOS x86-64 -> macOS x86-64. I would imagine it's not just an aarch64 thing in this case?

#10158 (comment)