cmd/go: -buildmode=pie does not build correct PIE binaries for ppc64le

[laboger]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

latest

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

ppc64le

What did you do?

Tried to run binaries built as PIE.

What did you expect to see?

Correct execution

What did you see instead?

Error messages like this:
./main.exe: error while loading shared libraries: R_PPC64_ADDR16_HA re10f2f8fa4 for symbol `' out of range
2018/09/05 09:53:14 Failed: exit status 127
on newer distros.

The problem is related to issues #27510 and #21954 and can still be reproduced on many distros by passing -pie as a linker option.

The solution to the above issues was to use -buildmode=pie but that doesn't correct all problems. When linking a program with -pie on ppc64le then all code should be built as PIC. When some code is not PIC there could be relocations created like R_PPC64_ADDR16_HA which will fail with the above error if the relocated addresses are too big. On some distros/kernels we get lucky and no failures occur if the addresses stay small enough but on newer distros the runtime relocation errors can occur.

When using -buildmode=pie the -shared option is set to force the compiles to be built as PIC but it still links in stdlib which was not built as PIC. Likewise any library linked in should be PIC. When all the code is has been built as PIC then this relocation would not be generated and the PIE should work.

One further question is whether all code should be PIC if the gcc being used for the link passes -pie by default and stop setting -no-pie.

[ianlancetaylor]

When you use -buildmode=pie the go tool is supposed to rebuild all dependencies using -shared, including the standard library. If that is not happening, something is wrong.

As far as a gcc defaulting to PIE, we already have support when using -compiler=gccgo for passing -fno-pie; perhaps we need to extend that support to the gc compiler side. See gccNoPie in cmd/go/internal/work/.

[laboger]

I'm sorry there's some confusion here. This was reported to me through a bugzilla that even with -buildmode=pie a bad relocation is generated but I was mistaken above about whether stdlib is built with PIC in this case. In my experiments it seems to be. The reported failure is on RHEL8 which I don't have access to so I'm not sure yet where it is coming from.

One related question is whether golang should recognize whether the gcc on the system sets -pie by default when invoking the linker and if so just build it as -buildmode=pie. I'm not sure if that's the right answer but just what was suggested.

[ianlancetaylor]

I don't think we should change the default of go build to the default of GCC. That seems like an unnecessary tying that will tend to increase confusion.

Perhaps we should have an easy way for a distro to change the default of go build, but that seems like a different issue. (For example, we could consider adding another environment variable recognized by src/make.bash.)

[laboger]

@ianlancetaylor Is there a way to build a toolchain for ppc64le so that the default type of executable is always PIE through the use of existing environment variables or other options? Or is it more complicated than that?

I tried to just change the default in cmd/go/internal/work/init.go as a test to see what would happen and got this:

Building Go toolchain2 using go_bootstrap and Go toolchain1.
# cmd/cgo
loadinternal: cannot find runtime/cgo
# cmd/asm
loadinternal: cannot find runtime/cgo
# cmd/link
loadinternal: cannot find runtime/cgo
# cmd/compile
loadinternal: cannot find runtime/cgo
Building Go toolchain3 using go_bootstrap and Go toolchain2.
# cmd/cgo
loadinternal: cannot find runtime/cgo
# cmd/asm
loadinternal: cannot find runtime/cgo
# cmd/link
loadinternal: cannot find runtime/cgo
# cmd/compile
loadinternal: cannot find runtime/cgo
HASH[build runtime/internal/sys]
HASH[build runtime/internal/sys]: "devel +745ec8b Fri Nov 2 17:56:07 2018 +0000"
HASH[build runtime/internal/sys]: "compile\n"
HASH[build runtime/internal/sys]: "goos linux goarch ppc64le\n"
HASH[build runtime/internal/sys]: "import \"runtime/internal/sys\"\n"
HASH[build runtime/internal/sys]: "omitdebug false standard true local false prefix \"\"\n"
HASH[build runtime/internal/sys]: "compile ymJr-uldWHMJ1NKhKQaW X:framepointer [\"-shared\"] []\n"
HASH[build runtime/internal/sys]: "GO$GOARCH=\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/arch.go: d9b0b7e72538d421b2607acaba60ca49f20ef584b3d1d191c6729e35fbb8101d
HASH[build runtime/internal/sys]: "file arch.go 2bC35yU41CGyYHrKumDK\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/arch_ppc64le.go: d43466ea44c56eba4514ad41e6652168a0ce0dcb720578ea6dadea343266a24a
HASH[build runtime/internal/sys]: "file arch_ppc64le.go 1DRm6kTFbrpFFK1B5mUh\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/intrinsics.go: 8b469a461e1d983706e0b3635715ce70691adc5db7c4e067b88cc59f40cd66f4
HASH[build runtime/internal/sys]: "file intrinsics.go i0aaRh4dmDcG4LNjVxXO\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/stubs.go: 376cb070d0d0d7094a8a5ed372853e00c377c2aba3d8c118dee821dd526a6022
HASH[build runtime/internal/sys]: "file stubs.go N2ywcNDQ1wlKil7TcoU-\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/sys.go: 55e021891200a7e6a5c371c8a1ab71b6c15aeb16ea6c1b192185d17df8c8b18f
HASH[build runtime/internal/sys]: "file sys.go VeAhiRIAp-alw3HIoatx\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/zgoarch_ppc64le.go: 157322cbcc4e3c5d80effef8a81d3d9a480cc2558550fad0df46a10b48af334c
HASH[build runtime/internal/sys]: "file zgoarch_ppc64le.go FXMiy8xOPF2A7_74qB09\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/zgoos_linux.go: 32284a5a70366b56fe207ea3a16ad5d61dd208d425ad28b81cc2fe59db9180e8
HASH[build runtime/internal/sys]: "file zgoos_linux.go MihKWnA2a1b-IH6joWrV\n"
HASH /home/boger/golang/clean/go/src/runtime/internal/sys/zversion.go: a0901e5adb74da30a6a6643ab36cfaa28c4c5136142e9a1fdeec8ca6c30f000a
HASH[build runtime/internal/sys]: "file zversion.go oJAeWtt02jCmpmQ6s2z6\n"
HASH[build runtime/internal/sys]: 664409261b5745f2e0926a0ef91599e67e2adb4a49da5a0b03eef7256f034c44
runtime/internal/sys true
go tool dist: unexpected stale targets reported by /home/boger/golang/clean/go/pkg/tool/linux_ppc64le/go_bootstrap list -gcflags="" -ldflags="" for [cmd/asm cmd/cgo cmd/compile cmd/link runtime/internal/sys]:
	STALE cmd/asm: stale dependency: internal/cpu
	STALE cmd/cgo: stale dependency: internal/cpu
	STALE cmd/compile: stale dependency: internal/cpu
	STALE cmd/link: stale dependency: internal/cpu
	STALE runtime/internal/sys: build ID mismatch

[ianlancetaylor]

There is no simple way today to force the gc toolchain to default to PIE.

I'm not sure why changing buildModeInit didn't work. It looks like it got confused with the cache. Maybe some extra step is required to add the different build mode to the cache key somehow.

[jfkw]

This issue is also present on SUSE and openSUSE for ppc64le.

With go1.10.5, the build stops at the following error:

cmd/vendor/github.com/google/pprof/internal/symbolizer
cmd/vendor/github.com/google/pprof/internal/driver
cmd/vendor/github.com/google/pprof/driver
cmd/pprof
/home/abuild/rpmbuild/BUILD/go/bin/go: error while loading shared libraries: R_PPC64_ADDR16_HA re10143fb0c for symbol `' out of range

By ensuring gcc does not use PIE by default, the build can complete, although this is not ideal.

It would be a great improvement to have an environment variable and option to force the toolchain to default to PIE.

[ianlancetaylor]

@jfkw Now in 1.12 you should be able to use GOFLAGS=-buildmode=pie.

[daixiang0]

@ianlancetaylor seems not work for 1.12:

# go version
go version go1.12.10 linux/mips64le
# GOFLAGS=-buildmode=pie go build components/cli
-buildmode=pie not supported on linux/mips64le

[ianlancetaylor]

@daixiang0 This bug report is about a problem with ppc64le. You are using mips64le.

[daixiang0]

@ianlancetaylor sorry for mistake, BTW any plan for mips to support pie build mode?

[ianlancetaylor]

@daixiang0 Sorry, I do not know of anyone working on adding support for PIE for MIPS.

[laboger]

I believe the title is is misleading. -buildmode=pie works fine for most things on ppc64le. The problem happens when trying to build a toolchain that will generate all binaries as pie by default, including those that are part of the toolchain. This came about because of a request from a single user.

No one has asked about this again, so if we want to close it that is fine with me.

I retried the same experiment as I did back in November and got the same result.

[hramrach]

So now you have to add 'me too' to bugs? It his some web 2.0 thing?

Anyway, the code is just broken.

How do you suppose the code is position independent if the libraries included in the interpreter are not?

[laboger]

If you have an example of a failure when using -buildmode=pie on ppc64le please provide it. I'm not sure what libraries you are referring to, my understanding is that if it is Go code built using -buildmode=pie it will be built as position independent, and C or C++ code built for ppc64le should be position independent by default.

…

On Thu, Oct 17, 2019 at 11:07 AM Michal Suchánek ***@***.***> wrote: So now you have to add 'me too' to bugs? It his some web 2.0 thing? Anyway, the code is just broken. How do you suppose the code is position independent if the libraries included in the interpreter are not? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#28531?email_source=notifications&email_token=ACH7BDCEUEUQKTT56QRZISTQPCEV5A5CNFSM4GA2TLN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBQULNY#issuecomment-543245751>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACH7BDGW7BMX7JJBJW54FBDQPCEV5ANCNFSM4GA2TLNQ> .

[hramrach]

C or C++ code built for ppc64le should be position independent by default.

Apparently not. Some distributions choose to set this default which breaks go completely.

[laboger]

I'm very sorry. I didn't notice that an earlier comment was made regarding this topic related to SUSE.
@ianlancetaylor I tried a build setting GOFLAGS=-buildmode=pie, and the build finished but that didn't seem to build all the toolchain binaries as PIE.

If I change the default in init.go, I still get the errors that some files are stale.

I cannot reproduce the problem identified earlier on SUSE using go 1.10.5. Were there any env variables used during the build? What distro was it.

[hramrach]

In openSUSE there is a package gcc-PIE which sets the gcc defaults options to PIC and PIE (since Leap 15.0 which is the oldest version still developed). They can still be overridden on the commandline or the package may opt to not install these defaults.

Not using the PIE options was what was done in openSUSE to build go and go packages on ppc64le.
I can look up the current state if there is some interest in debugging this further.

[ianlancetaylor]

@laboger I don't know why setting GOFLAGS didn't work when building the toolchain. Does it work when using the toolchain? That is, does GOFLAGS=-buildmode=pie go build hello.go build a PIE?

[ianlancetaylor]

@hramrach Please be polite when making comments, and bear in mind the Gopher Code of Conduct (https://golang.org/conduct). In particular, be patient and be charitable. Thanks.

[hramrach]

What the R_PPC64_ADDR16_HA suggests is you generate direct references which are filled in as immediates in the code which are only 16bit and you need to generate indirect jumps or loads to be able to link non-trivial ppc binaries referencing code or data more than +-32kb away.

[jfkw]

@laboger I don't know why setting GOFLAGS didn't work when building the toolchain. Does it work when using the toolchain? That is, does GOFLAGS=-buildmode=pie go build hello.go build a PIE?

@ianlancetaylor Currently, the openSUSE ppc64le go package builds by not installing the gcc-PIE package. I will do some testing in openSUSE ppc64le using the GOFLAGS=-buildmode=pie used uniformly for building go, go tools and small programs written in go, and report back here in the coming weeks to correlate results with the observations from @laboger.

[hramrach]

I suspect this is a bootstrap issue to some extent. You cannot build pie binaries if your stdlib is not PIC. You probably need to rebuild twice, once to build PIC stdlib injecting the PIC flag somewhere, and second time to build the PIE binaries.

[ianlancetaylor]

@hramrach If I understand you correctly, then the go tool should consistently do the right thing. Using go build -buildmode=pie will build all Go code, including the standard library, as appropriate for PIE. Using go build without -buildmode=pie will build all Go code, including the standard library, as appropriate for ordinary executables.

[hramrach]

that's not possible in the case when you are trying to build a pie compiler and all you have is non-PIE compiler with non-PIC stdlib. The new compiler will use the old stdlib which is not suitable. So you first need to build a compiler with PIC stdlib which does not default to PIE and then fully PIE compiler. Assuming the code generation works and correct relocations are generated.

[laboger]

@laboger I don't know why setting GOFLAGS didn't work when building the toolchain. Does it work when using the toolchain? That is, does GOFLAGS=-buildmode=pie go build hello.go build a PIE?

Yes, if I use -x I can see it uses -buildmode=pie on the final link of the executable for hello.

[ianlancetaylor]

@hramrach I'm sorry, I'm not sure what you mean. When you say "stdlib", what are you referring to? When you say "interpreter", do you mean the ELF interpreter, also known as the dynamic linker?

[hramrach]

sorry, I meant compiler. Looks like interpreter is not standard part of go.

[ianlancetaylor]

@hramrach When you say "stdlib", what are you referring to?

[hramrach]

whatever comes with the compiler that might be linked into binaries.

[ianlancetaylor]

Assuming you mean the Go compiler, then, as I explained at #28531 (comment), the Go standard library will be rebuilt, if necessary, with the options required to support the requested build mode. This happens automatically and does not require any user intervention. The results are cached to avoid excess rebuilding.

[laboger]

After I do a normal build, I can do this to make these binaries built as pie:

go install -buildmode=pie cmd/go
go install -buildmode=pie cmd/trace
go install -buildmode=pie cmd/pprof

Does this help?

[laboger]

I would like to find a solution for this in Go 1.15. There are several options but they all have build failures due to the cache. I would need some help from someone who knows more about how the build cache is supposed to work.

[ianlancetaylor]

It's not clear to me what the build cache is doing wrong. Can you provide more details?

[laboger]

I tried various combinations to force binaries to pie but could get nothing to work. Back to original example above, if I try to force everything built for ppc64le to -buildmode=pie then I get the stale errors above. Since I'm not sure how the cache is supposed to work, I don't know why it is complaining. I thought I saw a comment somewhere that said the buildmode was not saved in the cache which could be related but I still don't understand how something would get into the cache at one point and then look stale later if the same build options were used throughout.

Assuming we could get it to work using pie as default, then there could be other issues. One would be how this default gets set, and the fact that we now check gcc's default link mode and change it from pie to no-pie all the time. That would have to change, and maybe that should be the indicator as to when to set Go's default buildmode to pie? If gcc wants pie as default then Go's buildmode should default to pie. Otherwise we provide an environment variable, maybe one used at build time to say this toolchain defaults to pie?

Another question is, does this mean everything defaults to pie, even binaries that would have been built as static? I believe static should still work fine on systems that expect pie as the default unless there is some other reason we don't want static in this environment.

[ianlancetaylor]

The "stale dependency" errors shown in #28531 (comment) aren't about the build cache. Those are coming from cmd/dist, and mean that the package installed by go install is stale by comparison to the source files. When cmd/dist has finished building the tools, it runs a go list command to find all package for which the installed version is considered to be stale. See checkNotStale in cmd/dist/build.go. In your case this may have something to with the -gcflags and -ldflags that cmd/dist passes to go list.

[laboger]

I thought the original suggestion was to set the default buildmode to pie and codegenArg to shared in init.go. Doing that gives the output shown above.

Then I tried modifying lib.go to avoid the errors from loadInternal about runtime/cgo. That eliminates those errors but the rest of the error message is still there.

I did spend some time looking at checkNotStale but I don't understand why go list is complaining that these packages are stale because I don't know how it decides that and looking at the code didn't help to figure that out. Even if I look at the go list help documentation, it doesn't say what causes something to be stale.

Are you saying that the -gcflags and -ldflags passed to go list in this function should use flags for pie and shared instead of blank as the output shows?

[ianlancetaylor]

I'm sorry, I guess I don't understand what is going on.

This issue started as a problem using -buildmode=pie. Do you still have that problem?

Then there was the separate discussion of changing the toolchain to default to generating a PIE. I guess the suggestion of changing cmd/go/internal/work may have been wrong. What was the patch that you have tried? I don't it above; maybe I missed it.

[laboger]

I'm trying to build a toolchain that generates pie binaries by default, including the binaries that are generated during the build. Then the goal would be to have some environment variable or other switch to indicate that it should be done that way. Sorry for the confusion. This has been requested by more than one user but I personally do not have any specific issues with reproducible directions on how to cause errors when we don't default to pie on systems that expect pie. Based on previous test failures we saw with this error message when building for pie, it is probably related to mixing pic and non-pic code which we may or may not be able to fix. (i.e., if linking in a C library that was not built for pic).

I tried a change to cmd/go/internal/work to force buildmode=pie by default, but that causes the errors during the build about runtime/cgo and stale packages coming from checkNotStale in cmd/dist/build.go.

I don't understand why go -list would say the package or binary is stale if it just built it and therefore should be newer than the source. What else would cause it to be considered stale.

Here is an initial patch to set the default to pie and fix up the errors about runtime/cgo. I also added something to remove defaulting to no-pie to see if that made any difference but it doesn't.

golang-pie-simple.diff.txt

[ganl]

go version
go version go1.20.3 linux/amd64

mips64le

release failed after 11s error=failed to build for linux_mips64le_hardfloat: exit status 1: -buildmode=pie not supported on linux/mips64le

mipsle

failed to build for linux_mipsle_hardfloat: exit status 1: -buildmode=pie not supported on linux/mipsle