cmd/link: unexpected trampoline when cross-compiling to ppc64le

[dbenoit17]

What version of Go are you using (go version)?

$ go version
1.16.3

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOVERSION="go1.16.3"

What did you do?

Cross compile OpenShift's oc command from x86_64 to ppc64le.

git clone https://github.com/openshift/oc.git && pushd oc
make cross-build-linux-ppc64le

What did you expect to see?

Program builds.

What did you see instead?

Linker error at

go/src/cmd/link/internal/ppc64/asm.go

Line 708 in f12cf76

ctxt.Errorf(s, "unexpected trampoline for shared or dynamic linking")

# github.com/openshift/oc/cmd/oc
github.com/openshift/oc/pkg/cli/observe.(*ObserveOptions).Run.func3: unexpected trampoline for shared or dynamic linking
type..eq.github.com/openshift/oc/pkg/cli/observe.restListWatcher: unexpected trampoline for shared or dynamic linking
type..eq.github.com/openshift/oc/pkg/cli/observe.restListWatcher: unexpected trampoline for shared or dynamic linking
type..eq.github.com/openshift/oc/pkg/cli/observe.newlineTrailingWriter: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.injectUserVars: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.processTemplateLocally: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).Complete.func2: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess.func1: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/process.(*ProcessOptions).RunProcess.func1: unexpected trampoline for shared or dynamic linking
type..eq.github.com/openshift/oc/pkg/cli/process.processPrinter: unexpected trampoline for shared or dynamic linking
github.com/openshift/oc/pkg/cli/project.ProjectOptions.Run: unexpected trampoline for shared or dynamic linking
/root/go/pkg/tool/linux_amd64/link: too many errors

Additional Notes

• This issue is not seen when compiling natively on ppc64le, only during cross-compilation from x86_64.
• The issue was not seen prior to upgrading openshift/oc to use kubernetes 1.21.
• The issue also occurs in Go versions 1.15.8 through 1.15.10, which are the only versions of 1.15 I've tested. However, it produces the error against different oc functions in Go 1.15 vs. Go 1.16.
• Interestingly, 1.16 and 1.16.1 seem unaffected. The issue appears again in 1.16.2, and seems to be triggered by a9547ad#diff-aa02dd8eebb2420ba09bee48832b239fc5f612a07cb7b1709595511fa1d8d0ca. This commit might just uncover the issue rather than be it's direct cause, though, I'm not certain.

[mknyszek]

CC @cherrymui @mdempsky via https://dev.golang.org/owners

[cherrymui]

It is on my plan to rework trampoline insertion in the linker, that should address this. Hopefully I can get it in for 1.17.

[dbenoit17]

Thanks for the quick feedback!

I'm curious if there might be also a possibility for an interim patch to 1.16 in the meantime. One thing that does seem to get the linker past the issue is adjusting this condition to check for ctxt.IsExternal(), to closer match the earlier conditional here. I'm not sure if this is the correct thing to do though. I think it depends whether it's valid for the linker to arrive at line 706 with any of those configuration options when using internal linking.

[cherrymui]

@dbenoit17 thanks for the comment. I'll take a careful look.

[cherrymui]

@dbenoit17 sorry for the delay. As you said, I don't think we use internal linking for any of those configurations. I.e. IsExternal should always be true at that point.

Could you share how the "go" command or the linker is invoked in your build process, and what build mode are you building for? Thanks.

[pmur]

This seems to be caused by the combination of:
CGO_ENABLED=0 and something in openshift/installer using the plugin package causing canUsePlugins to be set, which causes (*Link).DynlinkingGo() to evaulate true, and causes the panic noted above.

Given that plugins are effectively disabled in the above case, shouldn't canUsePlugins also be set to false when linking?

[gopherbot]

Change https://golang.org/cl/314449 mentions this issue: cmd/link: disable plugin support if cgo is explicitly disabled

[dbenoit17]

To hit the issue, go build is called against openshift/oc commit 90af8a5d7a1ae91e3e675607c4f76a403e6ec817 by the Makefile using CGO_ENABLED=0, with the invocation:

go build -mod=vendor -tags 'include_gcs include_oss containers_image_openpgp' -ldflags "-X github.com/openshift/oc/pkg/version.versionFromGit="v4.2.0-alpha.0-1048-g90af8a5" -X github.com/openshift/oc/pkg/version.commitFromGit="90af8a5d7" -X github.com/openshift/oc/pkg/version.gitTreeState="clean" -X github.com/openshift/oc/pkg/version.buildDate="<date>" -X k8s.io/component-base/version.gitMajor="1" -X k8s.io/component-base/version.gitMinor="21" -X k8s.io/component-base/version.gitVersion="v0.21.0-beta.1" -X k8s.io/component-base/version.gitCommit="90af8a5d7" -X k8s.io/component-base/version.buildDate="<date>" -X k8s.io/component-base/version.gitTreeState="clean" -X k8s.io/client-go/pkg/version.gitVersion="v4.2.0-alpha.0-1048-g90af8a5" -X k8s.io/client-go/pkg/version.gitCommit="90af8a5d7" -X k8s.io/client-go/pkg/version.buildDate="<date>" -X k8s.io/client-go/pkg/version.gitTreeState="clean"" -o '_output/bin/linux_ppc64le/oc' github.com/openshift/oc/cmd/oc

I think this is using using buildmode=default since there is no buildmode specified.

As a follow-up, I've had a chance to test https://golang.org/cl/314449 and can confirm it resolves the issue.