Support s390x cross-platform go-build by huoqifeng · Pull Request #379 · projectcalico/toolchain

huoqifeng · 2022-03-23T05:23:51Z

Signed-off-by: huoqifeng huoqif@cn.ibm.com

Signed-off-by: huoqifeng <huoqif@cn.ibm.com>

Dockerfile.s390x

caseydavenport · 2022-04-13T23:03:45Z

Dockerfile.s390x

 MAINTAINER LoZ Open Source Ecosystem (https://www.ibm.com/developerworks/community/groups/community/lozopensource)

 ARG MANIFEST_TOOL_VERSION=v1.0.2
+ARG GO111MODULE=auto


I think this should always be "on" now

Agree, corrected.

Also appears to still be here

caseydavenport · 2022-04-13T23:04:10Z

Dockerfile.s390x

+     chmod 0755 /sbin/su-exec; \
+     rm /sbin/su-exec.c
+
+# Install fossa for foss license checks


fossa can actually be removed - we don't use this any more

Looks like this is still here @huoqifeng

Dockerfile.s390x

caseydavenport · 2022-04-13T23:05:32Z

Makefile

 	# Make sure we re-pull the base image to pick up security fixes.
 	# Limit the build to use only one CPU, This helps to work around qemu bugs such as https://bugs.launchpad.net/qemu/+bug/1098729
+ifeq ($(BUILDARCH),s390x)
+	docker build $(DOCKER_BUILD_ARGS) -t $(ARCHIMAGE) -f $(DOCKERFILE) .


Why no --pull here?

Drop --pull for s390x build is because there is no base image calico/bpftool:v5.3-s390x available in docker hub, so I built it locally. Otherwise, it will fail like below:

# ARCH=s390x make image # Make sure we re-pull the base image to pick up security fixes. # Limit the build to use only one CPU, This helps to work around qemu bugs such as https://bugs.launchpad.net/qemu/+bug/1098729 docker build --cpuset-cpus 0 --pull -t calico/go-build:latest-s390x -f Dockerfile.s390x . Sending build context to Docker daemon 720.9kB Step 1/30 : FROM calico/bpftool:v5.3-s390x as bpftool manifest for calico/bpftool:v5.3-s390x not found: manifest unknown: manifest unknown make: *** [Makefile:77: calico/go-build] Error 1

Maybe, the right approach is to build calico/bpftool and go-build for s390x and push them to docker hub as official base image. Could it be achieved after the fix get merged for go-build? Do we have a s390x box in CI? The same reason is for the changes in calico PRs, like projectcalico/calico#5778

There is no s390x version bpftool here https://hub.docker.com/r/calico/bpftool/tags but it can be built successfully based on https://github.com/projectcalico/bpftool, possible to generate a s390x image? so that we don't need drop --pull option.

@caseydavenport May I know why there is no s390x images built for calico/bpftool, go-build and calico, I guess travis supports s390x already for github, right?

Yeah, the "correct" fix here would be to publish an s390x image to dockerhub.

I suspect the problem is that we weren't able to cross-build such an image before? We don't have s390x infra ourselves so need to rely on cross compilation.

@caseydavenport I see, so, is it reasonable to not use --pull option for dependencies when build the s390x image now, so that a local one can be used? Maybe, it could be rollback when there is a s390x build box, like https://github.com/travis-ci/travis-ci.

@caseydavenport are you happy if we drop --pull option for s390x images because which is not enabled in CI and the required images can not be pull from docker hub, We drop the fix as this so make it possible to build all images locally?

@caseydavenport I enabled cross-platform build also for s390x in this PR, to use --pull option we'll depends on bpftool s390x image get built and pushed, which is addressed in projectcalico/bpftool#6

Signed-off-by: huoqifeng <huoqif@cn.ibm.com>

huoqifeng · 2022-05-26T14:11:38Z

We can revise this to get --pull option back after projectcalico/bpftool#6 merged.

Signed-off-by: huoqifeng <huoqif@cn.ibm.com>

…390x

Signed-off-by: huoqifeng <huoqif@cn.ibm.com>

huoqifeng · 2022-05-28T02:44:34Z

Given image calico/bpftool:v5.3-s390x available, the s390x cross-platform build is OK on an amd64 ubuntu20.04 box, here is the evidence:

root@geodesic1:~/go/src/github.com/projectcalico/go-build# uname -a
Linux geodesic1 5.4.0-109-generic #123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

root@geodesic1:~/go/src/github.com/projectcalico/go-build# ARCH=s390x make image
...
 # Make sure we re-pull the base image to pick up security fixes.
# Limit the build to use only one CPU, This helps to work around qemu bugs such as https://bugs.launchpad.net/qemu/+bug/1098729
docker build  --cpuset-cpus 0 -t calico/go-build:latest-s390x -f Dockerfile.s390x .
Sending build context to Docker daemon  714.8kB
Step 1/41 : FROM calico/bpftool:v5.3-s390x as bpftool
 ---> 7ae0c91c5d59
Step 2/41 : FROM alpine:3.11 as qemu
 ---> a787cb986503
Step 3/41 : ARG QEMU_VERSION=6.1.0-8
 ---> Using cache
 ---> 112bb4822efc
Step 4/41 : ARG QEMU_ARCHS="s390x"
 ---> Using cache
 ---> f82b0a2c34b1
Step 5/41 : RUN apk --update add curl
 ---> Using cache
 ---> 16312a59055c
Step 6/41 : RUN for i in ${QEMU_ARCHS}; do curl -L https://github.com/multiarch/qemu-user-static/releases/download/v${QEMU_VERSION}/qemu-${i}-static.tar.gz | tar zxvf - -C /usr/bin; done
 ---> Using cache
 ---> ae9f9082b01a
Step 7/41 : RUN chmod +x /usr/bin/qemu-*
 ---> Using cache
 ---> feb8f989cce8
Step 8/41 : FROM s390x/golang:1.17.9-buster
 ---> 50d68e540606
Step 9/41 : MAINTAINER LoZ Open Source Ecosystem (https://www.ibm.com/developerworks/community/groups/community/lozopensource)
 ---> Using cache
 ---> e9c3a41229f4
Step 10/41 : ARG MANIFEST_TOOL_VERSION=v1.0.2
 ---> Using cache
 ---> 686e1f363c11
Step 11/41 : ARG GO111MODULE=auto
 ---> Using cache
 ---> 47aba8d04819
Step 12/41 : COPY --from=qemu /usr/bin/qemu-*-static /usr/bin/
 ---> Using cache
 ---> 578347a29c01
Step 13/41 : RUN echo 'APT::Default-Release "buster";' > /etc/apt/apt.conf.d/99defaultrelease &&     echo 'deb     http://ftp.am.debian.org/debian/    buster-backports main contrib non-free' > /etc/apt/sources.list.d/buster-backports.list &&     apt-get -y -o Acquire::Check-Valid-Until=false update &&      apt-get -y upgrade &&     apt-get install --no-install-recommends -y -t buster-backports         libbpf-dev linux-headers-5.10.0-0.bpo.9-s390x  &&     apt-get install --no-install-recommends -y         curl bash git openssh-client mercurial make wget util-linux file grep sed jq zip         lsb-release software-properties-common binutils file iproute2         ca-certificates gcc mingw-w64 libc-dev bsdmainutils strace libpcap-dev         apt-utils autoconf automake build-essential         libcurl4-openssl-dev libgeoip-dev liblmdb-dev         libpcre++-dev libtool libxml2-dev libyajl-dev         pkgconf zlib1g-dev &&     rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> ae3d635d2cc3
Step 14/41 : RUN wget https://apt.llvm.org/llvm.sh
 ---> Using cache
 ---> 8b0bc219f01e
Step 15/41 : RUN sed -i 's/apt-get update/apt-get -o Acquire::Check-Valid-Until=false update/g' llvm.sh
 ---> Using cache
 ---> 8f357cba1ee2
Step 16/41 : RUN bash ./llvm.sh 12
 ---> Using cache
 ---> dca493df6d84
Step 17/41 : RUN apt install clang-12
 ---> Using cache
 ---> 30cb8c4d2525
Step 18/41 : ARG SU_EXEC_VER=212b75144bbc06722fbd7661f651390dc47a43d1
 ---> Using cache
 ---> acd8f2421c4f
Step 19/41 : RUN  set -ex;      curl -o /sbin/su-exec.c https://raw.githubusercontent.com/ncopa/su-exec/${SU_EXEC_VER}/su-exec.c;      gcc -Wall /sbin/su-exec.c -o/sbin/su-exec;      chown root:root /sbin/su-exec;      chmod 0755 /sbin/su-exec;      rm /sbin/su-exec.c
 ---> Using cache
 ---> d8ffa194fd6a
Step 20/41 : ARG FOSSA_VER=1.0.1
 ---> Using cache
 ---> 0af5b80cbdb2
Step 21/41 : RUN git clone -b v1.0.1 https://github.com/fossas/fossa-cli.git $GOPATH/src/github.com/fossas/fossa-cli;     cd $GOPATH/src/github.com/fossas/fossa-cli;     make;     cp $GOPATH/bin/fossa /usr/local/bin;     chmod +x /usr/local/bin/fossa
 ---> Using cache
 ---> c64f202c16cd
Step 22/41 : ARG MOCKERY_VER=2.3.0
 ---> Using cache
 ---> d9fc09310b5d
Step 23/41 : RUN git clone -b v2.3.0 https://github.com/vektra/mockery.git $GOPATH/src/github.com/vektra/mockery.git;     cd $GOPATH/src/github.com/vektra/mockery.git;     go get github.com/vektra/mockery/v2/.../;     cp $GOPATH/bin/mockery /usr/local/bin/mockery;     chmod +x /usr/local/bin/mockery
 ---> Using cache
 ---> 2de25e0efe68
Step 24/41 : RUN echo 'Host *' >> /etc/ssh/ssh_config   && echo '    StrictHostKeyChecking no' >> /etc/ssh/ssh_config
 ---> Using cache
 ---> 2ef32acf1b2d
Step 25/41 : ENV GOROOT=/usr/local/go
 ---> Using cache
 ---> 7b9c97ccacde
Step 26/41 : ENV GOCGO=/usr/local/go-cgo
 ---> Using cache
 ---> 67f333e016b6
Step 27/41 : ENV CGO_ENABLED=0
 ---> Using cache
 ---> 93ed9e926f3f
Step 28/41 : RUN cp -a $GOROOT $GOCGO &&   go install -v std &&   rm -rf /go/src/* /root/.cache
 ---> Using cache
 ---> 6e4bdafa28cd
Step 29/41 : RUN wget https://github.com/golangci/golangci-lint/releases/download/v1.46.2/golangci-lint-1.46.2-linux-s390x.tar.gz &&   tar -xvf golangci-lint-1.46.2-linux-s390x.tar.gz && mv golangci-lint-1.46.2-linux-s390x/golangci-lint /usr/local/bin/ &&   rm golangci-lint-1.46.2-linux-s390x.tar.gz && rm -rf golangci-lint-1.46.2-linux-s390x &&   golangci-lint --version
 ---> Using cache
 ---> 273e85204ead
Step 30/41 : RUN go get github.com/onsi/ginkgo/ginkgo &&   go get golang.org/x/tools/cmd/goimports &&   go get github.com/pmezard/licenses &&   go get github.com/wadey/gocovmerge &&   GO111MODULE=on go get github.com/mikefarah/yq/v3 &&   go get -u github.com/jstemmer/go-junit-report &&   go get -u golang.org/x/tools/cmd/stringer &&   GO111MODULE=on go get k8s.io/code-generator/cmd/openapi-gen@v0.24.0 &&   GO111MODULE=on go get k8s.io/code-generator/cmd/deepcopy-gen@v0.24.0 &&   GO111MODULE=on go get k8s.io/code-generator/cmd/client-gen@v0.24.0 &&   GO111MODULE=on go get k8s.io/code-generator/cmd/lister-gen@v0.24.0 &&   GO111MODULE=on go get k8s.io/code-generator/cmd/informer-gen@v0.24.0 &&   GO111MODULE=on go get k8s.io/code-generator/cmd/defaulter-gen@v0.24.0 &&   GO111MODULE=on go get k8s.io/code-generator/cmd/conversion-gen@v0.24.0 &&   rm -rf /go/src/* /root/.cache
 ---> Using cache
 ---> 894a9a5c1713
Step 31/41 : RUN mv /go/bin/ginkgo /go/bin/ginkgo1 &&   go install github.com/onsi/ginkgo/v2/ginkgo@v2.1.3 &&   mv /go/bin/ginkgo /go/bin/ginkgo2 &&   mv /go/bin/ginkgo1 /go/bin/ginkgo
 ---> Using cache
 ---> 9fcc240013c4
Step 32/41 : RUN wget https://dl.k8s.io/v1.24.0/bin/linux/s390x/kube-apiserver -O /usr/local/bin/kube-apiserver && chmod +x /usr/local/bin/kube-apiserver && 	wget https://dl.k8s.io/release/v1.24.0/bin/linux/s390x/kubectl -O /usr/local/bin/kubectl && chmod +x /usr/local/bin/kubectl && 	wget https://dl.k8s.io/v1.24.0/bin/linux/s390x/kube-controller-manager -O /usr/local/bin/kube-controller-manager && chmod +x /usr/local/bin/kube-controller-manager
 ---> Using cache
 ---> fa49f8743e49
Step 33/41 : RUN wget -O ${GOPATH}/bin/controller-gen https://github.com/projectcalico/controller-tools/releases/download/calico-0.1/controller-gen && chmod +x ${GOPATH}/bin/controller-gen
 ---> Using cache
 ---> e2bff9191d6b
Step 34/41 : RUN chmod -R 777 $GOPATH
 ---> Using cache
 ---> 5a5f2fb0fa5d
Step 35/41 : RUN curl -sSL https://github.com/estesp/manifest-tool/releases/download/${MANIFEST_TOOL_VERSION}/manifest-tool-linux-s390x > manifest-tool &&     chmod +x manifest-tool &&     mv manifest-tool /usr/bin/
 ---> Using cache
 ---> 14ddc161b895
Step 36/41 : RUN wget https://github.com/google/go-containerregistry/releases/download/v0.7.0/go-containerregistry_Linux_s390x.tar.gz &&     tar -xvf go-containerregistry_Linux_s390x.tar.gz &&     chmod +x crane &&     mv crane /usr/bin
 ---> Using cache
 ---> ef74329bce0b
Step 37/41 : COPY --from=bpftool /bpftool /usr/bin
 ---> Using cache
 ---> 08551db6eeb0
Step 38/41 : COPY scripts/modsec.sh /usr/local/bin/scripts/modsec.sh
 ---> Using cache
 ---> 4f230328835a
Step 39/41 : RUN /usr/local/bin/scripts/modsec.sh
 ---> Using cache
 ---> 641909866255
Step 40/41 : COPY entrypoint.sh /usr/local/bin/entrypoint.sh
 ---> Using cache
 ---> 0d6fbe97b905
Step 41/41 : ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
 ---> Using cache
 ---> fe491979a555
Successfully built fe491979a555
Successfully tagged calico/go-build:latest-s390x

fasaxc · 2022-05-30T15:37:55Z

/sem-approve

huoqifeng · 2022-05-31T01:05:26Z

@caseydavenport PR build looks good, may you help review it again?

huoqifeng · 2022-05-31T06:58:01Z

@caseydavenport I think the EXCLUDEARCH here https://github.com/projectcalico/go-build/blob/master/Makefile.common#L29 is not used in go-build but I don't see s390x image get built in the PR's pipeline, any suggestions?

huoqifeng · 2022-06-09T07:04:11Z

@caseydavenport any comments here?

caseydavenport

LGTM but maybe you forgot to push a commit?

…390x

huoqifeng · 2022-06-10T02:01:30Z

LGTM but maybe you forgot to push a commit?

@caseydavenport yeah, I copied new contents from latest Dockerfile.amd64, and revised it again. regard fossa, I think it still used in Dockerfile.amd64, should we keep it?

Signed-off-by: huoqifeng <huoqif@cn.ibm.com>

huoqifeng · 2022-06-14T10:03:22Z

@caseydavenport is the new commit OK?

caseydavenport · 2022-06-16T16:53:43Z

/sem-approve

huoqifeng · 2022-06-20T01:31:43Z

/sem-approve

@caseydavenport looks the amd64 and arm64 build failed, but there is no change for that in this PR. Any suggestion?

hjiawei · 2023-12-16T21:27:34Z

All supported architectures are unified in #490. Closing as done.

huoqifeng mentioned this pull request Mar 23, 2022

Enable s390x image cross platform build Dockerfile.s390x in calico projectcalico/calico#5793

Merged

3 tasks

fix s390x go-build for felix build

3b375ef

Signed-off-by: huoqifeng <huoqif@cn.ibm.com>

huoqifeng force-pushed the fix-s390x branch from d4057d2 to 3b375ef Compare March 23, 2022 05:38

Merge branch 'master' into fix-s390x

343aa9d

huoqifeng mentioned this pull request Apr 7, 2022

Enable s390x image cross platform build for calico projectcalico/calico#5778

Closed

3 tasks

caseydavenport reviewed Apr 13, 2022

View reviewed changes

Dockerfile.s390x Outdated Show resolved Hide resolved

caseydavenport reviewed Apr 13, 2022

View reviewed changes

Dockerfile.s390x Show resolved Hide resolved

caseydavenport reviewed Apr 13, 2022

View reviewed changes

Dockerfile.s390x Show resolved Hide resolved

caseydavenport reviewed Apr 13, 2022

View reviewed changes

huoqifeng added 3 commits April 14, 2022 09:20

address review comments

23b96b0

Signed-off-by: huoqifeng <huoqif@cn.ibm.com>

Merge branch 'fix-s390x' of github.com:huoqifeng/go-build into fix-s390x

51c94bf

Merge branch 'master' into fix-s390x

86b66c6

huoqifeng added 4 commits May 27, 2022 09:33

enable cross-platform build

d0fd5b5

Signed-off-by: huoqifeng <huoqif@cn.ibm.com>

Merge branch 'fix-s390x' of github.com:huoqifeng/go-build into fix-s390x

e0bcd67

Merge branch 'master' of github.com:projectcalico/go-build into fix-s…

8b3f47a

…390x

fix golint error

7744a45

Signed-off-by: huoqifeng <huoqif@cn.ibm.com>

huoqifeng force-pushed the fix-s390x branch from 46ca990 to 7744a45 Compare May 27, 2022 14:01

huoqifeng changed the title ~~fix s390x go-build for felix build~~ s390x cross-platform go-build May 28, 2022

huoqifeng changed the title ~~s390x cross-platform go-build~~ Support s390x cross-platform go-build May 28, 2022

caseydavenport reviewed Jun 9, 2022

View reviewed changes

Merge branch 'master' of github.com:projectcalico/go-build into fix-s…

76af8b9

…390x

huoqifeng force-pushed the fix-s390x branch from f035bfa to e380c87 Compare June 10, 2022 02:06

keep align with amd64 Dockerfile

eb4a7c2

Signed-off-by: huoqifeng <huoqif@cn.ibm.com>

huoqifeng force-pushed the fix-s390x branch from e380c87 to eb4a7c2 Compare June 10, 2022 02:44

huoqifeng added 2 commits June 10, 2022 11:12

use latest patch

653b787

Signed-off-by: huoqifeng <huoqif@cn.ibm.com>

drop fossa

09e0351

Signed-off-by: huoqifeng <huoqif@cn.ibm.com>

caseydavenport approved these changes Jun 16, 2022

View reviewed changes

hjiawei closed this Dec 16, 2023

Conversation

huoqifeng commented Mar 23, 2022

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

huoqifeng Apr 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

huoqifeng commented May 26, 2022

Uh oh!

huoqifeng commented May 28, 2022

Uh oh!

fasaxc commented May 30, 2022

Uh oh!

huoqifeng commented May 31, 2022

Uh oh!

huoqifeng commented May 31, 2022

Uh oh!

huoqifeng commented Jun 9, 2022

Uh oh!

caseydavenport left a comment

Choose a reason for hiding this comment

Uh oh!

huoqifeng commented Jun 10, 2022

Uh oh!

huoqifeng commented Jun 14, 2022

Uh oh!

caseydavenport commented Jun 16, 2022

Uh oh!

huoqifeng commented Jun 20, 2022

Uh oh!

hjiawei commented Dec 16, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

huoqifeng Apr 14, 2022 •

edited

Loading