minikube 1.33.1 s390x build doesn't run on s390x

[djfitzgerald]

What Happened?

Environment:
Linux s390x

Minikube version (use minikube version): v1.33.1

• OS (e.g. from /etc/os-release):
  NAME="Red Hat Enterprise Linux"
  VERSION="9.4 (Plow)"
  ID="rhel"
  ID_LIKE="fedora"
  VERSION_ID="9.4"
  PLATFORM_ID="platform:el9"
  PRETTY_NAME="Red Hat Enterprise Linux 9.4 (Plow)"
  ANSI_COLOR="0;31"
  LOGO="fedora-logo-icon"
  CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
  HOME_URL="https://www.redhat.com/"
  DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
  BUG_REPORT_URL="https://bugzilla.redhat.com/"

  REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
  REDHAT_BUGZILLA_PRODUCT_VERSION=9.4
  REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
  REDHAT_SUPPORT_PRODUCT_VERSION="9.4"

• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): none

• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): -

• Install tools:

What happened:
Binary installed by minikube-1.33.1-0.s390x.rpm is not runnable on s390x Linux

What you expected to happen:
Running minikube start to do something other than give me a bash exec error

How to reproduce it (as minimally and precisely as possible):
Using a s390x Linux system (like one of the ones available here), perform the following:

wget https://github.com/kubernetes/minikube/releases/download/v1.33.1/minikube-1.33.1-0.s390x.rpm
sudo rpm -i minikube-1.33.1-0.s390x.rpm
minikube start

You will get the following error:

-bash: /usr/bin/minikube: cannot execute binary file: Exec format error

Output of minikube logs (if applicable):

Anything else do we need to know:
You can sign up for access to the IBM LinuxONE community cloud here and obtain a RHEL 9 s390x virtual machine to test with.

Attach the log file

Not applicable.

Operating System

Redhat/Fedora

Driver

None

[medyagh]

djfitzgerald is that for the mainframe linux ? we dont have a official support for them or integration tests but I am curious does the Older version of minikube work on it ?

[afbjorklund]

There is some bug in the release script, the RPM contains the arm binary - not the s390x binary.

./usr/bin/minikube: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, Go BuildID=Ot8jrmKwY6wlYg-SYwCj/gWb_TQXyvMg3tSnl-m8j/xltcIiTQW60Dc3Ekop/M0lDlx1eCRDr9qvoeDoc, with debug_info, not stripped

The binary is OK:

minikube-linux-s390x: ELF 64-bit MSB executable, IBM S/390, version 1 (SYSV), statically linked, Go BuildID=AdItl4nitWKhoNWM2crG/Btk6NkU7OheQpOoMd9SL/9jy-Tw-5rkbxZeFQqbCq/wDUYzlGxmcoDKHx2Ld-7, with debug_info, not stripped

[djfitzgerald]

@medyagh Correct, mainframe linux. You may want to look into using the LinuxONE community cloud at Marist (I put a couple of links above), those systems are for porting stuff to Linux on Z (mainframe Linux). But make sure you double-check the terms of service first.

[medyagh]

I think @afbjorklund is right that could be related to the script that generates the RPM, if anyone makes PR I would be happy to review and accept it

[spowelljr]

@afbjorklund How is v1.33.0? I wonder if this is also related to updating the Jenkins machine OS.

[afbjorklund]

Unfortunately the RPM build directory does not include arch, so there is a race condition when using more than -j1

Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.bEeGwK
+ umask 022
+ cd /home/anders/rpmbuild/BUILD
+ mkdir -p minikube-1.33.1
+ cd minikube-1.33.1
+ cp /home/anders/minikube/out/minikube-linux-x86_64 minikube
+ exit 0
Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.60UYYI
+ umask 022
+ cd /home/anders/rpmbuild/BUILD
+ cd minikube-1.33.1
+ mkdir -p /home/anders/rpmbuild/BUILDROOT/minikube-1.33.1-0.x86_64/usr/bin
+ install -m 755 minikube /home/anders/rpmbuild/BUILDROOT/minikube-1.33.1-0.x86_64/usr/bin/minikube

The release build uses make -j16, so there is plenty of opportunity. A simple workaround is to add the arch:

%prep
mkdir -p %{name}-%{version}
cd %{name}-%{version}
cp --OUT--/minikube-linux-%{_arch} minikube

%install
cd %{name}-%{version}
mkdir -p %{buildroot}%{_bindir}
install -m 755 minikube %{buildroot}%{_bindir}/%{name}

[spowelljr]

Ah, I created a PR (#18948) to run the RPM builds one at a time, but I can do your above suggestion instead @afbjorklund

[spowelljr]

I'll rebuild the RPM packages and push them to the release

[spowelljr]

I downloaded the existing s390x minikube binary to ensure the binary SHA matches, and used it to build the s390x rpm. I replaced it both is GCS and GitHub release page. I checked the rest of the RPMs but only s390x was affected. Issue should be resolved now, could you confirm @djfitzgerald

[djfitzgerald]

@spowelljr Fix is in, can confirm

[djfitzgerald]

@medyagh Also: I was just told by IBM's OSS folks that they have a specific program for open source developers, so they can apply for a VM that stays around for longer so it can be used for things such as CI: https://community.ibm.com/zsystems/form/l1cc-oss-vm-request/

[spowelljr]

This also doesn't seem like the first time, just the first time someone reported it. I went through past releases and there are RPMs that have malformed binaries, likely caused by overlapping cp commands to the same location. Abnormally small files sizes is a dead give away.

We get a too large section header offset error

$ file -b usr/bin/minikube 
ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically linked, too large section header offset 50424640

[spowelljr]

Since this is resolved I'll close the issue, thanks for bringing it to our attention @djfitzgerald

[djfitzgerald]

@spowelljr You're welcome! I would strongly recommend that your team consider getting an s390x VM from IBM to use in your CI pipeline