Instructions at https://minikube.sigs.k8s.io/docs/contrib/building/iso/ are out of date

[luna-duclos]

What Happened?

I followed the instructions on https://minikube.sigs.k8s.io/docs/contrib/building/iso/ to build a custom ISO and couldn't follow the steps.

I tried this on both an M1 mac as well as an amd64 linux machine and met different issues, lets go over them:

Let's start with linux amd64 as the instructions seem to be intended to be done there and arm64 to be a cross-compile:

Some of the packages built rely on the docker CLI and thus need docker-in-docker working as well as the docker command line tool, to get this working I added the following to deploy/iso/minikube-iso/Dockerfile:

COPY --from=docker:dind /usr/local/bin/docker /usr/local/bin/

With that added, the command from the doc worked:

make out/minikube-arm64.iso

Trying to get things running natively on my M1 mac, I met further issues:

• gcc-multilib only exists for amd64. for arm64, I had to remove it and instead add gcc-9-multilib-x86-64-linux-gnu to be able to compile the amd64 iso, the arm64 iso meanwhile didn't need any multilib packages.
• The approach in the Makefile that binds the entire repo into a docker volume on /mnt does not work at all, it causes odd issues with sed, as well as weird build errors that don't make sense. Instead I had to git clone the repo onto the container's native filesystem. The command for docker run ended up being as follows: docker run -i -t -v /var/run/docker.sock:/var/run/docker.sock --workdir / --volume $HOME/Code/github/kubernetes/minikube:/mnt --user 0:1 --env HOME=/tmp --env IN_DOCKER=1 gcr.io/k8s-minikube/buildroot-image /bin/bash

From there, I could finally successfully do the following in the container:

make /build
git clone https://github.com/kubernetes/minikube.git
cd minikube
make out/minikube-arm64.iso
mkdir -p /mnt/out
cp out/minikube-arm64.iso /mnt/out

HOWEVER ... The adventure continues, and I noticed from there that the iso didn't have any Go based tools on it, this is because HOSTARCH was detected as aarch64, which is not a valid go bootstrap arch.

This bit I am still working on :) (Previous instructions were incorrect, edited them out)

I'm super happy to either make PRs to fix the docker images, update instructions, etc. but would need some guidance on how you folks would like this done.

Attach the log file

N/A

Operating System

macOS (Default)

Driver

N/A

[afbjorklund]

The gcc-multilib is a known issue, inherited from upstream. The workaround sounds reasonable.

https://github.com/buildroot/buildroot/blob/master/support/docker/Dockerfile

As far as I know, the build is not regularly tested on Macs or in Docker. Think builder runs Debian?

Binding is standard procedure on Linux, but might run into issues when using Mac filesystems.

[afbjorklund]

• Building an ISO under linux/arm64 does not work #14500

[luna-duclos]

I did some further research and to bootstrap from an Mac OS M1 laptop, you'd want:

• An arm64 VM (Not a docker container to get around the filesystem and docker-in-docker issues)
• This patch applied on buildroot to be able to bootstrap go on arm64: https://bugs.buildroot.org/show_bug.cgi?id=13866

Would it be worth trying to get this working for minikube or is bootstrapping from M1 not worth the effort ?

(I haven't tried and have reverted to using a seperate amd64 linux laptop for bootstrapping the iso)

[afbjorklund]

We want do some Buildroot cleanup, and this could be a part of it...

Using a VM would be the recommended workaround (you have to, anyway)

[luna-duclos]

Any tracking issue or PR I can keep an eye on ? Would love to follow along

[afbjorklund]

Not really, it was supposed to go on the updated roadmap though.

The overall plan is to break out both libmachine with drivers and minikube-iso with toolbox, from the minikube monorepo... https://github.com/minikube-machine

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.