"minikube image save" is extremely slow

[Pictor13]

What Happened?

I'm trying to backup (over local host) some docker images that take hours to build (not images of mine; I have no way to reduce that).
This because I want to minikube delete and then restore everything, removing the re-build time.

Running

cd /mini-backup/
minikube image save myimage:mytag myimage_mytag.tar

works......
...but takes so long to complete.
Is the command supposed to be that slow?
Should I expect the same slowness also when putting the images back with minikube image load?

The whole reason I was trying to backup & restore images, was to be able to delete the VM and come back to productivity quickly.
But with these timings the strategy doesn't really serve, and I might well delete and rebuild everything from scratch.

Am I doing something wrong?
How can I take the images out of Minikube to restore them afterwards? (in decent times)

Additionally:

• I could run a local repository that stores in a local volume mount, but that would require renaming the repository name everywhere in my codebase in order to interface with it.
• just in case, yes, I'm using Virtualbox & no, I can't switch to hyperkit.

Attach the log file

[log contains too much private data; and is not relevant here: eventually ask for specific info from it]

Operating System

macOS (Default)

Driver

VirtualBox

[spowelljr]

Hi @Pictor13, thanks for reporting your issue with minikube!

Curious, how big is the image you're trying to save?

[Pictor13]

The image size is ~3.77GB 🙁

[spowelljr]

I just tried myself but on a different setup, I did it on Linux using the Docker driver

REPOSITORY                                TAG       IMAGE ID       CREATED          SIZE
benchmark                                 latest    8018f5a27c80   59 seconds ago   3.01GB

$ minikube image save benchmark:latest benchmark_latest.tar
real    1m5.835s
user    0m8.389s
sys     0m8.849s

How long was yours taking to complete?

[afbjorklund]

Is the command supposed to be that slow?

Yes. It will save a local copy on the VM, before transfering. A local registry is probably the easiest solution.

[afbjorklund]

• I could run a local repository that stores in a local volume mount, but that would require renaming the repository name everywhere in my codebase in order to interface with it.

Just retag the image after loading or pulling it. You could also set your registry up as a pull-through cache.

[Pictor13]

@afbjorklund could the pull-through cache spare me from re-tagging the images?

I used to run a local registry on Minikube, as cache (ideal for travelling & no connection); but I couldn't make it "transparent":
proxying the remote registry (myprivate.registry.net) via the local registry (registry2.local, set as pull-through cache), required additional operations on my side (re-tagging; or updating URIs in codebase to fetch images from registry2.local rather than myprivate.registry.net).

I wanted to echo "myprivate.registry.net 127.0.0.1" > /etc/hosts on Minikube VM, to use the local registry transparently; but the hosts change would prevent the local registry to resolve the remote registry IP. 😕

[Pictor13]

How long was yours taking to complete?

I don't have an exact benchmark, but the saving was taking some 10-20 minutes. And the loading 5-10 minutes.

Thanks for the benchmark @spowelljr, tho. I'm not sure if the different engine & OS could cause such difference.

[afbjorklund]

The registry might be hard-coded to only proxy the "default" (docker.io)

https://docs.docker.com/registry/recipes/mirror/

---

If you run with a higher log-level, you can see the actual commands...
It will go faster, if you skip the temporary file and pipe directly to output.

e.g. docker image save myimage:mytag > myimage_mytag.tar

We couldn't do this in the general API, because every CRI is "special"
So it only works with local (to the VM) files, not with the actual streams.

[afbjorklund]

A separate data disk would have helped with extreme (GB) cases like this one...

• Add possibility to add data volumes #12166

You can do it manually, with VirtualBox. Create a second disk, and attach it to the VM.

Then you can keep your disk between clusters, and save and load the image from there ?

[medyagh]

@Pictor13
have u tried adding a extra disk to minikube ? (currently only supported by hyperkit and kvm2)

      --extra-disks=0: Number of extra disks created and attached to the minikube VM (currently only implemented for hyperkit and kvm2 drivers)

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[klaases]

Hi @Pictor13, have you had a chance to try adding extra disk to minikube as mentioned above by @medyagh?

[klaases]

Hi @Pictor13 – is this issue still occurring? Are additional details available? If so, please feel free to re-open the issue by commenting with /reopen. This issue will be closed as additional information was unavailable and some time has passed.

Additional information that may be helpful:

• Whether the issue occurs with the latest minikube release

• The exact minikube start command line used

• Attach the full output of minikube logs, run minikube logs --file=logs.txt to create a log file

Thank you for sharing your experience!

[cerebrotecnologico]

/reopen

I am observing the same issue.
I have docker running with lima in MacOSX (intel)
I have several images that need to be available in minikube.
I decided to pull the images with docker and then load them into minikube.
But loading the images is extremely slow.
There is no advantage on perhaps pulling the images directly in minikube rather than copying them locally with minikube image load

$ minikube version
minikube version: v1.30.1
commit: 08896fd1dc362c097c925146c4a0d0dac715ace0

$ docker pull some-large-image:v1

$ minikube start --extra-config=apiserver.service-node-port-range=1000-65535 \
               --memory=16384m \
               --cpus=max \
               --disk-size 128GB

$ minikube image load "some-large-image:v1" --overwrite=false

[cerebrotecnologico]

/reopen

[k8s-ci-robot]

@cerebrotecnologico: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.