optimize memory usage in image load with Docker client integration

[yankay]

Fix: #17785

Because bufferedOpener attempts to load the entire layer into memory, this issue occurs. See https://github.com/google/go-containerregistry/blob/main/pkg/v1/daemon/image.go#L71

This PR addresses high memory consumption issues during minikube image load operations by using docker client directly.

Comparative Analysis: Minikube Image Load Performance

Performance Metrics Summary

Metric	Original Method (minikube image load)	Modified Method (./out/minikube-linux-amd64)	Split Method (docker save + minikube image load)
Total Elapsed Time	104.91 s	23.37 s (↓77.7%)	18.25 s (↓82.6%)
User CPU Time	81.23 s	1.12 s (↓98.6%)	1.18 s (↓98.5%)
System CPU Time	5.29 s	4.32 s (↓18.3%)	4.44 s (↓16.1%)
Peak Memory (RSS)	2,844 MB	90 MB (↓96.8%)	117 MB (↓95.9%)
File System Outputs	1,486,840 blocks	2,678,848 blocks (↑80.2%)	2,678,856 blocks (↑80.2%)
Minor Page Faults	809,957	14,010 (↓98.3%)	17,001 (↓97.9%)
Voluntary Context Switches	235,386	121,542 (↓48.4%)	165,825 (↓29.5%)

---

Original :

/usr/bin/time -v -o output1.log minikube image load elasticsearch:9.0.3  --daemon=true

root@gpu-3090:~# cat  output1.log
        Command being timed: "minikube image load elasticsearch:9.0.3 --daemon=true"
        User time (seconds): 81.23
        System time (seconds): 5.29
        Percent of CPU this job got: 82%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 1:44.91
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 2844216
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 34
        Minor (reclaiming a frame) page faults: 809957
        Voluntary context switches: 235386
        Involuntary context switches: 356
        Swaps: 0
        File system inputs: 5848
        File system outputs: 1486840
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

Modified 

	Command being timed: "./out/minikube-linux-amd64 image load elasticsearch:9.0.3 --daemon=true"
	User time (seconds): 1.12
	System time (seconds): 4.32
	Percent of CPU this job got: 23%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:23.37
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 90328
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 14010
	Voluntary context switches: 121542
	Involuntary context switches: 120
	Swaps: 0
	File system inputs: 0
	File system outputs: 2678848
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0
	
Split Command:

/usr/bin/time -v -o output1.log docker save elasticsearch:9.0.3 -o image.tar

        Command being timed: "docker save elasticsearch:9.0.3 -o image.tar"
        User time (seconds): 0.20
        System time (seconds): 2.86
        Percent of CPU this job got: 22%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:13.77
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 29464
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 2860
        Voluntary context switches: 106670
        Involuntary context switches: 11
        Swaps: 0
        File system inputs: 8
        File system outputs: 2678696
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

 /usr/bin/time -v -o output2.log minikube image load image.tar
         Command being timed: "minikube image load image.tar"
        User time (seconds): 0.98
        System time (seconds): 1.58
        Percent of CPU this job got: 57%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:04.48
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 87716
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 1
        Minor (reclaiming a frame) page faults: 14141
        Voluntary context switches: 59155
        Involuntary context switches: 96
        Swaps: 0
        File system inputs: 0
        File system outputs: 160
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

[minikube-bot]

Can one of the admins verify this patch?

[yankay]

HI @medyagh
Would you please help review it

[nirs]

@yankay So with this we don't need to save the image before loading?

podman save registry/repo/name:tag -o image.tar
minikube load image.tar

This will not help the case when we ned to load into multiple clusters:

podman save registry/repo/name:tag -o image.tar
minikube load -p cluster1 image.tar
minikube load -p cluster2 image.tar

Where the image is saved? how is it cleaned up?

Regardless, loading entire image to memory does not make sense in any case, so this looks like a good change.

[medyagh]

thank you @yankay do you mind putting a Before/After this PR in the description ? to proof the effectiveness ?

[medyagh]

/ok-to-test

[yankay]

@yankay So with this we don't need to save the image before loading?

podman save registry/repo/name:tag -o image.tar
minikube load image.tar

This will not help the case when we ned to load into multiple clusters:

podman save registry/repo/name:tag -o image.tar
minikube load -p cluster1 image.tar
minikube load -p cluster2 image.tar

Where the image is saved? how is it cleaned up?

Regardless, loading entire image to memory does not make sense in any case, so this looks like a good change.

HI @nirs

The image is saved at ~/.minikube/cache/images , ref: https://github.com/kubernetes/minikube/blob/master/pkg/minikube/machine/cache_images.go#L247 .
It seems not cleaned up

[yankay]

@yankay So with this we don't need to save the image before loading?

podman save registry/repo/name:tag -o image.tar
minikube load image.tar

This will not help the case when we ned to load into multiple clusters:

podman save registry/repo/name:tag -o image.tar
minikube load -p cluster1 image.tar
minikube load -p cluster2 image.tar

Where the image is saved? how is it cleaned up?
Regardless, loading entire image to memory does not make sense in any case, so this looks like a good change.

HI @nirs

The image is saved at ~/.minikube/cache/images , ref: https://github.com/kubernetes/minikube/blob/master/pkg/minikube/machine/cache_images.go#L247 .

thank you @yankay do you mind putting a Before/After this PR in the description ? to proof the effectiveness ?

Thanks @medyagh

Memory usage has indeed been reduced, but the execution time has increased significantly. I'll investigate the cause.​​

This resulted in multiple repeated ImageGet. If the image is buffered, it only needs to be loaded once; however, with an unbuffered approach, it must be reloaded repeatedly.
Reference: https://docs.docker.com/reference/api/engine/version/v1.50/#tag/Image/operation/ImageGet

So the PR is changed with another solution by using the docker client directly.

[minikube-pr-bot]

kvm2 driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21103 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 53.2s    │ 53.0s                  │
│ enable ingress │ 15.5s    │ 15.0s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 54.2s 53.0s 52.8s 52.7s 53.2s
Times for minikube (PR 21103) start: 50.6s 57.9s 52.6s 52.2s 51.8s

Times for minikube ingress: 15.6s 16.1s 14.6s 15.2s 16.1s
Times for minikube (PR 21103) ingress: 15.1s 15.0s 15.1s 14.6s 15.1s

docker driver with docker runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21103 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 24.7s    │ 24.5s                  │
│ enable ingress │ 13.1s    │ 12.6s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube (PR 21103) start: 24.6s 23.6s 23.0s 24.5s 26.9s
Times for minikube start: 25.4s 22.9s 25.1s 24.2s 25.9s

Times for minikube ingress: 13.3s 13.8s 12.3s 12.3s 13.8s
Times for minikube (PR 21103) ingress: 12.9s 13.3s 13.3s 13.3s 10.3s

docker driver with containerd runtime

┌────────────────┬──────────┬────────────────────────┐
│    COMMAND     │ MINIKUBE │ MINIKUBE  ( PR 21103 ) │
├────────────────┼──────────┼────────────────────────┤
│ minikube start │ 22.4s    │ 22.4s                  │
│ enable ingress │ 26.0s    │ 26.3s                  │
└────────────────┴──────────┴────────────────────────┘

Times for minikube start: 20.9s 22.6s 21.3s 24.4s 22.6s
Times for minikube (PR 21103) start: 21.6s 21.2s 21.5s 24.3s 23.3s

Times for minikube ingress: 22.8s 22.8s 22.8s 22.9s 38.8s
Times for minikube (PR 21103) ingress: 23.3s 22.8s 38.8s 23.8s 22.8s

[nirs]

@yankay Results looks very good!

Can you compare with the split process?

podman save image -o image.tar
minikube image load image.tar

We still have the issue of cleanup - keeping the image in the cache directory is not good, the user does not know that the image was added and nobody will clean it up. But I think this is not an issue in your change and it was already there.

[yankay]

@yankay Results looks very good!

Can you compare with the split process?

podman save image -o image.tar
minikube image load image.tar

We still have the issue of cleanup - keeping the image in the cache directory is not good, the user does not know that the image was added and nobody will clean it up. But I think this is not an issue in your change and it was already there.

Thanks, the performance information has been added to the PR description :-)
And create a issue for "disk space leak"

[medyagh]

Thank you very much @yankay for providing the benchmarks before/after this PR sounds amazing, I look forward to see more contributions from you

[medyagh]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh, yankay

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [medyagh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[nirs]

@medyagh this broke image load on macOS:

% time minikube image load docker.io/library/alpine:latest -p c1

❌  Exiting due to GUEST_IMAGE_LOAD: Failed to load image: save to dir: caching images: caching image "/Users/nir/.minikube/cache/images/arm64/docker.io/library/alpine_latest": docker save: inspect image docker.io/library/alpine:latest via docker client: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                        │
│    😿  If the above advice does not help, please let us know:                                                          │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                                                        │
│                                                                                                                        │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.                                 │
│    Please also attach the following file to the GitHub issue:                                                          │
│    - /var/folders/v1/lcvpk6v567x43cfsjqg_97t40000gn/T/minikube_image_dda93f623f5c27f0f7578776de8bc3f360c78630_0.log    │
│                                                                                                                        │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

minikube image load docker.io/library/alpine:latest -p c1  0.06s user 0.04s system 4% cpu 2.054 total

I guess this change introduces dependency on docker. Before this change we did not assume docker or podman client.

We don't have automated tests on macOS so this is not easy to test. To ensure we don't have regressions all changes must be tested on macOS.

[nirs]

Before this change:

% git checkout HEAD^
% make
% time minikube image load docker.io/library/alpine:latest -p c1
minikube image load docker.io/library/alpine:latest -p c1  0.07s user 0.12s system 3% cpu 4.989 total

[yankay]

I'm sorry to trouble everyone. 😥