feat(image): Research nydus/stargz

[gaocegege]

Description

stargz/nydus can accelerate the image load process on Kubernetes. Let's investigate how to integrate and the benefits to AI/ML use case.

---

Message from the maintainers:

Love this enhancement proposal? Give it a 👍. We prioritise the proposals with the most 👍.

[xieydd]

In production enviroment, i rarely see images below 20G, some user even put the data in image. I think image accelerate is highlight and practical solutions to user problems.

[gaocegege]

Yep, I think so.

[gaocegege]

filename: usr/local/lib/python3.8/dist-packages/wrapt-1.14.1.dist-info/top_level.txt, offset: 1046700032, size: 6
filename: usr/local/lib/python3.8/dist-packages/zipp/, offset: 1046701056, size: 0
filename: usr/local/lib/python3.8/dist-packages/zipp/__init__.py, offset: 1046701568, size: 8659
filename: usr/local/lib/python3.8/dist-packages/zipp/__pycache__/, offset: 1046710784, size: 0
filename: usr/local/lib/python3.8/dist-packages/zipp/__pycache__/__init__.cpython-38.pyc, offset: 1046711296, size: 10762
filename: usr/local/lib/python3.8/dist-packages/zipp/__pycache__/py310compat.cpython-38.pyc, offset: 1046723072, size: 406
filename: usr/local/lib/python3.8/dist-packages/zipp/py310compat.py, offset: 1046724096, size: 309
filename: usr/local/lib/python3.8/dist-packages/zipp-3.10.0.dist-info/, offset: 1046725120, size: 0
filename: usr/local/lib/python3.8/dist-packages/zipp-3.10.0.dist-info/INSTALLER, offset: 1046725632, size: 4
filename: usr/local/lib/python3.8/dist-packages/zipp-3.10.0.dist-info/LICENSE, offset: 1046726656, size: 1050
filename: usr/local/lib/python3.8/dist-packages/zipp-3.10.0.dist-info/METADATA, offset: 1046728704, size: 3672
filename: usr/local/lib/python3.8/dist-packages/zipp-3.10.0.dist-info/RECORD, offset: 1046733312, size: 707
filename: usr/local/lib/python3.8/dist-packages/zipp-3.10.0.dist-info/WHEEL, offset: 1046734848, size: 92
filename: usr/local/lib/python3.8/dist-packages/zipp-3.10.0.dist-info/top_level.txt, offset: 1046735872, size: 5

The index generated by https://github.com/awslabs/soci-snapshotter

[gaocegege]

SOCI addresses these issues by loading from the original, unmodified OCI image. Instead of converting the image, it builds a separate index artifact (the "SOCI index"), which lives in the remote registry, right next to the image itself. At container launch time, SOCI Snapshotter queries the registry for the presence of the SOCI index using the mechanism developed by the OCI Reference Types working group.

[gaocegege]

About what the snapshot is:

https://github.com/containerd/containerd/blob/main/docs/content-flow.md

[gaocegege]

• https://github.com/moby/buildkit/blob/master/docs/stargz-estargz.md
• https://github.com/moby/buildkit/blob/master/docs/nydus.md

buildkit can build images with nydus/estargz formats.

[cutecutecat]

Difference of stargz/nydus:

• relationship with stargz-snapshotter? dragonflyoss/nydus#50

Design report:

• estargz: https://github.com/containerd/stargz-snapshotter/blob/main/docs/estargz.md
• nydus: https://github.com/dragonflyoss/image-service/blob/master/docs/nydus-design.md

Pros & Cons:
From my perspective, nydus might be faster, with lower CPU load, but need to introduce an standalone executable file. While estargz is more compatible with buildkit.

There seems some difference in image format of them either, need to do more research.

[gaocegege]

@cutecutecat Are you interested in this? You can pick it up. And I'd appreciate it.

[cutecutecat]

@cutecutecat Are you interested in this? You can pick it up. And I'd appreciate it.

@gaocegege
Yes, I would like to pick it. Are there anything else that need to investigate?

As we have known buildkit can built both of them, I think I could build a large image by buildctl and test the time cost and image size?

• bug(build): Python basic image is too large #1086
• feat(docker): Support buildkitd moby worker in dockerd 22.06-beta #51

[cutecutecat]

Restriction

Nydus

Nydus is conflict with --export-cache and --import-cache, is this acceptable in envd? @gaocegege
I think it might be not.

Since exported Nydus image will always have one more metadata layer than images in other compression types, Nydus image cannot be exported/imported as cache.

ref: https://github.com/moby/buildkit/blob/master/docs/nydus.md and moby/buildkit#2581

Estargz

Rootless execution is currently unsupported.

It seems more acceptable. The cache python, R and julia don't work with rootless now, but we should be careful if we would support rootless of cache in the future.
If we pick Estargz, we should keep a configurable item for image format, instead of substitude origin image format.