Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Install pigz and unpigz to speed up docker image extraction #471

Merged

Conversation

peterzhuamazon
Copy link
Member

Description

Install pigz and unpigz to speed up docker image extraction

Issues Resolved

Closes #374

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Peter Zhu <zhujiaxi@amazon.com>
@peterzhuamazon
Copy link
Member Author

Have double verify that pigz will also install unpigz.

Copy link
Member

@gaiksaya gaiksaya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this package on windows as well? To full the images faster?

@prudhvigodithi
Copy link
Collaborator

Thanks, just installing pigz would take care of parallel docker extraction? I'm curious why docker dont directly install it as pre-req. :)

@peterzhuamazon
Copy link
Member Author

peterzhuamazon commented Jul 15, 2024

Do we need this package on windows as well? To full the images faster?

It is already installed on Windows and reduced time from 15-20min to 12min 😄
https://github.com/opensearch-project/opensearch-ci/blob/main/packer/scripts/windows/scoop-install-commons-docker-support.ps1#L76-L87

@peterzhuamazon
Copy link
Member Author

Thanks, just installing pigz would take care of parallel docker extraction? I'm curious why docker dont directly install it as pre-req. :)

Yes, there is a long history of thread to get pigz/unpigz support in docker.
As of now, docker build is still single thread and docker pull/extraction can support unpigz if detected in path.
The thing is docker maintainers seems still thinking parallelism can cause trouble between, thus not suggesting it by default. Thanks.

@peterzhuamazon peterzhuamazon merged commit 482509e into opensearch-project:main Jul 15, 2024
4 checks passed
@peterzhuamazon peterzhuamazon deleted the pigz-unpigz-docker branch July 15, 2024 22:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: ✅ Done
Development

Successfully merging this pull request may close these issues.

Improve Docker Image extraction speed for linux ec2 instances
3 participants