Make building the default python concurrent rather than sequential #7949
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thought of this while I was driving down the road running an errand and couldn't resist throwing it up... 😁 |
jeffwidman
force-pushed
the
make-installing-apt-packages-concurrent-with-python-3.11
branch
from
d867aa0 to
fe09d18
Compare
jeffwidman
commented
Sep 1, 2023
|
|
||
| RUN pyenv global $PY_3_11 | ||
|
|
||
| FROM python-core | ||
| # Install C-libs needed to build users' Python packages. Please document why each package is needed. | ||
| USER root | ||
| RUN apt-get update \ |
There was a problem hiding this comment.
This is the list of package installs that we now do concurrently with the python builds... It shouldn't take that long, but nevertheless it's faster to do this concurrently, plus I expect this list to grow over time as users report C-libs that they need installed to build their deps.
jeffwidman
force-pushed
the
make-installing-apt-packages-concurrent-with-python-3.11
branch
from
fe09d18 to
c467a6e
Compare
Previously we'd build all the non-default pythons concurrently with building the default python, and then sequence installing the `apt` packages behind building the default Python. However, these `apt` packages can take a little while to fetch/install, and they don't require the default Python, so instead parallelize building the default python at the same time as installing the apt packages. This also makes the flow consistent across all pythons of building in a separate stage and then copying to the final image.
jeffwidman
force-pushed
the
make-installing-apt-packages-concurrent-with-python-3.11
branch
from
c467a6e to
bdfa604
Compare
jeffwidman
deleted the
make-installing-apt-packages-concurrent-with-python-3.11
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously we'd build all the non-default pythons concurrently with building the default python, and then sequence installing the
aptpackages behind building the default Python.However, these
aptpackages can take a little while to fetch/install, and they don't require the default Python, so instead parallelize building the default python at the same time as installing the apt packages.This also makes the flow consistent across all pythons of building in a separate stage and then copying to the final image.