Fix (DeepSpeed) docker image build issue #21002
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ydshieh
commented
Jan 4, 2023
| @@ -117,7 +117,6 @@ jobs: | |||
| name: "Latest PyTorch + DeepSpeed (Push CI - Daily Build)" | |||
| # Can't run in parallel, otherwise get an error: | |||
| # `Error response from daemon: Get "https://registry-1.docker.io/v2/": received unexpected HTTP status: 503 Service Unavailable` | |||
| needs: latest-torch-deepspeed-docker | |||
There was a problem hiding this comment.
No longer need this - and we need to remove this to avoid too long build time due to MAX_JOBS=1 below
ydshieh
commented
Jan 4, 2023
| @@ -27,7 +27,7 @@ RUN python3 -m pip install torch-tensorrt==1.3.0 --find-links https://github.com | |||
| # recompile apex | |||
| RUN python3 -m pip uninstall -y apex | |||
| RUN git clone https://github.com/NVIDIA/apex | |||
| RUN cd apex && python3 -m pip install --global-option="--cpp_ext" --global-option="--cuda_ext" --no-cache -v --disable-pip-version-check . | |||
| RUN cd apex && MAX_JOBS=1 python3 -m pip install --global-option="--cpp_ext" --global-option="--cuda_ext" --no-cache -v --disable-pip-version-check . | |||
There was a problem hiding this comment.
to avoid memory issue happened here.
ydshieh
changed the title
ydshieh
changed the title
ydshieh
changed the title
|
The documentation is not available anymore as the PR was closed or merged. |
stas00
approved these changes
Jan 4, 2023
There was a problem hiding this comment.
This is interesting as I have always seen apex build serially and no matter what I tried I couldn't make it build in parallel. Perhaps this has changed recently.
So yes, your fix is perfect, @ydshieh - and I'd add a comment to why it's there so future readers will not delete it to say make things go faster (as apex takes forever to build).
stas00
reviewed
Jan 4, 2023
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
silverriver
pushed a commit
to silverriver/transformers
that referenced
this pull request
Jan 6, 2023
* Fix docker image build issue * remove comment * Add comment * Update docker/transformers-pytorch-deepspeed-latest-gpu/Dockerfile Co-authored-by: Stas Bekman <stas00@users.noreply.github.com> Co-authored-by: ydshieh <ydshieh@users.noreply.github.com> Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
venkat-natchi
pushed a commit
to venkat-natchi/transformers
that referenced
this pull request
Jan 22, 2023
* Fix docker image build issue * remove comment * Add comment * Update docker/transformers-pytorch-deepspeed-latest-gpu/Dockerfile Co-authored-by: Stas Bekman <stas00@users.noreply.github.com> Co-authored-by: ydshieh <ydshieh@users.noreply.github.com> Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
miyu386
pushed a commit
to miyu386/transformers
that referenced
this pull request
Feb 9, 2023
* Fix docker image build issue * remove comment * Add comment * Update docker/transformers-pytorch-deepspeed-latest-gpu/Dockerfile Co-authored-by: Stas Bekman <stas00@users.noreply.github.com> Co-authored-by: ydshieh <ydshieh@users.noreply.github.com> Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Currently, the docker image build job Latest PyTorch + DeepSpeed from time to time. The issue occurs after #20788 where
apexis recompiled during the build. It seems a resource issue (most likely the memory issue) due to the parallel build (multiple worker). So setMAX_JOB=1to avoid the failure.This will increase the build time to
1h30m, but we have to build 2 same image (for daily CI and push CI), therefore 3h, and this is way too long. Previously those 2 images are built sequentially due to some issue, but now it seems the issue is gone and we can build them in parallel.