Skip to content

Fix horovod installation base-cuda Dockerfile #11811

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Feb 10, 2022
Merged

Fix horovod installation base-cuda Dockerfile #11811

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
1aa9682
pip install --user
akihironitta Feb 8, 2022
fa1ba74
add checks
Borda Feb 8, 2022
9411a15
Apply suggestions from code review
Borda Feb 8, 2022
e662ff9
rm unrelated comment
akihironitta Feb 10, 2022
f8e8333
consistent format
akihironitta Feb 10, 2022
5458041
Fail if horovod not found
akihironitta Feb 10, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion dockers/base-conda/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,4 +129,5 @@ RUN \
conda info && \
pip list && \
python -c "import sys; ver = sys.version_info ; assert f'{ver.major}.{ver.minor}' == '$PYTHON_VERSION', ver" && \
python -c "import torch; assert torch.__version__.startswith('$PYTORCH_VERSION'), torch.__version__"
python -c "import torch; assert torch.__version__.startswith('$PYTORCH_VERSION'), torch.__version__" && \
python -c "import horovod.torch"
14 changes: 8 additions & 6 deletions dockers/base-cuda/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,6 @@ ENV \
COPY ./requirements.txt requirements.txt
COPY ./requirements/ ./requirements/

# conda init
RUN \
wget https://bootstrap.pypa.io/get-pip.py --progress=bar:force:noscroll --no-check-certificate && \
python${PYTHON_VERSION} get-pip.py && \
Expand All @@ -88,11 +87,11 @@ RUN \
python ./requirements/adjust_versions.py requirements/extra.txt ${PYTORCH_VERSION} && \
python ./requirements/adjust_versions.py requirements/examples.txt ${PYTORCH_VERSION} && \
# Install all requirements
pip install -r requirements/devel.txt --no-cache-dir && \
pip install --user -r requirements/devel.txt --no-cache-dir && \
rm -rf requirements.* requirements/

RUN \
CUDA_VERSION_MAJOR=$(python -c "import torch ; print(torch.version.cuda.split('.')[0])") && \
CUDA_VERSION_MAJOR=$(python -c "import torch; print(torch.version.cuda.split('.')[0])") && \
py_ver=$(python -c "print(int('$PYTHON_VERSION'.split('.') >= '3.9'.split('.')))") && \
# install DALI, needed for examples
# todo: waiting for 1.4 - https://github.com/NVIDIA/DALI/issues/3144#issuecomment-877386691
Expand All @@ -108,15 +107,18 @@ RUN \

RUN \
# install FairScale
pip install fairscale==0.4.0
pip install fairscale==0.4.0 && \
python -c "import fairscale; print(fairscale.__version__)"

RUN \
# install DeepSpeed
pip install deepspeed==0.5.7
pip install deepspeed==0.5.7 && \
python -c "import deepspeed; print(deepspeed.__version__)"

RUN \
# Show what we have
pip --version && \
pip list && \
python -c "import sys; ver = sys.version_info ; assert f'{ver.major}.{ver.minor}' == '$PYTHON_VERSION', ver" && \
python -c "import torch; assert torch.__version__.startswith('$PYTORCH_VERSION'), torch.__version__"
python -c "import torch; assert torch.__version__.startswith('$PYTORCH_VERSION'), torch.__version__" && \
python -c "import horovod.torch"