parallel build no longer works

[stas00]

I used to prebuild deepspeed with:

DS_BUILD_CPU_ADAM=1 DS_BUILD_AIO=1 DS_BUILD_TRANSFORMER_INFERENCE=1 DS_BUILD_UTILS=1 pip install -e . --global-option="build_ext" --global-option="-j8" --no-cache -v --disable-pip-version-check 2>&1 | tee build.log

it no longer works. I have to use --global-option="-j1" for it to succeed.

Log file attached. build-log.txt

Thank you!

[mrwyattii]

Hi @stas00, I'm not able to reproduce the error you are seeing locally (I copy and pasted the command you shared). Can you share a bit more information?

• Is the error message the same when you repeat the command?
• What version of nvcc and cuda do you have?
• What version of pytorch do you have?
• Is this with the latest master branch of DeepSpeed or another version?

Thanks!

[stas00]

Hi @stas00, I'm not able to reproduce the error you are seeing locally (I copy and pasted the command you shared). Can you share a bit more information?

Of coure, @mrwyattii

* Is the error message the same when you repeat the command?

well, my build script removes the build dir before building.

I have tested that if I don't remove the build dir, and run the build 2 times, the process succeeds on the 2nd run.

Please make sure you remove the build dir to repro.

my script is:

#!/bin/bash
rm -rf build
DS_BUILD_CPU_ADAM=1 DS_BUILD_AIO=1 DS_BUILD_TRANSFORMER_INFERENCE=1 DS_BUILD_UTILS=1 pip install -e . --global-option="build_ext" --global-option="-j8" --no-cache -v --disable-pip-version-check 2>&1 | tee build.log

* What version of nvcc and cuda do you have?

Cuda compilation tools, release 11.7, V11.7.64

* What version of pytorch do you have?

pt=1.13.1, cuda=11.7, nccl=(2, 14, 3)

* Is this with the latest master branch of DeepSpeed or another version?

master, just retested with today's master - same. My apologies for not providing this info in the first place.

[mrwyattii]

Thanks @stas00 - I've matched your environment as close as I can, but I'm still not seeing this error. I'll try on a few different system and let you know. Just curious, do you see this on other systems as well?

[stas00]

I tried on another system, but DS_BUILD_TRANSFORMER_INFERENCE=1 build was failing completely with gcc errors, but it build just fine with -j8 w/o this component.

And actually once I removed DS_BUILD_TRANSFORMER_INFERENCE=1 from the system I made the OP report with (my desktop) the -j8 problem disappears as well. So it has something to do with the inference component. perhaps some dependency in its build files are wrong. As if the build runs the 2nd time over the existing files it suceeds.

[stas00]

I guess I should tag @RezaYazdaniAminabadi on this one, as Reza developed this component and it used to compile w/ -j just fine until recently. Absolutely no rush, just reporting the change.