[BUG] DeepSpeed on pypi not compatible with latest `numpy`

[younesbelkada]

Describe the bug

Importing deepspeed on a python env with numpy>=2.0.0 fails:

  File "/miniconda3/envs/py39/lib/python3.9/site-packages/deepspeed/autotuning/scheduler.py", line 8, in <module>
    from numpy import BUFSIZE
E   cannot import name 'BUFSIZE' from 'numpy' (/miniconda3/envs/py39/lib/python3.9/site-packages/numpy/__init__.py)

To Reproduce

pip install deepspeed on a env with python>=3.9 and import deepspeed

[loadams]

Thanks @younesbelkada, I believe the second linked PR should resolve this.

[adk9]

@loadams, is it really necessary for us to pin numpy to a pre-2.0.0 version?

I wonder if BUFSIZE was a botched import and the right fix for this is something like:

diff --git a/deepspeed/autotuning/scheduler.py b/deepspeed/autotuning/scheduler.py
index 7d2a1c08..35f709ea 100755
--- a/deepspeed/autotuning/scheduler.py
+++ b/deepspeed/autotuning/scheduler.py
@@ -5,7 +5,6 @@
 
 import copy
 
-from numpy import BUFSIZE
 import json
 import subprocess
 import sys
@@ -18,7 +17,7 @@ import hjson
 from tqdm import tqdm
 
 from ..utils import logger
-from .constants import AUTOTUNING, AUTOTUNING_METRIC_PATH
+from .constants import AUTOTUNING, AUTOTUNING_METRIC_PATH, BUFSIZE
 from .utils import get_val_by_key, search_error, was_interruptted
 """
 thread-0: loop over experiment queue dispatching experiments if they become available

Um, lmk if this makes sense and I can open a PR for it.

[loadams]

Okay @younesbelkada, DeepSpeed master branch should have this resolved here, one we confirm, we can do a new release as well.

#5680

[younesbelkada]

Thank you very much for the quick action @loadams !

[sedletsky-f5]

Okay @younesbelkada, DeepSpeed master branch should have this resolved here, one we confirm, we can do a new release as well.

#5680

please release the fix, to unblock users

[hvaara]

xref #5686

For code archeologists coming here after me: this seems like it was released in v0.14.4. If I don't make an additional update after this, it's probably safe to assume this holds (for package version bumps).

[loadams]

Yes @sedletsky-f5 and @hvaara - this was released in v0.14.4