Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignore reuse_dist_env #6623

Merged
merged 2 commits into from
Oct 14, 2024
Merged

Ignore reuse_dist_env #6623

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c3f7b44
ignore reuse_dist_env
tohtana Oct 12, 2024
c6768e3
set reuse_dist_env to False
tohtana Oct 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions tests/unit/common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@
# Worker timeout for tests that hang
DEEPSPEED_TEST_TIMEOUT = int(os.environ.get('DS_UNITTEST_TIMEOUT', '600'))

warn_reuse_dist_env = False


def is_rocm_pytorch():
return hasattr(torch.version, 'hip') and torch.version.hip is not None
Expand Down Expand Up @@ -179,6 +181,13 @@ def _launch_daemonic_procs(self, num_procs):
print("Ignoring reuse_dist_env for hpu")
self.reuse_dist_env = False

global warn_reuse_dist_env
if self.reuse_dist_env and not warn_reuse_dist_env:
# Currently we see memory leak for tests that reuse distributed environment
print("Ignoring reuse_dist_env and forcibly setting it to False")
warn_reuse_dist_env = True
self.reuse_dist_env = False

if self.reuse_dist_env:
if num_procs not in self._pool_cache:
self._pool_cache[num_procs] = mp.Pool(processes=num_procs)
Expand Down
Loading