Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[air/benchmark] Torch benchmarks for 4x4 #26692

Merged
merged 29 commits into from
Jul 19, 2022
Merged

[air/benchmark] Torch benchmarks for 4x4 #26692

Changes from 1 commit
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
72108b7
use local rank for cuda device
JiahaoYao Jul 13, 2022
ee70c88
adding the test
JiahaoYao Jul 13, 2022
e72378c
lint
JiahaoYao Jul 13, 2022
6a8e934
fix ci
JiahaoYao Jul 13, 2022
e4e967e
revert the test
JiahaoYao Jul 13, 2022
4a8112e
gpu test
JiahaoYao Jul 13, 2022
e2911c7
not use private utils
JiahaoYao Jul 13, 2022
09f2fa5
adding the test and comment
JiahaoYao Jul 18, 2022
0f9d437
Merge remote-tracking branch 'upstream/master' into 0712_gpuid
JiahaoYao Jul 18, 2022
de881c4
fix error
JiahaoYao Jul 18, 2022
0727edd
adding the tune test
JiahaoYao Jul 18, 2022
8975238
monkeypatch
JiahaoYao Jul 18, 2022
ab129db
update
richardliaw Jul 18, 2022
5d680e4
update
richardliaw Jul 18, 2022
08fdb5e
Merge branch 'master' into 4x4-torch
richardliaw Jul 18, 2022
e6c3f3a
update-gpu
richardliaw Jul 18, 2022
796edcc
update-benchmarks
richardliaw Jul 19, 2022
a5fa0f4
Merge branch 'master' into 4x4-torch
richardliaw Jul 19, 2022
cff0ff4
Merge branch 'master' into air/benchmark-torch-4x4-gpu
Jul 19, 2022
8d95a97
Correctly set GPU IDs
Jul 19, 2022
5ea36cc
Fix serialization issues
Jul 19, 2022
049a4e4
Fix serialization issue (again)
Jul 19, 2022
9119ff5
Configurable batch size
Jul 19, 2022
1650090
Sleep before Ray
Jul 19, 2022
0396fff
Increase epochs
Jul 19, 2022
bc82270
Decrease freuqnecy, add smoke test
Jul 19, 2022
62a814e
Wait for 2 nodes in smoke test
Jul 19, 2022
c871b88
Add smoke test argument
Jul 19, 2022
e68b62c
Fix tf benchmark
Jul 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
use local rank for cuda device
  • Loading branch information
@JiahaoYao
JiahaoYao authored Jul 13, 2022
commit 72108b792f88336664dd3cfdef4a618ec8c64413
4 changes: 3 additions & 1 deletion python/ray/train/torch/train_loop_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -467,7 +467,9 @@ def get_device(self) -> torch.device:
if torch.cuda.is_available():
gpu_ids = ray.get_gpu_ids()
if len(gpu_ids) > 0:
device_id = gpu_ids[0]
gpu_id = gpu_ids[0]
cuda_visible_list = list(map(int, ray._private.utils.get_cuda_visible_devices()))
device_id = cuda_visible_list.index(gpu_id)
else:
# If called on the driver or outside of Ray Train, return the
# 0th device.
Expand Down