[Dashboard] Provide GPU stats per actor or per worker process in the node table

[scottsun94]

Description

Chatted with @ArturNiederfahrenhorst and having GPU stats per actor or per worker process in the node table could be useful.

Use case

One potential use case for this: if someone is doing a HP sweep with many trials that each doesn't use a full GPU, it can be interesting for sure how much each of them ends up using to find out what parameters cause this.

I want to two Ray Tune trials on a cluster
The cluster has one GPU head node
Each trials should be allocated 1/2 GPU
Now one of the damn trials is super slow
In order to find out why, it would be cool to see how much of the GPU it’s actually utilizing
Since I only see the overall util, the GPU util of any given trial in underdetermined

[stale]

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

• If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
• If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.

[scottsun94]

Similar request: https://ray-distributed.slack.com/archives/CN2RGCHRR/p1686904398769719

[stale]

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

• If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
• If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.

[stale]

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

• If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
• If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.

[stale]

Hi again! The issue will be closed because there has been no more activity in the 14 days since the last message.

Please feel free to reopen or open a new issue if you'd still like it to be addressed.

Again, you can always ask for help on our discussion forum or Ray's public slack channel.

Thanks again for opening the issue!