[PROTON] Roctracer: convert agent id to gpu id for gpu ops #4090
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Jokeren
reviewed
Jun 7, 2024
| hsa::agentGetInfo<true>( | ||
| agent, static_cast<hsa_agent_info_t>(HSA_AGENT_INFO_DEVICE), | ||
| &deviceType); | ||
| if ((nodeId < deviceOffset) && (deviceType == HSA_DEVICE_TYPE_GPU)) |
There was a problem hiding this comment.
So HSA_GPU devices will have contiguous device offsets? In other words, the following scenario is not possible:
device ids: [2-4], gpu ids: [0-2]
device ids: [8-10], gpu ids: [2-4]
There was a problem hiding this comment.
Yes, that is the current "observation". All cpus are enumerated before any gpus. I do not think that is written in a "contract" anywhere but is 100% consistent. This is not ideal, but the hip change, to include agent/node id in the device info, will remedy any issues there AND deal with the HIP_VISIBLE_DEVICES remapping.
Jokeren
changed the title
Jokeren
approved these changes
Jun 7, 2024
jataylo
pushed a commit
to ROCm/triton
that referenced
this pull request
Jun 19, 2024
…ng#4090) Roctracer reports (global) agent ids for the location of async ops, e.g. kernels and copies. The profiler would be better suited with gpu indexes (zero based). Created a mapping function to apply to values stored in KernelMetric::DeviceId. Caveat: if devices are hidden using HIP_VISIBLE_DEVICES then the hip device id, e.g. via hipGetDevice()/hipSetDevice(), will not match the reported unfiltered id. Additional support in hip will be needed to map through the filtering correctly. --------- Co-authored-by: Keren Zhou <robinho364@gmail.com> (cherry picked from commit 60613fb)
ptillet
pushed a commit
that referenced
this pull request
Jun 20, 2024
Cherry picks for release/3.0.x General: - e8bc45d [BACKEND][AMD] Disable linear layout due to perf regression (#4126) - 9a0a7c2 [AMD] Add basic verification to MFMA encoding (#4117) for RDNA: - 100e2aa [AMD][WMMA] Support dot3d (#3674) - 4a1ea8e [AMD][gfx11] Fix BF16 wmma instr generation (#4135) Proton HIP PRs: - 328b86d [PROTON] Refactor GPU profilers (#4056) - 60613fb [PROTON] Roctracer: convert agent id to gpu id for gpu ops (#4090) - c1776fa [PROTON][AMD] Add Proton HIP GPU Utilization Metrics (#4119) --------- Co-authored-by: Lei Zhang <antiagainst@gmail.com> Co-authored-by: Alexander Efimov <efimov.alexander@gmail.com> Co-authored-by: Ilya V <152324710+joviliast@users.noreply.github.com> Co-authored-by: Keren Zhou <kerenzhou@openai.com> Co-authored-by: mwootton <michael.wootton@amd.com> Co-authored-by: Corbin Robeck <corbin.robeck@amd.com>
bertmaher
pushed a commit
to bertmaher/triton
that referenced
this pull request
Dec 10, 2024
…ng#4090) Roctracer reports (global) agent ids for the location of async ops, e.g. kernels and copies. The profiler would be better suited with gpu indexes (zero based). Created a mapping function to apply to values stored in KernelMetric::DeviceId. Caveat: if devices are hidden using HIP_VISIBLE_DEVICES then the hip device id, e.g. via hipGetDevice()/hipSetDevice(), will not match the reported unfiltered id. Additional support in hip will be needed to map through the filtering correctly. --------- Co-authored-by: Keren Zhou <robinho364@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Roctracer reports (global) agent ids for the location of async ops, e.g. kernels and copies.
The profiler would be better suited with gpu indexes (zero based).
Created a mapping function to apply to values stored in KernelMetric::DeviceId.
Caveat: if devices are hidden using HIP_VISIBLE_DEVICES then the hip device id, e.g. via hipGetDevice()/hipSetDevice(), will not match the reported unfiltered id. Additional support in hip will be needed to map through the filtering correctly.