GPU improved launch parameters, memory usage, etc. #1100
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…n block adjustment.
…ooks like a memory leak.
…lation tests work again
…ko_dgx_CI_launcparams
…ll a bit too for CI, A100 placements remain broken.
markusbattarbee
added
improved-memory-usage
gpu
profiling
CI
|
Now it would be possible to go back to a single ukko CUDA makefile, compiling and linking the binary to support both V100 and A100 cards (architectures 70 and 80) but compilation will take longer, and compiling for an older architecture should be safe. Thoughts? |
ykempf
approved these changes
Feb 19, 2025
|
And here too the output file is truncated mid-Ionos |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Differentiates between Ukko DGX (V100 nodes) and A100 nodes. Updates to up-to-date modules. DGX has much better queue times, but performance is still a bit too slow for CI (about 1h 20 mins). A100 nodes suffer, in addition to terrible queue times, from terrible core/thread/GPU placement, awaiting fixes from IT4sci.
GPU initial memory buffers were changed to scale with WID value so WID=8 binaries don't hog up crazy amounts of memory. Also fixed a mis-indexing in GPU memory reporting. Some tests still report more GPU memory use than expected, pending investigation.
Cleans away some broken timers from GPU branch.
Improves readability of batch operation launch parameters (use of
std::max)Improves launch parameters for GPU kernels, especially when not using warp accessors (which is the default).