Count shard state in HBM usage #3114
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for pytorch-fbgemm-docs ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
|
This pull request was exported from Phabricator. Differential Revision: D61576911 |
levythu
added a commit
to levythu/torchrec
that referenced
this pull request
Sep 11, 2024
Summary: X-link: facebookresearch/FBGEMM#203 X-link: pytorch/FBGEMM#3114 This PR improve sparse HBM cost by accounting the size of auxilirary state for maintaining UVM cache. As noted in the comment of split_table_batched_embeddings_ops_training, for now the significant space is `4 * table_height + 8 * cache_height + 8 * cache_height`. This is becoming more nontrivial if we have a table with many rows but few dimensions. Impact: - Not UVM-offloaded job: NoOp - UVM-offloaded job: More balanced memory usage from precise estimation, but for existing UVM jobs with scale up proposer + fixed percentage reservation this might lead to scale up proposer making less aggressive cache scale-up and therefore leading to worse performance. In this case we should tune to more slack reservation percentage . Differential Revision: D61576911
|
This pull request was exported from Phabricator. Differential Revision: D61576911 |
levythu
added a commit
to levythu/torchrec
that referenced
this pull request
Sep 11, 2024
Summary: Pull Request resolved: pytorch#2380 X-link: facebookresearch/FBGEMM#203 X-link: pytorch/FBGEMM#3114 This PR improve sparse HBM cost by accounting the size of auxilirary state for maintaining UVM cache. As noted in the comment of split_table_batched_embeddings_ops_training, for now the significant space is `4 * table_height + 8 * cache_height + 8 * cache_height`. This is becoming more nontrivial if we have a table with many rows but few dimensions. Impact: - Not UVM-offloaded job: NoOp - UVM-offloaded job: More balanced memory usage from precise estimation, but for existing UVM jobs with scale up proposer + fixed percentage reservation this might lead to scale up proposer making less aggressive cache scale-up and therefore leading to worse performance. In this case we should tune to more slack reservation percentage . Differential Revision: D61576911
levythu
added a commit
to levythu/FBGEMM
that referenced
this pull request
Sep 11, 2024
Summary: X-link: pytorch/torchrec#2380 X-link: facebookresearch/FBGEMM#203 Pull Request resolved: pytorch#3114 This PR improve sparse HBM cost by accounting the size of auxilirary state for maintaining UVM cache. As noted in the comment of split_table_batched_embeddings_ops_training, for now the significant space is `4 * table_height + 8 * cache_height + 8 * cache_height`. This is becoming more nontrivial if we have a table with many rows but few dimensions. Impact: - Not UVM-offloaded job: NoOp - UVM-offloaded job: More balanced memory usage from precise estimation, but for existing UVM jobs with scale up proposer + fixed percentage reservation this might lead to scale up proposer making less aggressive cache scale-up and therefore leading to worse performance. In this case we should tune to more slack reservation percentage . Differential Revision: D61576911
levythu
force-pushed
the
export-D61576911
branch
from
1549482 to
5a3d1ae
Compare
|
This pull request was exported from Phabricator. Differential Revision: D61576911 |
levythu
added a commit
to levythu/FBGEMM
that referenced
this pull request
Sep 11, 2024
Summary: X-link: pytorch/torchrec#2380 X-link: facebookresearch/FBGEMM#203 Pull Request resolved: pytorch#3114 This PR improve sparse HBM cost by accounting the size of auxilirary state for maintaining UVM cache. As noted in the comment of split_table_batched_embeddings_ops_training, for now the significant space is `4 * table_height + 8 * cache_height + 8 * cache_height`. This is becoming more nontrivial if we have a table with many rows but few dimensions. Impact: - Not UVM-offloaded job: NoOp - UVM-offloaded job: More balanced memory usage from precise estimation, but for existing UVM jobs with scale up proposer + fixed percentage reservation this might lead to scale up proposer making less aggressive cache scale-up and therefore leading to worse performance. In this case we should tune to more slack reservation percentage . Differential Revision: D61576911
levythu
force-pushed
the
export-D61576911
branch
from
5a3d1ae to
1c3e63f
Compare
levythu
added a commit
to levythu/torchrec
that referenced
this pull request
Sep 11, 2024
Summary: Pull Request resolved: pytorch#2380 X-link: facebookresearch/FBGEMM#203 X-link: pytorch/FBGEMM#3114 This PR improve sparse HBM cost by accounting the size of auxilirary state for maintaining UVM cache. As noted in the comment of split_table_batched_embeddings_ops_training, for now the significant space is `4 * table_height + 8 * cache_height + 8 * cache_height`. This is becoming more nontrivial if we have a table with many rows but few dimensions. Impact: - Not UVM-offloaded job: NoOp - UVM-offloaded job: More balanced memory usage from precise estimation, but for existing UVM jobs with scale up proposer + fixed percentage reservation this might lead to scale up proposer making less aggressive cache scale-up and therefore leading to worse performance. In this case we should tune to more slack reservation percentage . Differential Revision: D61576911
|
This pull request was exported from Phabricator. Differential Revision: D61576911 |
levythu
added a commit
to levythu/FBGEMM
that referenced
this pull request
Sep 12, 2024
Summary: X-link: pytorch/torchrec#2380 X-link: facebookresearch/FBGEMM#203 Pull Request resolved: pytorch#3114 This PR improve sparse HBM cost by accounting the size of auxilirary state for maintaining UVM cache. As noted in the comment of split_table_batched_embeddings_ops_training, for now the significant space is `4 * hash_size + 8 * cache_slot_size + 8 * cache_slot_size`. This is becoming more nontrivial if we have a table with many rows but few dimensions. Impact: - Not UVM-offloaded job: NoOp - UVM-offloaded job: More balanced memory usage from precise estimation, but for existing UVM jobs with scale up proposer + fixed percentage reservation this might lead to scale up proposer making less aggressive cache scale-up and therefore leading to worse performance. In this case we should tune to more slack reservation percentage . Reviewed By: sarckk Differential Revision: D61576911
levythu
force-pushed
the
export-D61576911
branch
from
1c3e63f to
f73cc5e
Compare
levythu
added a commit
to levythu/torchrec
that referenced
this pull request
Sep 12, 2024
Summary: Pull Request resolved: pytorch#2380 X-link: facebookresearch/FBGEMM#203 X-link: pytorch/FBGEMM#3114 This PR improve sparse HBM cost by accounting the size of auxilirary state for maintaining UVM cache. As noted in the comment of split_table_batched_embeddings_ops_training, for now the significant space is `4 * hash_size + 8 * cache_slot_size + 8 * cache_slot_size`. This is becoming more nontrivial if we have a table with many rows but few dimensions. Impact: - Not UVM-offloaded job: NoOp - UVM-offloaded job: More balanced memory usage from precise estimation, but for existing UVM jobs with scale up proposer + fixed percentage reservation this might lead to scale up proposer making less aggressive cache scale-up and therefore leading to worse performance. In this case we should tune to more slack reservation percentage . Reviewed By: sarckk Differential Revision: D61576911
|
This pull request was exported from Phabricator. Differential Revision: D61576911 |
levythu
added a commit
to levythu/FBGEMM
that referenced
this pull request
Sep 12, 2024
Summary: X-link: pytorch/torchrec#2380 X-link: facebookresearch/FBGEMM#203 Pull Request resolved: pytorch#3114 This PR improve sparse HBM cost by accounting the size of auxilirary state for maintaining UVM cache. As noted in the comment of split_table_batched_embeddings_ops_training, for now the significant space is `4 * hash_size + 8 * cache_slot_size + 8 * cache_slot_size`. This is becoming more nontrivial if we have a table with many rows but few dimensions. Impact: - Not UVM-offloaded job: NoOp - UVM-offloaded job: More balanced memory usage from precise estimation, but for existing UVM jobs with scale up proposer + fixed percentage reservation this might lead to scale up proposer making less aggressive cache scale-up and therefore leading to worse performance. In this case we should tune to more slack reservation percentage . Reviewed By: sarckk Differential Revision: D61576911
levythu
force-pushed
the
export-D61576911
branch
from
f73cc5e to
38fee75
Compare
levythu
added a commit
to levythu/torchrec
that referenced
this pull request
Sep 12, 2024
Summary: Pull Request resolved: pytorch#2380 X-link: facebookresearch/FBGEMM#203 X-link: pytorch/FBGEMM#3114 This PR improve sparse HBM cost by accounting the size of auxilirary state for maintaining UVM cache. As noted in the comment of split_table_batched_embeddings_ops_training, for now the significant space is `4 * hash_size + 8 * cache_slot_size + 8 * cache_slot_size`. This is becoming more nontrivial if we have a table with many rows but few dimensions. Impact: - Not UVM-offloaded job: NoOp - UVM-offloaded job: More balanced memory usage from precise estimation, but for existing UVM jobs with scale up proposer + fixed percentage reservation this might lead to scale up proposer making less aggressive cache scale-up and therefore leading to worse performance. In this case we should tune to more slack reservation percentage . Reviewed By: sarckk Differential Revision: D61576911
levythu
added a commit
to levythu/FBGEMM
that referenced
this pull request
Sep 12, 2024
Summary: X-link: pytorch/torchrec#2380 X-link: facebookresearch/FBGEMM#203 Pull Request resolved: pytorch#3114 This PR improve sparse HBM cost by accounting the size of auxilirary state for maintaining UVM cache. As noted in the comment of split_table_batched_embeddings_ops_training, for now the significant space is `4 * hash_size + 8 * cache_slot_size + 8 * cache_slot_size`. This is becoming more nontrivial if we have a table with many rows but few dimensions. Impact: - Not UVM-offloaded job: NoOp - UVM-offloaded job: More balanced memory usage from precise estimation, but for existing UVM jobs with scale up proposer + fixed percentage reservation this might lead to scale up proposer making less aggressive cache scale-up and therefore leading to worse performance. In this case we should tune to more slack reservation percentage . Reviewed By: sarckk Differential Revision: D61576911
levythu
force-pushed
the
export-D61576911
branch
from
38fee75 to
c54e31b
Compare
|
This pull request was exported from Phabricator. Differential Revision: D61576911 |
levythu
added a commit
to levythu/torchrec
that referenced
this pull request
Sep 12, 2024
Summary: Pull Request resolved: pytorch#2380 X-link: facebookresearch/FBGEMM#203 X-link: pytorch/FBGEMM#3114 This PR improve sparse HBM cost by accounting the size of auxilirary state for maintaining UVM cache. As noted in the comment of split_table_batched_embeddings_ops_training, for now the significant space is `4 * hash_size + 8 * cache_slot_size + 8 * cache_slot_size`. This is becoming more nontrivial if we have a table with many rows but few dimensions. Impact: - Not UVM-offloaded job: NoOp - UVM-offloaded job: More balanced memory usage from precise estimation, but for existing UVM jobs with scale up proposer + fixed percentage reservation this might lead to scale up proposer making less aggressive cache scale-up and therefore leading to worse performance. In this case we should tune to more slack reservation percentage . Reviewed By: sarckk Differential Revision: D61576911
Summary: X-link: pytorch/torchrec#2380 X-link: facebookresearch/FBGEMM#203 Pull Request resolved: pytorch#3114 This PR improve sparse HBM cost by accounting the size of auxilirary state for maintaining UVM cache. As noted in the comment of split_table_batched_embeddings_ops_training, for now the significant space is `4 * hash_size + 8 * cache_slot_size + 8 * cache_slot_size`. This is becoming more nontrivial if we have a table with many rows but few dimensions. Impact: - Not UVM-offloaded job: NoOp - UVM-offloaded job: More balanced memory usage from precise estimation, but for existing UVM jobs with scale up proposer + fixed percentage reservation this might lead to scale up proposer making less aggressive cache scale-up and therefore leading to worse performance. In this case we should tune to more slack reservation percentage . Reviewed By: sarckk Differential Revision: D61576911
levythu
added a commit
to levythu/torchrec
that referenced
this pull request
Sep 12, 2024
Summary: Pull Request resolved: pytorch#2380 X-link: facebookresearch/FBGEMM#203 X-link: pytorch/FBGEMM#3114 This PR improve sparse HBM cost by accounting the size of auxilirary state for maintaining UVM cache. As noted in the comment of split_table_batched_embeddings_ops_training, for now the significant space is `4 * hash_size + 8 * cache_slot_size + 8 * cache_slot_size`. This is becoming more nontrivial if we have a table with many rows but few dimensions. Impact: - Not UVM-offloaded job: NoOp - UVM-offloaded job: More balanced memory usage from precise estimation, but for existing UVM jobs with scale up proposer + fixed percentage reservation this might lead to scale up proposer making less aggressive cache scale-up and therefore leading to worse performance. In this case we should tune to more slack reservation percentage . Reviewed By: sarckk Differential Revision: D61576911
|
This pull request was exported from Phabricator. Differential Revision: D61576911 |
levythu
force-pushed
the
export-D61576911
branch
from
c54e31b to
bc84822
Compare
facebook-github-bot
pushed a commit
to pytorch/torchrec
that referenced
this pull request
Sep 12, 2024
Summary: Pull Request resolved: #2380 X-link: facebookresearch/FBGEMM#203 X-link: pytorch/FBGEMM#3114 This PR improve sparse HBM cost by accounting the size of auxilirary state for maintaining UVM cache. As noted in the comment of split_table_batched_embeddings_ops_training, for now the significant space is `4 * hash_size + 8 * cache_slot_size + 8 * cache_slot_size`. This is becoming more nontrivial if we have a table with many rows but few dimensions. Impact: - Not UVM-offloaded job: NoOp - UVM-offloaded job: More balanced memory usage from precise estimation, but for existing UVM jobs with scale up proposer + fixed percentage reservation this might lead to scale up proposer making less aggressive cache scale-up and therefore leading to worse performance. In this case we should tune to more slack reservation percentage . Reviewed By: sarckk Differential Revision: D61576911 fbshipit-source-id: 6b501dc63cbe86c5274661b1d985af6a7a0a87c6
|
This pull request has been merged in 6a349df. |
q10
pushed a commit
to q10/FBGEMM
that referenced
this pull request
Apr 10, 2025
Summary: X-link: pytorch/torchrec#2380 Pull Request resolved: facebookresearch/FBGEMM#203 X-link: pytorch#3114 This PR improve sparse HBM cost by accounting the size of auxilirary state for maintaining UVM cache. As noted in the comment of split_table_batched_embeddings_ops_training, for now the significant space is `4 * hash_size + 8 * cache_slot_size + 8 * cache_slot_size`. This is becoming more nontrivial if we have a table with many rows but few dimensions. Impact: - Not UVM-offloaded job: NoOp - UVM-offloaded job: More balanced memory usage from precise estimation, but for existing UVM jobs with scale up proposer + fixed percentage reservation this might lead to scale up proposer making less aggressive cache scale-up and therefore leading to worse performance. In this case we should tune to more slack reservation percentage . Reviewed By: sarckk Differential Revision: D61576911 fbshipit-source-id: 6b501dc63cbe86c5274661b1d985af6a7a0a87c6
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
This PR improve sparse HBM cost by accounting the size of auxilirary state for maintaining UVM cache. As noted in the comment of split_table_batched_embeddings_ops_training, for now the significant space is
4 * table_height + 8 * cache_height + 8 * cache_height. This is becoming more nontrivial if we have a table with many rows but few dimensions.Impact:
Differential Revision: D61576911