optimize VRAM for calculating pos_bias in LayoutLM v2, v3 #26139
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
|
Hey! Thanks for opening a PR, pinging @rafaelpadilla for a review here 😉 |
rafaelpadilla
approved these changes
Sep 27, 2023
Contributor
rafaelpadilla
left a comment
rafaelpadilla
left a comment
There was a problem hiding this comment.
Nice catch @NormXU !
Sorry for the delay on reviewing it. I had problems with the pytesseract dependency.
I was able to dig into the code and saw that your changes make the process simpler and produce the same outputs.
Just make sure to make the other tests pass.
Contributor
Author
|
@rafaelpadilla I've reformatted the codes. It's ready to be merged. |
LysandreJik
approved these changes
Sep 28, 2023
Member
LysandreJik
left a comment
LysandreJik
left a comment
There was a problem hiding this comment.
Thank you for your PR @NormXU and for your review @rafaelpadilla
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
ArthurZucker
pushed a commit
that referenced
this pull request
May 24, 2024
…ating pos_bias in LayoutLM v2, v3 (#26139)" (#30988) * Revert "optimize VRAM for calculating pos_bias in LayoutLM v2, v3 (#26139)" This reverts commit a7e0ed8. * Instead of reverting commit, wrap indexing in torch.no_grad context * Apply wrapping in LayoutLMv2 * Add comments explaining reason for no_grad * Fix code format --------- Co-authored-by: Kevin Koehncke <kevin.koehncke@uipath.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
The current implementation of calculating 1d_pos_bias/2d_pos_bias in LayoutLMv2, v3 is VRAM-consuming due to the large one-hot matrix.
Considering the idea of 1d_pos_bias/2d_pos_bias is to categorize all relative positions into several buckets, assign each position id to a specific bucket based on its relative distance to another token, and embed the position id into a feature, we can drop the large one-hot matrix and directly use the Linear weight features like an nn.Embedding.
In my tests, as for an input sequence of$[10, 1024]$ (bz, nseq), this can save 3 Gb VRAM for 2d_pos_bias calculations
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@ArthurZucker and @younesbelkada