-
Notifications
You must be signed in to change notification settings - Fork 26.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Workaround for #27758 to avoid ZeroDivisionError #28756
Workaround for #27758 to avoid ZeroDivisionError #28756
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM thanks for opening the PR 🤗
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just missing a small test imo to make sure this does not break anything! 🤗
@ArthurZucker sounds good, I'll look into adding a test. If you know of any existing tests that I should look at, LMK! |
this one seems relevant https://github.com/huggingface/transformers/blob/main/tests/trainer/test_trainer.py#L835 |
Hey! If you can rebase we can merge as is it's alright |
Yes will do
…On Thu, Feb 29, 2024 at 11:12 AM Arthur ***@***.***> wrote:
Hey! If you can rebase we can merge as is it's alright
—
Reply to this email directly, view it on GitHub
<#28756 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AACIPLEOAPOMV3TF3GUSWHLYV37J5AVCNFSM6AAAAABCPO6Z5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZQG44TANRSGI>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
5fd8ca8
to
0f6a59f
Compare
@ArthurZucker I just rebased - can you approve the workflow? |
@ArthurZucker @pacman100 PTAL at approving workflow |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you @tleyden for the fix!
Sure np @pacman100, but I would call it more of a workaround. At some point it's worth finding out why |
@pacman100 The github merge automation seems to be stuck .. or does it just take a while to kick in? |
Sorry for the delay just merged! |
What does this PR do?
It can speed up devloops to test with very small datasets which end up being a single batch. However, that can trigger the error described in #27758.
This PR works around it by changing the division by zero to division by a very small number. The loss metric will already be meaningless if
self.state.global_step == 0
. This PR won't change that, however it will prevent the unhelpfulZeroDivisionError
I have not written any tests yet, but would be happy to if the reviewers agree with the overall approach.
Fixes #27758
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@ArthurZucker @pacman100