-
Notifications
You must be signed in to change notification settings - Fork 26.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Trainer] Add nan/inf logging filter #13619
[Trainer] Add nan/inf logging filter #13619
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for adding this! I wonder if the default of the new feature shouldn't be True? It's a little bi breaking, but only for the logging, not the actual training, and it makes more sense to ignore thoses NaN/infs (which are ignored by default by the optimizer since there is some gradient clipping).
src/transformers/training_args.py
Outdated
"Filter nan and inf losses for logging. " | ||
"Note this flag only affects the logging output and not the optimization step." |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No need to be this long here, but there should be a proper documentation above that can be longer!
Happy to let it default to True! |
@@ -468,6 +478,7 @@ class TrainingArguments: | |||
) | |||
logging_first_step: bool = field(default=False, metadata={"help": "Log the first global_step"}) | |||
logging_steps: int = field(default=500, metadata={"help": "Log every X updates steps."}) | |||
logging_nan_inf_filter: str = field(default=True, metadata={"help": "Filter nan and inf losses for logging."}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The default is now True
* finish * add test * push * remove unnecessary code * up * correct test * Update src/transformers/training_args.py
* finish * add test * push * remove unnecessary code * up * correct test * Update src/transformers/training_args.py
What does this PR do?
Some losses sometimes produce
inf
losses which however doesn't necessarily mean that the training went bad. When using the CTC loss for speech recognition - see: #13620 this is often the case. The problem is that as soon as one single loss step isinf
ornan
the rest of the training logs will displayinf
ornan
. In this PR a flag is added that allows the user to filter outnan
andinf
values for training. It defaults toFalse
and will be set toTrue
in all CTC training scripts.Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.