Workaround for #27758 to avoid ZeroDivisionError

[tleyden]

What does this PR do?

It can speed up devloops to test with very small datasets which end up being a single batch. However, that can trigger the error described in #27758.

This PR works around it by changing the division by zero to division by a very small number. The loss metric will already be meaningless if self.state.global_step == 0. This PR won't change that, however it will prevent the unhelpful ZeroDivisionError

I have not written any tests yet, but would be happy to if the reviewers agree with the overall approach.

Fixes #27758

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@ArthurZucker @pacman100

[ArthurZucker]

LGTM thanks for opening the PR 🤗

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

just missing a small test imo to make sure this does not break anything! 🤗

[tleyden]

@ArthurZucker sounds good, I'll look into adding a test. If you know of any existing tests that I should look at, LMK!

[ArthurZucker]

this one seems relevant https://github.com/huggingface/transformers/blob/main/tests/trainer/test_trainer.py#L835

[ArthurZucker]

Hey! If you can rebase we can merge as is it's alright

[tleyden]

Yes will do

…

On Thu, Feb 29, 2024 at 11:12 AM Arthur ***@***.***> wrote: Hey! If you can rebase we can merge as is it's alright — Reply to this email directly, view it on GitHub <#28756 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACIPLEOAPOMV3TF3GUSWHLYV37J5AVCNFSM6AAAAABCPO6Z5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZQG44TANRSGI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[tleyden]

@ArthurZucker I just rebased - can you approve the workflow?

[tleyden]

@ArthurZucker @pacman100 PTAL at approving workflow

[pacman100]

Thank you @tleyden for the fix!

[tleyden]

Sure np @pacman100, but I would call it more of a workaround. At some point it's worth finding out why self.state.global_step can end up as 0 in the first place

[tleyden]

@pacman100 The github merge automation seems to be stuck .. or does it just take a while to kick in?

[ArthurZucker]

Sorry for the delay just merged!