Fix: Correct loss normalization in training_step for multi-GPU training #41530
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Fix bug where mean token loss was incorrectly normalized with multiple GPUs - When num_items_in_batch is provided, per-device losses are already normalized by total tokens across all devices - Changed from .mean() to .sum() for aggregating these normalized losses - Fixes issue where reported loss was 1/n_gpu of expected value With 4 GPUs, loss was incorrectly reported as 2.7 instead of 10.8. This fix ensures consistent loss reporting across different GPU configurations. Fixes huggingface#37474
Member
|
This should already be fixed in #40799 @KaparthyReddy can you cool it down with these PRs, or at least read issues carefully and only make PRs when you understand the issue and you've checked they're not duplicates? You've made like 6 of them and they're all obviously written by Copilot, so it's very hard to tell if any of them actually do anything useful! |
Author
|
Noted. I’ll proceed as I see fit. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix: Correct loss normalization in training_step for multi-GPU training
Description
Fixes #37474
This PR corrects the loss aggregation logic in
Trainer.training_stepwhen training with multiple GPUs.Problem
When
num_items_in_batchis provided (e.g., for token-level loss normalization), each device computes loss as: