This repository was archived by the owner on Aug 7, 2024. It is now read-only.
switch from just-in-time scaling to delayed scaling #18
+92
−29
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: Before: all scaling was done just-in-time After: 1. scaling is done in a delayed fashion with a history of 1 2. there is special logic to populate initial amaxes (TE doesn't have this) A future PR will add windowed calculation Test Plan: ``` with-proxy ./tests/test_everything.sh ``` Reviewers: Subscribers: Tasks: Tags:
facebook-github-bot
added
the
CLA Signed
vkuzo
added a commit
that referenced
this pull request
Aug 14, 2023
Summary: In #18, the MNIST finetuning script broke because the casts were not saturated. By default, casts to float8 are not saturated. With delayed scaling, we need to saturate to avoid `NaN`s everywhere. For now, write the saturation logic in eager mode. In the future we would ideally lower this to a hardware accelerated saturated cast via PT2.0. Test Plan: ``` // loss now converges, again with-proxy python finetune/mnist.py --batch-size 4096 --use-pt-fp8 ``` Reviewers: Subscribers: Tasks: Tags:
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Before: all scaling was done just-in-time
After:
A future PR will add windowed calculation
Test Plan:
Reviewers:
Subscribers:
Tasks:
Tags: