[feat] add trainer factory #489
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
SamitHuang
reviewed
Mar 7, 2023
Songyuanwei
reviewed
Mar 8, 2023
| loss_scale_value=loss_scale, scale_factor=2, scale_window=2000 | ||
| ) | ||
| else: | ||
| raise ValueError(f"Loss scale type only support ['fixed', 'dynamic'], but got{loss_scale_type}.") |
There was a problem hiding this comment.
这里的loss scale type和上面的不一样，不支持auto，用户会产生歧义吗？
There was a problem hiding this comment.
同样需要自定义的TrainStep支持auto
Songyuanwei
reviewed
Mar 8, 2023
| # instead of cell, and TrainStep should be TrainOneStepCell. If drop_overflow_update is True, | ||
| # scale_sense should be FixedLossScaleUpdateCell, and TrainStep should be TrainOneStepWithLossScaleCell. | ||
| train_step_kwargs["scale_sense"] = nn.FixedLossScaleUpdateCell(loss_scale_value=loss_scale) | ||
| elif loss_scale_type.lower() == "dynamic": |
There was a problem hiding this comment.
train_step中目前仅有不检测溢出的情况，如果设置成dynamic loss scale， train_step中应该要检测溢出，但是目前中代码没有检测溢出。
There was a problem hiding this comment.
后面加了loss scale的限制
Songyuanwei
approved these changes
Mar 8, 2023
4 tasks
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Thank you for your contribution to the MindCV repo.
Before submitting this PR, please make sure:
Motivation
We abstracted the fragment of the training script about creating a
mindspore.Model(actually a Trainer) into the functioncreate_trainer, a factory method of creating trainers. We believe this abstraction facilitates code readability.When creating the trainer we need to pass in common components such as networks, optimizer, loss functions etc. as input parameters. Meanwhile, we also need to pass in additional parameters to support Auto Mixed Precision(AMP). Next, we elaborate on the creator's design and principles regarding AMP in details.
The level of amp:
We follow the definition from MindSpore
We may subsequently consider customized black and white lists
The type of loss scale:
For
fixedordynamic, we will explicitly construct theLossScaleManagerand pass inmindspore.Model. Forauto, we will not actively construct theLossScaleManager, butmindspore.Modelmay use theLossScaleManagersilently, see mindspore.train.amp for details.The value of loss scale
We will raise an error if the value of loss scale is less than 1. We no longer make a special case for
loss_scale=1, although for a while we did. You can now set the type of loss scale toautoto achieve the same effect.We have also considered support for customized
TrainStep. The current customizedTrainStepsupports EMA and gradient clipping. Note: the current customizedTrainStepcan only be used with fixed loss scale without dropping overflow!Test Plan
st is already in
tests/tasks/test_train_val_imagenet_subset.py. Do we need an additional unit test?Related Issues and PRs
Nope