Add wait_for_first_optimizer_step mode #737
Conversation
linshokaku
requested review from
asi1024,
emcastillo and
kmaehashi
as code owners
|
For record: |
| self.scheduler.optimizer._step_count < 1 | ||
| and self.wait_for_first_optimizer_step | ||
| self.wait_for_first_optimizer_step | ||
| and hasattr(self.scheduler.optimizer.step, "_with_counter") |
There was a problem hiding this comment.
A very nit: I think assuming the existence of private APIs is fragile, because this feature will silently stop working if PyTorch renames this attribute. I guess this check is not essentially needed and can be removed?
There was a problem hiding this comment.
When using the base class of PyTorch's LRScheduler, the addition of the _step_count attribute and the addition of a hook for counting are performed at the time of initialization. However, as ReduceLROnPlateau does not inherit from this base class, access to these attributes results in an error. Therefore, it is appropriate to perform an existence check for the _with_counter attribute, following the code that checks for PyTorch's lr_scheduler _step_count.
There was a problem hiding this comment.
Ah got it, thanks! I'd prefer this to be commented somewhere, but it can be done separately 😄
There was a problem hiding this comment.
I think this may be a very detailed matter and not essential to the functionality of the code, would it be possible to include the link to the comment made on this PR as a comment in the code?
|
/test mini |
We will add a mode to delay the execution of scheduler.step until optimizer.step is run. This is to prevent a PyTorch warning that can occur when scheduler.step is executed before optimizer.step, a situation that can arise when GradScaler does not execute optimizer.step on the first iteration.