Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove warnings from autodoc and sphinx #6788

Merged
merged 11 commits into from
Dec 13, 2024
Merged
4 changes: 3 additions & 1 deletion deepspeed/runtime/fp16/onebit/zoadam.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,11 @@


class ZeroOneAdam(torch.optim.Optimizer):
"""Implements the 0/1 Adam algorithm. Currently GPU-only.
"""
Implements the 0/1 Adam algorithm. Currently GPU-only.
For usage example please see https://www.deepspeed.ai/tutorials/zero-one-adam/
For technical details please read https://arxiv.org/abs/2202.06009

Arguments:
params (iterable): iterable of parameters to optimize or dicts defining
parameter groups.
Expand Down
4 changes: 2 additions & 2 deletions deepspeed/runtime/lr_schedules.py
Original file line number Diff line number Diff line change
Expand Up @@ -274,7 +274,7 @@ class LRRangeTest(object):
"""Sets the learning rate of each parameter group according to
learning rate range test (LRRT) policy. The policy increases learning
rate starting from a base value with a constant frequency, as detailed in
the paper `A disciplined approach to neural network hyper-parameters: Part1`_.
the paper `A disciplined approach to neural network hyper-parameters: Part 1 <https://arxiv.org/abs/1803.09820>`_

LRRT policy is used for finding maximum LR that trains a model without divergence, and can be used to
configure the LR boundaries for Cyclic LR schedules.
Expand Down Expand Up @@ -379,7 +379,7 @@ class OneCycle(object):
1CLR policy changes the learning rate after every batch.
`step` should be called after a batch has been used for training.

This implementation was adapted from the github repo: `pytorch/pytorch`_
This implementation was adapted from the github repo: `PyTorch <https://github.com/pytorch/pytorch>`_.

Args:
optimizer (Optimizer): Wrapped optimizer.
Expand Down
18 changes: 9 additions & 9 deletions docs/code-docs/source/monitor.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ overview of what DeepSpeed will log automatically.
:header: "Field", "Description", "Condition"
:widths: 20, 20, 10

`Train/Samples/train_loss`,The training loss.,None
`Train/Samples/lr`,The learning rate during training.,None
`Train/Samples/loss_scale`,The loss scale when training using `fp16`.,`fp16` must be enabled.
`Train/Eigenvalues/ModelBlockParam_{i}`,Eigen values per param block.,`eigenvalue` must be enabled.
`Train/Samples/elapsed_time_ms_forward`,The global duration of the forward pass.,`flops_profiler.enabled` or `wall_clock_breakdown`.
`Train/Samples/elapsed_time_ms_backward`,The global duration of the forward pass.,`flops_profiler.enabled` or `wall_clock_breakdown`.
`Train/Samples/elapsed_time_ms_backward_inner`,The backward time that does not include the gradient reduction time. Only in cases where the gradient reduction is not overlapped, if it is overlapped then the inner time should be about the same as the entire backward time.,`flops_profiler.enabled` or `wall_clock_breakdown`.
`Train/Samples/elapsed_time_ms_backward_allreduce`,The global duration of the allreduce operation.,`flops_profiler.enabled` or `wall_clock_breakdown`.
`Train/Samples/elapsed_time_ms_step`,The optimizer step time,`flops_profiler.enabled` or `wall_clock_breakdown`.
`Train/Samples/train_loss`,"The training loss.",None
`Train/Samples/lr`,"The learning rate during training.",None
`Train/Samples/loss_scale`,"The loss scale when training using `fp16`.",`fp16` must be enabled.
`Train/Eigenvalues/ModelBlockParam_{i}`,"Eigen values per param block.",`eigenvalue` must be enabled.
`Train/Samples/elapsed_time_ms_forward`,"The global duration of the forward pass.",`flops_profiler.enabled` or `wall_clock_breakdown`.
`Train/Samples/elapsed_time_ms_backward`,"The global duration of the forward pass.",`flops_profiler.enabled` or `wall_clock_breakdown`.
`Train/Samples/elapsed_time_ms_backward_inner`,"The backward time that does not include the gradient reduction time. Only in cases where the gradient reduction is not overlapped, if it is overlapped then the inner time should be about the same as the entire backward time.",`flops_profiler.enabled` or `wall_clock_breakdown`.
`Train/Samples/elapsed_time_ms_backward_allreduce`,"The global duration of the allreduce operation.",`flops_profiler.enabled` or `wall_clock_breakdown`.
`Train/Samples/elapsed_time_ms_step`,"The optimizer step time.",`flops_profiler.enabled` or `wall_clock_breakdown`.

TensorBoard
-----------
Expand Down
Loading