Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

max_grad_norm schedule #375

Open
ffuuugor opened this issue Mar 11, 2022 · 0 comments
Open

max_grad_norm schedule #375

ffuuugor opened this issue Mar 11, 2022 · 0 comments
Labels
enhancement New feature or request

Comments

@ffuuugor
Copy link
Contributor

max_grad_norm is functionally very similar to noise_multplier - it's also stored in the optimizer field as a scalar and never updated.

We have convenient scheduling option for optimizer, but not for max_grad_norm.
I can't point to any particular paper proposing this, but some (e.g. https://arxiv.org/pdf/2108.01624.pdf) show that over time singal-to-noise ratio can decline due to natural decline in gradient norm (expected in many training scenarions)
@ffuuugor ffuuugor added the enhancement New feature or request label Mar 11, 2022
@Darktex Darktex mentioned this issue Jan 20, 2023
Closed
Darktex added a commit to Darktex/opacus that referenced this issue Jan 20, 2023
@Darktex @facebook-github-bot
Add Clipping schedulers
15d5d51 
Summary:
This diff introduces gradient clipping schedulers that can be used to vary gradient clipping throughout training.

Addresses pytorch#375 in OSS.

Differential Revision: D42644261

fbshipit-source-id: 50dc48a29254b33d581f858a09fc4e03ec9a598a
Darktex added a commit to Darktex/opacus that referenced this issue Jan 20, 2023
@Darktex @facebook-github-bot
Add Clipping schedulers (pytorch#556)
6358c20 
Summary:
Pull Request resolved: pytorch#556

This diff introduces gradient clipping schedulers that can be used to vary gradient clipping throughout training.

Addresses pytorch#375 in OSS.

Differential Revision: D42644261

fbshipit-source-id: 7480e2f81432bd4a05d58af420ee4a783bdd63f1
Darktex added a commit to Darktex/opacus that referenced this issue Jan 20, 2023
@Darktex @facebook-github-bot
Add Clipping schedulers (pytorch#556)
ec71f8b 
Summary:
Pull Request resolved: pytorch#556

This diff introduces gradient clipping schedulers that can be used to vary gradient clipping throughout training.

Addresses pytorch#375 in OSS.

Differential Revision: D42644261

fbshipit-source-id: 0af54b977d41b164531c18d4804a2506a359ce28
Darktex added a commit to Darktex/opacus that referenced this issue Jan 24, 2023
@Darktex @facebook-github-bot
Add Clipping schedulers (pytorch#556)
28cfe8c 
Summary:
Pull Request resolved: pytorch#556

This diff introduces gradient clipping schedulers that can be used to vary gradient clipping throughout training.

Addresses pytorch#375 in OSS.

Reviewed By: karthikprasad

Differential Revision: D42644261

fbshipit-source-id: 91bedb4c3dd68f336917d16cec42f939ace02406
Darktex added a commit to Darktex/opacus that referenced this issue Jan 24, 2023
@Darktex @facebook-github-bot
Add Clipping schedulers (pytorch#556)
3412d26 
Summary:
Pull Request resolved: pytorch#556

This diff introduces gradient clipping schedulers that can be used to vary gradient clipping throughout training.

Addresses pytorch#375 in OSS.

Reviewed By: karthikprasad

Differential Revision: D42644261

fbshipit-source-id: 94f5fb756dce8ec576cce2ff003c2054eb926e27
Darktex added a commit to Darktex/opacus that referenced this issue Jan 24, 2023
@Darktex @facebook-github-bot
Add Clipping schedulers (pytorch#556)
5d3ed10 
Summary:
Pull Request resolved: pytorch#556

This diff introduces gradient clipping schedulers that can be used to vary gradient clipping throughout training.

Addresses pytorch#375 in OSS.

Reviewed By: karthikprasad

Differential Revision: D42644261

fbshipit-source-id: a571569785697b369c4c6f9709d35a5d3ff7e73c
Darktex added a commit to Darktex/opacus that referenced this issue Jan 24, 2023
@Darktex @facebook-github-bot
Add Clipping schedulers (pytorch#556)
8c8e540 
Summary:
Pull Request resolved: pytorch#556

This diff introduces gradient clipping schedulers that can be used to vary gradient clipping throughout training.

Addresses pytorch#375 in OSS.

Reviewed By: karthikprasad

Differential Revision: D42644261

fbshipit-source-id: 57c87ba5f0b012359761fa015d046edc5c13da88
facebook-github-bot pushed a commit that referenced this issue Jan 24, 2023
@Darktex @facebook-github-bot
Add Clipping schedulers (#556)
d888fd0 
Summary:
Pull Request resolved: #556

This diff introduces gradient clipping schedulers that can be used to vary gradient clipping throughout training.

Addresses #375 in OSS.

Reviewed By: karthikprasad

Differential Revision: D42644261

fbshipit-source-id: 7e200d704d97d0b0f5432153af32753c1d4e6204
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ffuuugor