Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Deepspeed] add support for bf16 mode #14569

Merged
merged 34 commits into from
Mar 12, 2022
Merged
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
e449eaf
[WIP] add support for bf16 mode
stas00 Nov 30, 2021
ac337ed
Merge remote-tracking branch 'origin/master' into ds-bf16
stas00 Dec 5, 2021
758ed4c
prep for bf16
stas00 Dec 5, 2021
013e44d
prep for bf16
stas00 Dec 5, 2021
4cba76b
fix; zero2/bf16 is ok
stas00 Dec 5, 2021
85b16d1
check bf16 is available
stas00 Dec 5, 2021
4d3694c
Merge remote-tracking branch 'origin/master' into ds-bf16
stas00 Dec 9, 2021
e8b69a6
test fixes
stas00 Dec 9, 2021
087ba85
enable zero3_bf16
stas00 Dec 9, 2021
a9c1721
config files
stas00 Dec 9, 2021
967f383
docs
stas00 Dec 9, 2021
c9e16e6
split stage_dtype; merge back to non-dtype-specific config file
stas00 Dec 9, 2021
3dec8fd
fix doc
stas00 Dec 9, 2021
ac138b8
cleanup
stas00 Dec 9, 2021
8b516e3
cleanup
stas00 Dec 9, 2021
a403c31
Merge remote-tracking branch 'origin/master' into ds-bf16
stas00 Dec 10, 2021
dd9b8b9
merge/port to mdx
stas00 Jan 19, 2022
d48f68e
bfloat16 => bf16 to match the PR changes
stas00 Jan 19, 2022
0fca057
s/zero_gather_fp16_weights_on_model_save/zero_gather_16bit_weights_on…
stas00 Jan 19, 2022
41176e4
test fixes/skipping
stas00 Jan 19, 2022
f0d53c8
Merge remote-tracking branch 'origin/master' into ds-bf16
stas00 Jan 28, 2022
5a63855
Merge remote-tracking branch 'origin/master' into ds-bf16
stas00 Jan 28, 2022
9c20180
move
stas00 Feb 11, 2022
1f93706
Merge remote-tracking branch 'origin/master' into ds-bf16
stas00 Feb 11, 2022
9f50e90
fix
stas00 Feb 11, 2022
4ea8cab
Update docs/source/main_classes/deepspeed.mdx
stas00 Feb 11, 2022
b8df847
backticks
stas00 Feb 12, 2022
7062bb2
cleanup
stas00 Feb 12, 2022
31ce3fd
Merge branch 'ds-bf16' of github.com:stas00/transformers into ds-bf16
stas00 Feb 12, 2022
fcc40d3
cleanup
stas00 Feb 12, 2022
8529f87
cleanup
stas00 Feb 12, 2022
54b6c46
Merge remote-tracking branch 'origin/master' into ds-bf16
stas00 Mar 12, 2022
7cb6455
new version
stas00 Mar 12, 2022
75fc6ce
add note about grad accum in bf16
stas00 Mar 12, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
cleanup
  • Loading branch information
stas00 committed Dec 9, 2021
commit 8b516e3797ff06db3ffbae5a40017b1074b2640d
9 changes: 4 additions & 5 deletions docs/source/main_classes/deepspeed.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1278,11 +1278,10 @@ Automatic Mixed Precision

You can use automatic mixed precision with either a pytorch-like AMP way or the apex-like way:

fp16 / float16
fp16
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


To configure pytorch AMP-like mode set:
To configure pytorch AMP-like mode with fp16 (float16) set:

.. code-block:: json

Expand Down Expand Up @@ -1322,10 +1321,10 @@ configuration.

Here is the `documentation <https://www.deepspeed.ai/docs/config-json/#fp16-training-options>`__.

bf16 / bfloat16
bf16
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

If bf16 is desired instead of fp16 then the following configuration section is to be used:
If bf16 (bfloat16) is desired instead of fp16 then the following configuration section is to be used:

.. code-block:: json

Expand Down