documenting load_from_fp32_weights config parameter (#4449)

Co-authored-by: Alexander Jipa <azzhipa@amazon.com> Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
microsoft · Oct 4, 2023 · 9c22801 · 9c22801
1 parent 7099f99
commit 9c22801
Showing 1 changed file with 6 additions and 0 deletions.
diff --git a/docs/_pages/config-json.md b/docs/_pages/config-json.md
@@ -427,6 +427,12 @@ Enabling and configuring ZeRO memory optimizations
 | ------------------------------------------------------------------------------------------------------------------- | ------- |
 | Copies the gradients to a contiguous buffer as they are produced. Avoids memory fragmentation during backward pass. | `True`  |
 
+<i>**load_from_fp32_weights**</i>: [boolean]
+
+| Description                                                                                                                                                                                                                          | Default |
+| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- |
+| Initialize fp32 master weights from fp32 copies in checkpoint (no precision loss) or from model's fp16 copies (with precision loss). This can be used to initialize optimizer state even when checkpoint is missing optimizer state. | `True`  |
+
 <i>**grad_hooks**</i>: [boolean]
 
 | Description                                                                                                                               | Default |