Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: training strategy docs #137

Merged
merged 1 commit into from
Nov 22, 2024
Merged

fix: training strategy docs #137

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 18 additions & 21 deletions configs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ You can create your own config or use/edit one of the examples.
- [Callbacks](#callbacks)
- [Optimizer](#optimizer)
- [Scheduler](#scheduler)
- [Training Strategy](#training-strategy)
- [Exporter](#exporter)
- [`ONNX`](#onnx)
- [Blob](#blob)
Expand Down Expand Up @@ -302,7 +303,7 @@ trainer:
### Callbacks

Callbacks sections contain a list of callbacks.
More information on callbacks and a list of available ones can be found [here](../luxonis_train/callbacks/README.md)
More information on callbacks and a list of available ones can be found [here](../luxonis_train/callbacks/README.md).
Each callback is a dictionary with the following fields:

| Key | Type | Default value | Description |
Expand Down Expand Up @@ -378,33 +379,29 @@ trainer:

### Training Strategy

Defines the training strategy to be used. Currently, only the `TripleLRSGDStrategy` is supported, but more strategies will be added in the future.
Defines the training strategy to be used.
More information on training strategies and a list of available ones can be found [here](../luxonis_train/strategies/README.md).
Each training strategy is a dictionary with the following fields:

| Key | Type | Default value | Description |
| ----------------- | ------- | ----------------------- | ---------------------------------------------- |
| `name` | `str` | `"TripleLRSGDStrategy"` | Name of the training strategy |
| `warmup_epochs` | `int` | `3` | Number of epochs for the warmup phase |
| `warmup_bias_lr` | `float` | `0.1` | Learning rate for bias during the warmup phase |
| `warmup_momentum` | `float` | `0.8` | Momentum value during the warmup phase |
| `lr` | `float` | `0.02` | Initial learning rate |
| `lre` | `float` | `0.0002` | End learning rate |
| `momentum` | `float` | `0.937` | Momentum for the optimizer |
| `weight_decay` | `float` | `0.0005` | Weight decay value |
| `nesterov` | `bool` | `true` | Use Nesterov momentum or not |
| Key | Type | Default value | Description |
| -------- | ------ | ----------------------- | ----------------------------- |
| `name` | `str` | `"TripleLRSGDStrategy"` | Name of the training strategy |
| `params` | `dict` | `{}` | Parameters of the optimizer |

**Example:**

```yaml
training_strategy:
name: "TripleLRSGDStrategy"
warmup_epochs: 3
warmup_bias_lr: 0.1
warmup_momentum: 0.8
lr: 0.02
lre: 0.0002
momentum: 0.937
weight_decay: 0.0005
nesterov: true
params:
warmup_epochs: 3
warmup_bias_lr: 0.1
warmup_momentum: 0.8
lr: 0.02
lre: 0.0002
momentum: 0.937
weight_decay: 0.0005
nesterov: True
```

## Exporter
Expand Down
24 changes: 24 additions & 0 deletions luxonis_train/strategies/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Training Strategies

Training strategies define how the optimization process is orchestrated during training. Each strategy combines specific configurations and update mechanisms to achieve optimal convergence for various tasks. Strategies are flexible and customizable based on the needs of the training pipeline.

## Table Of Contents

- [`TripleLRSGDStrategy`](#triplelrsgdstrategy)

## `TripleLRSGDStrategy`

The `TripleLRSGDStrategy` implements a staged learning rate schedule with warmup and momentum adjustments, tailored for stable and efficient convergence.

**Parameters:**

| Key | Type | Default Value | Description |
| ----------------- | ------- | ------------- | ---------------------------------------------------------- |
| `warmup_epochs` | `int` | `3` | Number of epochs for the learning rate warmup phase. |
| `warmup_bias_lr` | `float` | `0.1` | Learning rate for bias parameters during the warmup phase. |
| `warmup_momentum` | `float` | `0.8` | Momentum used during the warmup phase. |
| `lr` | `float` | `0.02` | Base learning rate for the main training phase. |
| `lre` | `float` | `0.0002` | Ending learning rate after the scheduled decay. |
| `momentum` | `float` | `0.937` | Momentum factor for stable optimization. |
| `weight_decay` | `float` | `0.0005` | Weight decay (L2 penalty) for regularization. |
| `nesterov` | `bool` | `True` | Whether to use Nesterov momentum during optimization. |