|
| 1 | +# How to use Optimizer |
| 2 | + |
| 3 | +## What is Optimizer? |
| 4 | +In the context of machine learning and deep learning, |
| 5 | +an optimizer is an algorithm or method used to update the parameters of a model in order to minimize the error between the predicted output and the actual output. |
| 6 | + |
| 7 | +The goal of an optimizer is to find the optimal set of parameters that can achieve the best performance on a given task. |
| 8 | +This process is typically performed during the training phase of a machine learning model. |
| 9 | + |
| 10 | +Optimizers work by computing the gradients of the loss function with respect to the model parameters, |
| 11 | +and using this information to update the parameters in the direction that reduces the loss. |
| 12 | +There are various optimization algorithms available, |
| 13 | +such as stochastic gradient descent (SGD), Adagrad, Adam, RMSprop, and more, each with their own advantages and disadvantages. |
| 14 | + |
| 15 | +The choice of optimizer depends on the specific problem, the size of the dataset, |
| 16 | +the complexity of the model, and other factors. |
| 17 | +A good optimizer can significantly improve the training speed and accuracy of a model. |
| 18 | + |
| 19 | + |
| 20 | + |
| 21 | + |
| 22 | +## Loading optimizer |
| 23 | + |
| 24 | +### dependencies |
| 25 | +#### adan |
| 26 | +``` |
| 27 | +python3 -m pip install git+https://github.com/sail-sg/Adan.git |
| 28 | +``` |
| 29 | +#### lion |
| 30 | +``` |
| 31 | +$ pip install lion-pytorch |
| 32 | +``` |
| 33 | +#### lamb |
| 34 | +``` |
| 35 | +$ pip install torch_optimizer |
| 36 | +``` |
| 37 | +#### example |
| 38 | +```python |
| 39 | +>>> # currently FlagAI support adam, adamw, lion, adan, adafactor and lamb, which can be defined by setting optimizer_type when defining Trainer |
| 40 | +>>> trainer = Trainer(env_type='pytorch', |
| 41 | +>>> epochs=1, |
| 42 | +>>> batch_size=2, |
| 43 | +>>> eval_interval=100, |
| 44 | +>>> log_interval=10, |
| 45 | +>>> experiment_name='glm_large_bmtrain', |
| 46 | +>>> pytorch_device='cuda', |
| 47 | +>>> load_dir=None, |
| 48 | +>>> lr=1e-4, |
| 49 | +>>> num_gpus = 1, |
| 50 | +>>> weight_decay=1e-2, |
| 51 | +>>> save_interval=1000, |
| 52 | +>>> hostfile='./hostfile', |
| 53 | +>>> training_script=__file__, |
| 54 | +>>> deepspeed_config='./deepspeed.json', |
| 55 | +>>> optimizer_type='lion') #load optimizer |
| 56 | +``` |
| 57 | + |
0 commit comments