Enable other tunable parameters for create_optimizer in optimization.py

Prerequisites

I checked to make sure that this feature has not been requested already.

1. The entire URL of the file you are using

Lines 68 to 107 in 7ecbac3

    
           def create_optimizer(init_lr, 
        
                                num_train_steps, 
        
                                num_warmup_steps, 
        
                                end_lr=0.0, 
        
                                optimizer_type='adamw', 
        
                                beta_1=0.9): 
        
             """Creates an optimizer with learning rate schedule.""" 
        
             # Implements linear decay of the learning rate. 
        
             lr_schedule = tf.keras.optimizers.schedules.PolynomialDecay( 
        
                 initial_learning_rate=init_lr, 
        
                 decay_steps=num_train_steps, 
        
                 end_learning_rate=end_lr) 
        
             if num_warmup_steps: 
        
               lr_schedule = WarmUp( 
        
                   initial_learning_rate=init_lr, 
        
                   decay_schedule_fn=lr_schedule, 
        
                   warmup_steps=num_warmup_steps) 
        
             if optimizer_type == 'adamw': 
        
               logging.info('using Adamw optimizer') 
        
               optimizer = AdamWeightDecay( 
        
                   learning_rate=lr_schedule, 
        
                   weight_decay_rate=0.01, 
        
                   beta_1=beta_1, 
        
                   beta_2=0.999, 
        
                   epsilon=1e-6, 
        
                   exclude_from_weight_decay=['LayerNorm', 'layer_norm', 'bias']) 
        
             elif optimizer_type == 'lamb': 
        
               logging.info('using Lamb optimizer') 
        
               optimizer = tfa_optimizers.LAMB( 
        
                   learning_rate=lr_schedule, 
        
                   weight_decay_rate=0.01, 
        
                   beta_1=beta_1, 
        
                   beta_2=0.999, 
        
                   epsilon=1e-6, 
        
                   exclude_from_weight_decay=['LayerNorm', 'layer_norm', 'bias']) 
        
             else: 
        
               raise ValueError('Unsupported optimizer type: ', optimizer_type) 
        
             return optimizer

2. Describe the feature you request

Similar to beta_1 as a tunable parameter for creating optimizers, we can add beta_2, epsilon, weight_decay_rate, and exclude_from_weight_decay as tunable parameters by passing them as argument from create_optimizer.

3. Additional context

I was recently trying to finetune a Huggingface Roberta model and while doing so, I wanted to add a scheduler as well as AdamW with custom parameters, and thus I came across these methods.

4. Are you willing to contribute it? (Yes or No)

Yes

Hi, we are using the configurable optimization now: https://github.com/tensorflow/models/tree/master/official/modeling/optimization
All params should be configurable now. It is not used in the legacy bert/ example.

Nice!! I was following this guide for fine-tuning TF BERT, and the official nlp optimizer was used to set up the optimizer. So, if this configurable optimization is ready, where can I find some examples for that? or can we change the above-mentioned fine-tuning example by using this configurable optimization? Let me know what you think about this.

Here is the unit test: https://github.com/tensorflow/models/blob/master/official/modeling/optimization/optimizer_factory_test.py
Here is the use case:
https://github.com/tensorflow/models/blob/master/official/core/base_task.py#L67

The bert tutorial was limited to the particular use case as a demo.
Yes, we will consider updating the colab which is a bit old already.

Thank you for the resources!! Looking forward to the update in the Colab example. Let me know if I can contribute in any way.

This issue can be closed.

Navigation Menu

Repository navigation

Enable other tunable parameters for `create_optimizer` in `optimization.py` #10143

Prerequisites

1. The entire URL of the file you are using

2. Describe the feature you request

3. Additional context

4. Are you willing to contribute it? (Yes or No)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	def create_optimizer(init_lr,
	num_train_steps,
	num_warmup_steps,
	end_lr=0.0,
	optimizer_type='adamw',
	beta_1=0.9):
	"""Creates an optimizer with learning rate schedule."""
	# Implements linear decay of the learning rate.
	lr_schedule = tf.keras.optimizers.schedules.PolynomialDecay(
	initial_learning_rate=init_lr,
	decay_steps=num_train_steps,
	end_learning_rate=end_lr)
	if num_warmup_steps:
	lr_schedule = WarmUp(
	initial_learning_rate=init_lr,
	decay_schedule_fn=lr_schedule,
	warmup_steps=num_warmup_steps)

	if optimizer_type == 'adamw':
	logging.info('using Adamw optimizer')
	optimizer = AdamWeightDecay(
	learning_rate=lr_schedule,
	weight_decay_rate=0.01,
	beta_1=beta_1,
	beta_2=0.999,
	epsilon=1e-6,
	exclude_from_weight_decay=['LayerNorm', 'layer_norm', 'bias'])
	elif optimizer_type == 'lamb':
	logging.info('using Lamb optimizer')
	optimizer = tfa_optimizers.LAMB(
	learning_rate=lr_schedule,
	weight_decay_rate=0.01,
	beta_1=beta_1,
	beta_2=0.999,
	epsilon=1e-6,
	exclude_from_weight_decay=['LayerNorm', 'layer_norm', 'bias'])
	else:
	raise ValueError('Unsupported optimizer type: ', optimizer_type)

	return optimizer

Enable other tunable parameters for create_optimizer in optimization.py #10143

Description

Prerequisites

1. The entire URL of the file you are using

2. Describe the feature you request

3. Additional context

4. Are you willing to contribute it? (Yes or No)

Activity

saberkun commented on Jul 21, 2021

01-vyom commented on Jul 21, 2021

saberkun commented on Jul 21, 2021

01-vyom commented on Jul 22, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Enable other tunable parameters for `create_optimizer` in `optimization.py` #10143