Fix ineffective no_decay bug when using BERTAdam

[xiaoda99]

With the original code, all parameters are decayed because the condition "parameter_name in no_decay" will never be satisfied.

[thomwolf]

thanks!

[xelibrion]

Question - wouldn't .named_parameters() for the model return a tuple (name, param_tensor), where name looks similar to these

['bert.embeddings.word_embeddings.weight',
 'bert.embeddings.position_embeddings.weight',
 'bert.embeddings.token_type_embeddings.weight',
 'bert.embeddings.LayerNorm.weight',
 'bert.embeddings.LayerNorm.bias',
 'bert.encoder.layer.0.attention.self.query.weight',
 'bert.encoder.layer.0.attention.self.query.bias',
 'bert.encoder.layer.0.attention.self.key.weight',
 'bert.encoder.layer.0.attention.self.key.bias',
 'bert.encoder.layer.0.attention.self.value.weight',
 'bert.encoder.layer.0.attention.self.value.bias',
 'bert.encoder.layer.0.attention.output.dense.weight',
 'bert.encoder.layer.0.attention.output.dense.bias',
 'bert.encoder.layer.0.attention.output.LayerNorm.weight',
 'bert.encoder.layer.0.attention.output.LayerNorm.bias',
...
...
'classifier.linear.weight',
'classifier.linear.bias']

therefore requiring slightly smarter conditions than just in? Something along the lines?

[p for n, p in param_optimizer if any(True for x in no_decay if n.endswith(x))]

[artemisart]

I think all(nd not in n for nd in no_decay) would be clearer

[xelibrion]

Don't mind my comment, tested it further this morning and everything seems to work as expected!