activation function in BERTIntermediate

[lukovnikov]

BERTConfig is not used for BERTIntermediate's activation function. intermediate_act_fn is always gelu. Is this normal?

https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/modeling.py#L240

[thomwolf]

Yes, I hard coded that since the pre-trained models are all trained with gelu anyway.

[lukovnikov]

ok. but since config is there anyway, isn't it cleaner to use it (to avoid errors for people using configs that use a different activation for some reason) ?

[thomwolf]

Yes we can, I'll change that in the coming first release (unless you would like to submit a PR which I would be happy to merge).

[lukovnikov]

yeah let me clean up and I'll PR