GPT-2 models are unpickable

[Lawiss]

🐛 Bug

Information

Model I am using (Bert, XLNet ...): GPT-2

Language I am using the model on (English, Chinese ...): English

The problem arises when using:

• the official example scripts: (give details below)
• my own modified scripts:

Hi,

I'm trying to train a GPT-2 Double Heads Model (based on your transfer-learning-conv-ai guide) using Pytorch Lightning. However I have a problem when trying to train the model on ddp distributed backend : the GPT2DoubleHeadsModel class seems to be unpickable and my training script fails with the following error :
TypeError: can't pickle torch._C.ScriptFunction objects

To reproduce

Run :

from transformers import GPT2DoubleHeadsModel
import pickle
model = GPT2DoubleHeadsModel.from_pretrained("gpt2-medium")
pickle.dump(model,open("test.bin","wb"))

The problem does not occur when using bert-base-uncased for example. I tried to search which part of GPT-2 class contains torch._C.ScriptFunction objects without success. Do you have an idea to avoid this error ?

Thanks in advance.

• transformers version: 2.8
• Platform: Ubuntu 18.04
• Python version: 3.7
• PyTorch version (GPU?): 1.5 Cuda 10.2
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: Yes, ddp

[LysandreJik]

Hi! I think this may come from the activation functions, some of which are jitted. This allows better performance.

In what situation would you use pickle rather than our API save_pretrained ?

from transformers import GPT2DoubleHeadsModel
import pickle
model = GPT2DoubleHeadsModel.from_pretrained("gpt2-medium")
model.save_pretrained("test")

[Lawiss]

Hi Lysandre,

Thanks for your answer !
I use Pytorch Lightning which requires the model to be picklable (https://pytorch-lightning.readthedocs.io/en/stable/multi_gpu.html#make-model-picklable). I see from your link that if the PyTorch version is <1.4, the activation function is not jitted :

transformers/src/transformers/activations.py

Lines 32 to 44 in fa49b9a

	if torch.__version__ < "1.4.0":
	gelu = _gelu_python
	else:
	gelu = F.gelu
	try:
	import torch_xla # noqa F401
	logger.warning(
	"The torch_xla package was detected in the python environment. PyTorch/XLA and JIT is untested,"
	" no activation function will be traced with JIT."
	)
	except ImportError:
	gelu_new = torch.jit.script(gelu_new)

I think i'll test again downgrading my PyTorch version to 1.3.1.

[LysandreJik]

I understand, this is indeed an issue. We've had other several issues due to this, I think it would be better to revert to picklable option. Will open a PR with this objective in a bit.

[Lawiss]

Thank you for your responsiveness !

[LysandreJik]

It should be fixed on master now. Could you let me know if running the code from the master branch solves your issue?

[Lawiss]

Yes, it works, thank you 👌

[paramansh]

I am getting the same error trying to save AlbertForTokenClassification() model using torch.save(model, 'temp.pt')
But torch.save(model.state_dict(), 'temp.pt') works.

[LysandreJik]

@paramansh Do you mind opening an issue with your specific problem, your software versions and code sample so that we may debug on our side? Thanks.