Longformer Optimization

[Nouman97]

Hello, I wanted to use a similar procedure as the one mentioned in this notebook: PyTorch_Bert-Squad_OnnxRuntime_CPU.ipynb in order to optimize my question answering model based on the longformer. I wasn't able to find a mechanism to do this, can someone guide me regarding this?

[askhade]

@tianleiwu for comments

[tianleiwu]

@Nouman97 Did you convert the longformer model to onnx format? If so, please share the conversion script so that I could see whether the onnxruntime optimizer need some change to handle longformer onnx model.

If not, I would suggest to start with the conversion to onnx. You can follow the notebook to do that. Also, there is a tool in huggingface transformer: https://github.com/huggingface/transformers/blob/master/src/transformers/convert_graph_to_onnx.py. You may modify it by replacing the AutoModel to LongformerModel.

The Attention operator will support 2D attention mask in the coming onnxruntime 1.4.0. I think it is possible to help longformer.

[Nouman97]

Hello, thank you for your reply. I was using the convert_graph_to_onnx.py code, but was getting the following error: RuntimeError: Exporting the operator as_strided to ONNX opset version 11 is not supported. Please open a bug to request ONNX export support for the missing operator.
I haven't been able to figure it out

[tianleiwu]

@Nouman97, can you change the last parameter to be opset=12? If not working, try pytorch nightly package as well.

If the problem still exists, that means as_strided cannot be converted to ONNX right now. Then there are two alternatives:
(1) Raise an issue in https://github.com/pytorch/pytorch/issues, and wait until PyTorch supports as_strided to onnx conversion. I could also help forward this request to the owner of onnx exporter.

(2) Modify your modeling script to replace torch.as_strided by using other torch functions that can be converted to ONNX. This is a walk-around, and need not wait for PyTorch change.

[Nouman97]

Hello, thanks for following up; I tried changing the opset but the error still persists. It will be difficult to perform (2) because I am importing the entire architecture from huggingface transformers. Instead, I will be raising it as an issue as you mentioned.

[tosi-n]

Hi @Nouman97 did you get any luck with fixing the runtime error during longformer optimisation on onnx

[Nouman97]

Hi @Nouman97 did you get any luck with fixing the runtime error during longformer optimisation on onnx

Unfortunately not. I am still looking for solutions.

[stale]

This issue has been automatically marked as stale due to inactivity and will be closed in 7 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

[tianleiwu]

For anyone who is interested in Longformer, currently we are working on the optimization. Hopefully, we will have it working in two months.