Add Nystromformer

[novice03]

What does this PR do?

This PR adds the Nystromformer transformer model to the repository.

Paper: https://arxiv.org/abs/2102.03902
Code: https://github.com/mlpen/Nystromformer
Checkpoint: Nystromformer sequence length 512

Who can review?

@NielsRogge

[stefan-it]

Hey @novice03 , really cool PR, can't wait to fine-tune models with it :)

I have some questions about the tokenization part: it seems that the ALBERT tokenizer (and an albert spm model) is used and uploaded to the model hub, so I think in config.json the entry "tokenizer_class": "AlbertTokenizer" should be added so that AutoTokenizer is working.

But I've looked at the original pre-processing and pre-training code and it seems that the RoBERTa tokenizer is used for pre-training a Nyströmformer model, why is an albert spm model used here 🤔

[novice03]

Hello, thanks for taking a look at this PR. I've added the tokenizer entry to the config.json file. It is true that the code seems to use RoBERTa tokenizer. However, the model checkpoint released is of a model with vocab_size=30,000. Furthermore, they've also used albert tokenizer in their code. I've tried using BertTokenizer and RobertaTokenizer, but they give errors or incorrect results. The checkpoint released by the author only works with this specific albert tokenizer configuration.

[SaulLu]

Very cool to have this new model @novice03 🤗 .

Let me share with you a little information about the tokenizers of uw-madison/nystromformer-512 on this PR.

@NielsRogge showed me that there is currently an inconsistency in the treatment of special tokens with the slow and fast versions of uw-madison/nystromformer-512 tokenizers.

I think that to generate into the slow_and_fast folder the files for the fast tokenizer, we should do:

from transformers import AlbertTokenizer

tokenizer = AlbertTokenizer.from_pretrained("uw-madison/nystromformer-512")
tokenizer.save_pretrained("slow_tokenizer")

fast_tokenizer = AlbertTokenizerFast.from_pretrained(
        "slow_tokenizer", 
        bos_token="[CLS]",
        eos_token="[SEP]",
        unk_token="<unk>",
        sep_token="[SEP]",
        pad_token="<pad>",
        cls_token="[CLS]",
        mask_token="[MASK]"
        )

fast_tokenizer.save_pretrained("slow_and_fast")

I hope this is helpful!

[LysandreJik]

This looks great! Only left a few comments.

Would you be open to also contribute the TensorFlow version of the model, or would you rather we start with the PyTorch model?

[sgugger]

Thanks a lot for working on this model, this PR is in great shape! I just left styling suggestions.