"Position numbers begin at padding_idx+1."

[leimao]

Hi,
This might not be an issue but a general question. In make_positions,
https://github.com/pytorch/fairseq/blob/ba5f829f64d2adf9de4502dab94abf0a1a54106b/fairseq/utils.py#L168
Why the token positions "begin at padding_idx+1"? Can someone justify this? Thank you.

[lematt1991]

This is so that the first position is not confused as a padding token

[leimao]

@lematt1991 This is interesting. Glad to know. Thanks.

[leimao]

Maybe you should put a comment in this function. I think this will also be confusing to others.

[lematt1991]

Feel free to submit a PR!

[dtamayo-nlp]

I am pre-training a Roberta from scratch and I would like to change this implementation. While it is true that this approach makes sense if the pad_token_id=1, it will fail if we change the tokenizer to another one with pad_token_id>1.