Position ids in RoBERTa

[qhd1996]

Environment info

• transformers version:
• Platform: Ubuntu 20.04
• Python version: Python 3.7.10
• PyTorch version (GPU?): 1.6.0_py3.7_cuda10.1.243_cudnn7.6.3_0
• Tensorflow version (GPU?): No
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: No

Who can help

@LysandreJik

Information

Position ids in RoBERTa is not implemented properly.
The problem arises when using:
create_position_ids_from_input_ids in transformers.models.roberta.modeling_roberta.py
Based on this function, position id 0 is never used. This may cause problem when the sequence is long, for example, 512. Token whose id >= 511 will not get its corresponding token ids.

[NielsRogge]

There's a reason why position ids don't start at 0 for RoBERTa, see #5285

[patil-suraj]

RoBERTa never uses 0 and 1 positional ids, in ROBERTa, all pad tokens have position id of 1, and the rest of the tokens have position ids in the range (2, seq_length - num_pad_tokens). It's implemented like this to match the original implementation in fairseq.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.