Return_offsets_mapping when decoding

[Boltzmachine]

I want to get the character offset mapping of tokens when decoding for model-generated ids, similar to the return_offsets_mapping in tokenizer.call. But I cannot find out how to do it. For some tokenizers, I try first decoding the generated ids to text and encoding it back and get the char offset mapping. However , input_ids == tokenizer.encode(tokenizer.decode(input_ids) is not always true.

[ArthurZucker]

This should be something like this:

from transformers import RobertaTokenizerFast
tokenizer = RobertaTokenizerFast.from_pretrained('path', add_prefix_space=True)
ids = tokenizer([['drieme', 'drieme'], ['drieme']],
    is_split_into_words=True,
    padding=True,
    return_offsets_mapping=True)

return_offsets_mapping to the __call__ rather than in encode