Let DPOTrainer Support padding_free

[fzyzcjy]

Feature request

Hi thanks for the library! It seems that https://huggingface.co/blog/packing-with-FA2 introduces a way to avoid a lot of pad tokens in SFT, and makes training faster. Therefore, it would be great if the same thing can be used for DPO.

Motivation

(see above)

Your contribution

n/a

[qgallouedec]

Thanks @fzyzcjy! Can you elaborate a bit? What is this padding free method?

[fzyzcjy]

Oh sorry I provided the wrong link, now the link is updated to point to the correct "padding_free" article

[qgallouedec]

Thanks for the pointer. This would be nice addition! Any contribution is welcome. I mark this one as good second issue

[qgallouedec]

The guideline is basically to:

1. Update PreferenceCollator to add padding_free like in add arg padding_free to DataCollatorForCompletionOnlyLM #1887
2. Update concatenated_inputs to (a) make xxx_attention_mask optional and add support for xxx_position_ids
3. Add a test

[fzyzcjy]

Thank you!

[dame-cell]

• should padding_free in PreferenceCollator be like kind of a optional arguement or like keep it default ?

but why make `xxx_attention_mask' optional ?
is it because padding-free sequences might not use attention masks at all.?
for example

• In regular training with padding, attention_masks are needed to tell the model which tokens are real and which are padding (0s for padding, 1s for real tokens)
• In padding-free training, since we remove all padding tokens, every token is a real token, so we don't need explicit masks to distinguish between real and padding

does this make sense?
Thank you for your patience - I wanted to verify that I understand these concepts correctly

[qgallouedec]

I think it makes sense yes.

[dame-cell]

@fzyzcjy @qgallouedec if no one is working on this I would like to help

[fzyzcjy]

@dame-cell I do not have time recently for that, and your PR would be great and many thanks!

[zwhe99]

Is it possible for PPO to support padding_free?