instruction finetuning IDEFICS

[schwarzwalder93]

Hi Team,

I would like to instruct fine tune IDEFICS on custom dataset. Could you provide the training script used to produce idefics-9b-instruct ?

I suppose changes will be need to following snippets:

labels should correspond to responses/answers to the input instructions:

notebooks/examples/idefics/idefics_zero3_finetuning/idefics_zero3_finetuning.py

Line 71 in 1dda4aa

inputs["labels"] = inputs["input_ids"]

Modifying the logic before loss computation:
https://github.com/huggingface/transformers/blob/bd50402b56980ff17e957342ef69bd9b0dd45a7b/src/transformers/models/idefics/modeling_idefics.py#L1508C1-L1523C97