Questions Regarding Fine-tuning the UDOP Model

[Arjunexperion]

Hi, I have a couple of questions about fine-tuning the UDOP model:

1. For key-value extraction, a sample CORD dataset is used. Is there any resource or guideline available to understand how this dataset is structured so that we can format our own data accordingly?

2. The current notebook supports single-page document classification. What modifications would be needed to extend it for multi-page document classification during fine-tuning?

Looking forward to your insights. Thanks!

[NielsRogge]

Hi,

1. Yes I would recommend an annotation tool such as Label Studio, Prodigy or CVAT. Basically you need to ensure to format the data in the same way as the CORD dataset (same columns, same formatting).
2. For multi-page document classification, I would recommend a small vision-language model (VLM) which supports multiple input images. A recommendation here (at the time of writing) is the Qwen3-VL series: https://huggingface.co/collections/Qwen/qwen3-vl. To fine-tune such a VLM, you basically train it to output text (whatever you want, could be a class label). Datacamp has a pretty good tutorial on this: https://www.datacamp.com/tutorial/fine-tuning-qwen3-vl-8b.