No script available for LiLt fine tuning on Document Image classification

[madhavi1102]

In the research paper, it has been shown the LiLT has performed well on RVIL-CDIP dataset for document classification. but in the git repo, there is no file/script/instructions present to fine-tune on custom dataset for document classification or RVL-CDIP dataset.
Please provide the necessary steps or file for the same.

I would appreciate the response.

[zhonglifan]

Hi, the authors described the fine-tuning details in the paper's Appendix.

we leverage the image features explicitly in the fine-tuning stage, following LayoutLMv2 (Xu et al., 2021a). We pool the visual feature of the ResNeXt101-FPN (Xie et al., 2017; Lin et al., 2017) backbone into a global feature, concatenate it with the [CLS] output feature, and feed them into the final classification layer.

I managed to implement a LiLT-based document classifier according to this explanation, with the LiLT codebase released in huggingface's transformers. (basically just implement a new class for sequence classification by imitating other XXXForSequenceClassification class in transformers)

I combined LiLT, RoBERTa, and an visual backbone just as described in paper except my visual bakcbone is without a FPN because in my data it decreased the accuracy.

Haven't got the time to evaluate it in RVL-CDIP, but, I tested on a private dataset of bussiness document, and the result was pretty well.

Hope it can help ~