Want any help?

[benrbray]

This is a fantastic tool and I would be interested in contributing, either on the ML or software eng side. Do you have any wishlist of ideas / backlog of tasks? I see there are some TODOs and projects. If you add a bit more detail to the current tasks, I'd be happy to contribute.

[lukas-blecher]

Hi Ben,
that would be awesome. Thanks for your interest in the project.

On the top of my list at the moment is support of handwritten formulae.
Datasets: CROHME, Im2Latex-Handwritten and maybe OCR-Data (Kaggle)
currently there are a couple small problems with the dataset generation pipeline in the preprocessing. Next combine all datasets and train on them. Hopefully that works.

The second item would be to compile the model into torchscript or onnx formats. That turned out to be more challenging than expected. The main problem is that the image input size can be dynamic but that doesn't play well with the tracing/scripting methods.
Ignoring that issue, I don't know how to pack the resulting model into a dependency-less application.

Next, I want to reduce the model size through distillation. At the moment the weights are ~100 Mb, which is not bad for the performance, but it would be better to reduce it anyway.

That's the situation at the moment. Is there something you want to try out?

[benrbray]

Thanks for the pointers. I've been meaning to learn more about distillation, so I might try to tackle that first. I'll read some more and dig into the existing code.

I'd also be interested in giving the handwriting recognition a shot, but it seems like a much larger, messier task :) I will need to read a few papers and get back to you. Are you thinking of just running the existing ViT model on the handwriting dataset to start with?