README.md

pix2tex - LaTeX OCR

The goal of this project is to create a learning based system that takes an image of a math formula and returns corresponding LaTeX code. As a physics student I often find myself writing down Latex code from a reference image. I wanted to streamline my workflow and began looking into solutions, but besides the Freemium Mathpix I could not find anything ready-to-use that runs locally. That's why I decided to create it myself.

Using the model

1. Download/Clone this repository
2. For now you need to install the Python dependencies specified in requirements.txt (look further down)
3. Download the weights.pth file from my Google Drive and place it in the checkpoints directory

The pix2tex.py file offers a quick way to get the model prediction of an image. First you need to copy the formula image into the clipboard memory for example by using a snipping tool (on Windows built in Win+Shift+S). Next just call the script with python pix2tex.py. It will print out the predicted Latex code for that image and also copy it into your clipboard.

Note: As of right now it works best with images of smaller resolution. Don't zoom in all the way before taking a picture. Double check the result carefully. You can try to redo the prediction with an other resolution if the answer was wrong.

Model

The model consist of a ViT [1] encoder with a ResNet backbone and a Transformer [2] decoder.

Data

We need paired data for the network to learn. Luckily there is a lot of LaTeX code on the internet, e.g. wikipedia, arXiv. We also use the formulae from the im2latex-100k dataset.

Fonts

Latin Modern Math, GFSNeohellenicMath.otf, Asana Math, XITS Math, Cambria Math

Requirements

Evaluation

• PyTorch (tested on v1.7.0)
• Python 3.7+ & dependencies (requirements.txt)

  pip install -r requirements.txt
  

  install timm directly pip install -U git+https://github.com/rwightman/pytorch-image-models.git

Dataset

In order to render the math in many different fonts we use XeLaTeX, generate a PDF and finally convert it to a PNG. For the last step we need to use some third party tools:

• XeLaTeX
• ImageMagick with Ghostscript. (for converting pdf to png)
• Node.js to run KaTeX (for normalizing Latex code)
• de-macro >= 1.4 (only for parsing arxiv papers)
• Python 3.7+ & dependencies (requirements.txt)

TODO

• support handwritten formulae
• reduce model size
• find optimal hyperparameters
• create a standalone application

Contribution

Contributions of any kind are welcome.

Acknowledgement

Code taken and modified from lucidrains, rwightman, im2markup, arxiv_leaks

References

[1] An Image is Worth 16x16 Words

[2] Attention Is All You Need