LaTeX OCR (pix2tex) - with Handwritten Recognition

Preview from Lukas Blecher (Repo Creator)

Build upon the pix2tex project. It is an LaTeX OCR that takes the most recent image [of a formula or equation] in your clipboard and converts it into a LaTeX format.

LaTeX is the preferred format when creating academic material. An OCR that specializes in converting image equation to LaTeX equations would allow students and academic professional to leverage the older material on more advanced mathematical subjects where most of the searchable resources out there are on subject closely related to algebra. Also, popular OCR tools like Socratic (made by Google) do not accurately read advanced calculus equations and creating the LaTeX equations from often lead to format mistakes.

Our Goals:

• Gain the competency of navigating, analyzing, and comprehending code bases; open-source contributing.

• Use additional data to expand upon the tool’s the handwritten scanning capabilities.

EXAMPLE: Uncovered and meticulously analyzed three distinct biases present in ChatGPT, employing advanced Python techniques and data analysis methodologies, all within AI4ALL's cutting-edge Apply AI program.

Problem Statement

When academic professionals use old documents, build upon the math found in scientific papers, or repurpose documents made by other academic professionals, they need the original LaTeX syntax to do their work. Currently, professors would manually create the equations or email the author assuming they can get a hold of them and they reply. This is an unreliable and tedious process.

When students do their homework or search for help online, they need a standard and reliable format to share their equations. Currently, students don’t search for help or hand write their equations. When using AI tools and search engines LaTeX is often the only format that computers can read large, complex, or calculus equations. This is an unrefined process when we have LaTeX, but LaTeX is hard to learn and not well known among students. When student do try LaTeX, it’s very easy to make mistakes.

Key Results

(UPDATE IN README.md) Enumerate the main results of this project in a list and describe them.

EXAMPLE:

1. Recorded over 1,000 unique prompts and their responses generated by ChatGPT
2. Identified three biases in ChatGPT's responses
   • When prompted about this world event
   • When prompted about this field of science
   • When prompted about this political party

Methodologies

(UPDATE IN README.md)

EXAMPLE: To accomplish this, we utilized the OpenAI API to interact with ChatGPT, and we designed a custom Python script to generate diverse prompts and collect corresponding responses. The data was then processed and analyzed using pandas, enabling us to detect patterns and biases in the AI model's outputs. Engineered a Python script to generate over 1,000 prompts and elicit their responses from ChatGPT, utilizing pandas to collect the data. When prompted for solutions to this specific relevant crisis, nearly 80% of ChatGPT's responses promoted a certain worldview.

Data Sources

Kaggle Datasets: https://www.kaggle.com/datasets/aidapearson/ocr-data

Technologies Used

(UPDATE IN README.md) List the technologies, libraries, and frameworks used in your project.

EXAMPLE:

• Python
• pandas
• OpenAI API

Authors

(UPDATE IN README.md) List the names and contact information (e.g., email, GitHub profiles) of the authors or contributors.

EXAMPLE: This project was completed in collaboration with:

• John Doe (john.doe@example.com)
• Jane Smith (jane.smith@example.com)

Using the model

To run the model you need Python 3.7+ ✔

If you don't have PyTorch installed. Follow their instructions here. ✔

Install the package pix2tex:

pip install "pix2tex[gui]" ✔

Model checkpoints will be downloaded automatically.

There are three ways to get a prediction from an image.

1. You can use the command line tool by calling pix2tex. Here you can parse already existing images from the disk and images in your clipboard.

2. Thanks to @katie-lim, you can use a nice user interface as a quick way to get the model prediction. Just call the GUI with latexocr. From here you can take a screenshot and the predicted latex code is rendered using MathJax and copied to your clipboard.

   If the model is unsure about the what's in the image it might output a different prediction every time you click "Retry". With the temperature parameter you can control this behavior (low temperature will produce the same result).

3. Use from within Python

   from PIL import Image
   from pix2tex.cli import LatexOCR
   
   img = Image.open('path/to/image.png')
   model = LatexOCR()
   print(model(img))

The model works best with images of smaller resolution. That's why I added a preprocessing step where another neural network predicts the optimal resolution of the input image. This model will automatically resize the custom image to best resemble the training data and thus increase performance of images found in the wild. Still it's not perfect and might not be able to handle huge images optimally, so don't zoom in all the way before taking a picture.

Always double check the result carefully. You can try to redo the prediction with an other resolution if the answer was wrong.

Training the model

Install a couple of dependencies pip install "pix2tex[train]".

1. First we need to combine the images with their ground truth labels. I wrote a dataset class (which needs further improving) that saves the relative paths to the images with the LaTeX code they were rendered with. To generate the dataset pickle file run

python -m pix2tex.dataset.dataset --equations path_to_textfile --images path_to_images --out dataset.pkl

To use your own tokenizer pass it via --tokenizer (See below).

You can find my generated training data on the Google Drive as well (formulae.zip - images, math.txt - labels). Repeat the step for the validation and test data. All use the same label text file.

2. Edit the data (and valdata) entry in the config file to the newly generated .pkl file. Change other hyperparameters if you want to. See pix2tex/model/settings/config.yaml for a template.
3. Now for the actual training run

python -m pix2tex.train --config path_to_config_file

If you want to use your own data you might be interested in creating your own tokenizer with

python -m pix2tex.dataset.dataset --equations path_to_textfile --vocab-size 8000 --out tokenizer.json

Don't forget to update the path to the tokenizer in the config file and set num_tokens to your vocabulary size.

Model

The model consist of a ViT [1] encoder with a ResNet backbone and a Transformer [2] decoder.

Performance

BLEU score	normed edit distance	token accuracy
0.88	0.10	0.60

Acknowledgment

Code taken and modified from lucidrains, rwightman, im2markup, arxiv_leaks, pkra: Mathjax, harupy: snipping tool

References

[1] An Image is Worth 16x16 Words

[2] Attention Is All You Need