Conversion Accuracy: PDF-to-MD

[AkhileshNegi]

Describe the solution you'd like
We’ve begun using Zerox to convert the Veddis project PDFs into Markdown files. Now, we need a way to test how accurate those conversions are, so we can make sure we’re using the best tool and retaining as much content as possible.

[AkhileshNegi]

Progress so far

Took the markdown file generated by Zerox as the base and created a golden MD.

Original File Path: Model CLF/Advisory on MCLF Immersion/001_Advisory on MCLF Immersion sites.md.

For creating golden MD, compared the original PDF with the markdown from Zerox and then editing directly Zerox-generated markdown, keeping the original format intact

Golden.md
Original File.pdf
Marker.md
Zerox.md

Comparison using ndiff in similarity_engines/similarity.py

Golden MD with Marker MD

Similarity Ratio: 10.57%

Golden MD with Zerox MD

Similarity Ratio: 87.71%

Golden MD with Zerox MD by adding few line breaks
Similarity Ratio: 87.08%

Few observations:

• Similarity comparison is done using 
from difflib import ndiff comparing sequences of lines of text, and producing human-readable differences or deltas
• Since it is comparing sequences of lines of text even linebreak or slight formatting differences vary the similarity ratio even though most of the content is same thus if the Golden MD is created on top of zerox MD then it will have higher similarity compared to Marker MD as the formatting is similar

Comparison using SequenceMatcher in similarity_engines/sequence.py

Golden MD with Marker MD
The similarity between the two files is: 46.29%

Golden MD with Zerox MD
The similarity between the two files is: 93.50%

Golden MD with Zerox MD by adding few line breaks
The similarity between the two files is: 93.16%

Few observations:

• Similarity comparison is done using difflib import SequenceMatcher which does comparing pairs of sequences of any type, so long as the sequence elements are hashable
• Similar behaviour is observed where linebreak or slight formatting differences vary the similarity ratio

[AkhileshNegi]

Expected behaviour from above testcases is

1. Golden MD with Marker MD: should give accuracy of 60-70% or more for this particular set of marker and golden MD
2. Golden MD with Zerox MD: should give accuracy in the ballpark of Marker MD(point 1) as content wise both the files are similar and only formatting is different
3. Golden MD with Zerox MD by adding few line breaks: it should same accuracy as with Zerox MD(point 2) as the content is same and only formatting is different

[jerome-white]

I've added the "golden markdown" to a branch of the Veddis data repository (here) using the naming convention from the Phase 2 Google Drive. It would be nice to use that branch/directory to manage our transcriptions.