Add utility to index PDF documents content

[benoit74]

Content of PDF documents is not indexed for full text search, while on some ZIM it is the "core" of the ZIM.

Extracting PDF info would be beneficial to many scrapers and should thus ideally be exposed in scraperlib.

See e.g. openzim/warc2zim#289

[benoit74]

pypdf (which was short replaced by PyPDF2) and PyMuPDF seems to be the two most versatile, widely popular and well maintained Python libraries to extract PDF content and metadata (for #168).

Sample PDFs for tests:

• https://irp.fas.org/doddir/milmed/first.pdf : 224 pages PDF with searchable text
• https://irp.fas.org/doddir/milmed/sfhandbook-pt1.pdf : 142 pages PDF with only images (containing text)
• https://irp.fas.org/doddir/milmed/sfhandbook.pdf : 407 pages PDF with only images (containing text)
• https://irp.fas.org/doddir/milmed/family.pdf : 84 pages PDF with only blury images (containing text)

[benoit74]

Sample encrypted PDF for tests: https://uwaterloo.ca/onbase/sites/ca.onbase/files/uploads/files/samplesecured_256bitaes_pdf.pdf

[benoit74]

pypdf and PyMuPDF seems to be very close in terms of feature, especially for the basic need we have today. They both opened the sample documents and provided nice metadata and text content (when the content was not an image).

The main difference seems to be that PyMuPDF already has embedded tesseract option for OCR of PDF images while this is not the case for pypdf. It is however probably not something we want to do for now since it can produce lot's of garbage data.

Another "slight" detail is also that PyMuPDF pretend to be "high performance". Not sure what it exactly mean however ^^

Since our needs are currently very modest, I propose to mostly randomly select PyMuPDF and anyway ensure that this implementation detail does not leak into the scraperlib API, so that we can easily and transparently switch to another library should this become an issue.

[kelson42]

If there is no big diff in perfs, then level of activity, number of maintainers, ... should bevthe key differenciator.

[benoit74]

If there is no big diff in perfs, then level of activity, number of maintainers, ... should bevthe key differenciator.

To extract text from the 224 pages https://irp.fas.org/doddir/milmed/first.pdf , it takes 0.9 sec with pypdf, 0.3 sec with pymupdf. Difference is not negligible.

PyMUpdf has a bit more activity than pypdf, and it's activity is rather increasing where it is decreasing for pypdf.

Finally PyMUpdf is an important part of https://artifex.com/ commercial offer (services and software). Where pypdf is a purely opensource project maintained (and at some point in time dropped during the 2.x fork) by the community.

Number of core committers seems to be mostly equivalent between both projects.