doc.xref_get_key for javascript cannot find object

[juviwhale]

Describe the bug

Some documents, that contain Javascript, throw a cannot find object in xref exception when using the .scrub method.

To Reproduce

pdf_doc = fitz.Document(stream=file_data, filetype="application/pdf")
pdf_doc.scrub(
    attached_files=True,
    clean_pages=True,
    embedded_files=True,
    javascript=True,
    metadata=True,
    xml_metadata=True,
    remove_links=True,
    # TODO: OCR'd text?
    hidden_text=True,
    # Don't remove images
    redact_images=0,
    # Don't apply redaction annotations at this point
    redactions=False,
    # Don't remove form responses
    reset_fields=False,
    reset_responses=False,
)

Unfortunately I cannot provide a document to produce this because the documents contain PII (I attached a screenshot of the exception trace, hope that is enough). The issue is not specific to a single document as we have seen the same exception occur with 468 different PDF documents.

Exception Screenshot (from Sentry)

Your configuration

3.8.2 (default, Jul 17 2020, 15:47:13) 
[Clang 11.0.3 (clang-1103.0.32.62)] 
 darwin 
 
PyMuPDF 1.18.9: Python bindings for the MuPDF 1.18.0 library.
Version date: 2021-02-26 13:46:32.
Built for Python 3.8 on darwin (64-bit).

[JorjMcKie]

Confusing. doc.xref_get_key(xref, ...) always succeeds, if xref points to a valid object in doc.
Indeed: the error says that "235 0 R" contains no valid PDF object definition. Is this a just created xref number without yet having been filled with any object source?

[JorjMcKie]

Another weird observation:
doc is a new PDF - do you need to scrub new documents?

[juviwhale]

I am not sure I understand. To give more context we 'sanitize' PDFs at the beginning of our pipeline by loading each pdf, then scrub()ing, then saving the PDF for the use in the rest of the pipeline.

[JorjMcKie]

Anway, previously that part of the method used the outcome of doc.xref_object(xref) and iterated through its lines to find the relevant PDF keys (/JavaScript, /Metadata). That method returns the empty string "" if xref is an invalid object even though being in valid xref range.
So this type of error simply was ignored.
I am about to enforce a similar behaviour for doc.xref_get_key(xref, key): it will assume that key is not part of xref's object definition.

[JorjMcKie]

I am not sure I understand. To give more context we 'sanitize' PDFs at the beginning of our pipeline by loading each pdf, then scrub()ing, then saving the PDF for the use in the rest of the pipeline.

Your error can only occur if the xref I am investigating does exist, but contains no object definition. As I wrote in previous post, this situation was not detected previously.

[JorjMcKie]

Could you send me a printout of doc.xref_object(235) of that document?

[JorjMcKie]

No, wait, that would be "" right?

[JorjMcKie]

What I can do:

• make a hotfix pre-version so you can download the wheel in an hour or so
• send you an updated utils.py which wrap the relevant parts in a try-except clause

[JorjMcKie]

Another weird observation:
doc is a new PDF - do you need to scrub new documents?

What I was referring to: in your trace output, doc is shown as Document("", <memory, doc# 2>).

[JorjMcKie]

Sorry - I was wrong, forget my last post please.

[JorjMcKie]

There is a pre-version 1.18.10 here that should resolve the problem.

[JorjMcKie]

fixed in just published v1.18.10