Bug: Incomplete deletion of annotations

[markgraves]

I'm deleting Highlight annotations, which have text content, from a page using deleteAnnot. However the annotations (generated using Adobe Acrobat reader) also have an Appearance Stream, which is not deleted, and thus it remains in the document. Is it possible to delete the Appearance Streams (/AP) from a page, too?

[JorjMcKie]

I am a little astonished that the /AP is not deleted (deleting an annotation is a utility function call to the MuPDF base library).
Maybe (and that would be typical for their cautious attitude), they do not do it because the /AP object might be shared / referenced by yet other objects.
But when you save the PDF with garbage=3, that would remove all unreferenced objects.
Please drop me a note if this is not working - I would include an extra deletion of the /AP object in the next PyMuPDF version.

[JorjMcKie]

Just confirmed in the MuPDF C source:
The delete annot function only removes the reference to the annot from the respective array of the page object definition - with an explicit comment that this approach is taken out of caution:

/* Remove the annot from the "Annots" array. */
annot_arr = pdf_dict_get(ctx, page->obj, PDF_NAME(Annots));
i = pdf_array_find(ctx, annot_arr, annot->obj);
if (i >= 0)
	pdf_array_delete(ctx, annot_arr, i);
/* The garbage collection pass when saving will remove the annot object,
 * removing it here may break files if multiple pages use the same annot. */

Just to be complete:
By the books, the PDF definition actually prohibits sharing annotations between pages ... but PDF files of the real world are full of abuses of the specs. Even PyMuPDF allows duplicating pages including annotations 😉

[markgraves]

Yes, the /AP is still there after save with garbage=4, though the annotation is no longer in the file.

[JorjMcKie]

ok, this then means that the object is referenced by still some other object. Can you search for nnn 0 R, where nnn is the xref of the /AP?
I am using SciTE editor for things like this ...

[markgraves]

Looks like when Acrobat Reader created the Annotation with a note, it created three objects: the Highlight Annotation, the /AP, and a popup <</F 28/Open false/Parent 20 0 R/Rect[612 561.921 816 675.921]/Subtype/Popup/Type/Annot>>. The /AP is referenced by the popup, and it also appears the popup maybe referenced by the /AP, as the /AP includes the nnn 0 R string for the popup xref.

[JorjMcKie]

I'm beginning to see what should be done to resolve this.
Can you whether the popup annot xref is still referenced by the page?
Print the page object definition with this statement:

print(doc._getXrefString(page.xref))

[JorjMcKie]

Just looked myself: yes, this is probably so.
But MuPDF ignores popup annots (does not support them). So we do not see them in PyMuPDF and thus cannot delete them.
But if an annot has an associated popup, it will contain an entry /Popup. So what should be done in PyMuPDF upon annot deletion, is looking for this entry and delete the popup found there ...

[JorjMcKie]

I will make the following changes to annot deletion in the indicated sequence:

1. Delete the /AP object. Even if the annotation is used elsewhere, that would be against the books, so no basis for complaints.
2. Delete any popup annotation connected to the annot. This entails deleting the popup from the annots array of the page and deleting the object itself. The justification being as before.
3. Remove the /AP and /Popup keys from the to-be-deleted annotation.
4. Delete the annot itself with existing MuPDF (just remove its reference from the page's array).

That should leave the PDF in a valid overall state. As is always the case, the objects deleted (/AP and /Popup annotation) will disappear after garbage collection. The deleted annot as well, if not used elsewhere. If, however used elsewhere, it will no longer have associated /AP nor /Popup objects. This has no adverse effect other than that updating it with PyMuPDF or any other capable software will create a new /AP or /Popup, respectively.

How urgent is your need?

[JorjMcKie]

The approach sketched above may still yield the following slight inconsistency:

If there exist one or more reply annotations to the deleted annot, they will be left over dangling. Replies are of annotation type 'Text' and have their own /AP and /Popup. So actually, each annot deletion would need to scan the page's /Annots array and look for 'Text' annotations with an /IRT pointing to the original annot. If found, delete it using the above logic ...

[markgraves]

One minor clarification on points 2-3, it appears the /Popup is associated with the /AP rather than directly with the annot being deleted.

Its not urgent. Now that I understand better what's happening, I can continue development using pymupdf and ignore the temporary issue of annotations not being completely removed when I reopen the modified file in Acrobat Reader.

A fix anytime in the next month or so would be great for me.

[JorjMcKie]

One minor clarification on points 2-3, it appears the /Popup is associated with the /AP rather than directly with the annot being deleted.

Interesting, would you mind sending a print of the onject definitions? Works like this:

>>> annot.xref
22
>>> print(doc._getXrefString(22))  # annot object definition
<<
  /AP <<
    /N 12 0 R  # /AP/N object xref
  >>
  /C [ 1 1 0 ]
  /F 4
  /NM (66ea58db-02a2-4211-86ae-e0b2343e692b)
  /P 11 0 R
  /Popup 26 0 R  # /Popup xref
  /QuadPoints [ 104.044528 753.1121 242.50353 702.7172 98.87524
      738.9096 237.33423 688.5147 100 653.825 247.34496 653.825
      100 638.711 247.34496 638.711 130.15807 559.1428 268.61714
      609.5377 135.32736 544.9403 273.7864 595.3352 ]
  /Rect [ 95.9653 544.335 276.521 753.717 ]
  /Subtype /Highlight
  /Type /Annot
>>
>>> print(doc._getXrefString(26))  # popup object definition
<<
  /F 28
  /Open false
  /Parent 22 0 R
  /Rect [ 595 639.112 799 753.112 ]
  /Subtype /Popup
  /Type /Annot
>>
>>> print(doc._getXrefString(12))  # /AP/N object definition
<<
  /BBox [ 0 0 180.556 209.383 ]
  /FormType 1
  /Length 20
  /Matrix [ 1 0 0 1 0 0 ]
  /Resources <<
    /ExtGState <<
      /R0 <<
        /AIS false
        /BM /Multiply
        /Type /ExtGState
      >>
    >>
    /ProcSet [ /PDF ]
    /XObject <<
      /MWFOForm 13 0 R
    >>
  >>
  /Subtype /Form
  /Type /XObject
>>
>>> 

[markgraves]

Oh, okay. That's helpful. I was confused. I get the same structure when I print the definitions like that.

[JorjMcKie]

Good, thank you.
I am developing the solution now. The only thing somewhat more tricky are those /IRT annotations. There are created, if you "reply" to an annotation or an existing popup. But - as mentioned before - they themselves are no popups, but regular annotations, and they have no backpointers to the answered annotation. So I need to parse the annot array and inspect every annot if it is a reply to the dead man walking.
If yes, delete it using the same full logic, because /IRTs may have popups and /APs.

What is your Python environment? Once done I can either send a Windows wheel or point to OSX / Linux wheels for you to test, before I publish to PyPI.