Deleting non-visible layers

[erand96]

Hello!

I'm working on a project to convert ~1,000 PDF portfolios into normal PDFs with multiple pages. My process is:

1. Read PDF portfolio
2. Extract all attachments from PDF portfolio
3. Merge all extracted attachments into single PDF as separate pages
4. Write single PDF

The rest of my code is complete except for one error I continue to encounter. Some of the portfolio-attached PDFs have non-visible layers. When this PDF is extracted, the layers retain their invisibility, but when it is merged/attached into a multi-page PDF, all layers become visible.

I have messed around in PyMuPDF and found that using doc1.insert_pdf(doc2) allows me to retain the layer visibility on doc1, but makes all layers visible on doc2.

Is there a different way to use PyMuPDF to ensure these layers remain invisible while merging/attaching multiple PDFs? Otherwise, can all invisible layers be deleted or have all images/content removed from them?

Thank you in advance for any help/ideas

[JorjMcKie]

Oha, that's an interesting one!

I only have a rough idea in my head, let me sketch it:

1. Visibility in the context of Optional Content is controlled by optional content groups - OCGs, which are special simple PDF dictionary objects.
2. There is a central subdictionary of the catalog /OCProperties which stores all relevant information
3. In this sub-dict, there is the array /OCGs , which in all cases contains the xrefs of all OCGs of the PDF
4. Method insert_pdf is a page-oriented copy process: a source page together with everything it references directly or indirectly is collected and stored in the target PDF. A scratchpad is used to prevent multiple copies of the same object here - called "graftmap".
5. Anything of the source PDF, that is not referenced by some page, will not be copied over. This specifically pertains to the source /OCProperties sub-dict. This is why no visibility info lands in the target, and everything will be visible.
6. The source OCGs however are referenced and hence will land in the target - but will no longer perform their function.
7. So additional logic is needed that does the following:
   • Look up the source OCGs and store them away with a unique identifier (the xref cannot be used for this ...) and their source ON/OFF status. This identifier must be some property of the OCG itself, that will accompany it across the copy.
   • Then execute the copy for the requested page range.
   • Then scan the target PDF's complete xref table for OCG objects. For all matches with the previous OCG store, insert them in the target /OCProperties arrays.

This should work ...
As an OCG's unique identifier, I could imagine a suffix or prefix to its name, like

4 0 obj
<<
  /Type /OCG
  /Name (ocg1)  % modify to "/Name (ocg1.fitz00001)"
...
>>
endobj

This must happen as a modification in the source PDF! Before the copy. Which is no harm as long as the source PDF is not saved.
After spotting an OCG named with our suffix in the target, remove the suffix and store the xref in the right arrays of target /OCProperties.

[erand96]

Hi Jorj, thank you for the quick reply again. I appreciate the thought and time put into this

I believe I follow your thoughts on this. My only question is regarding how to alter the /OCProperties.

Initially, I thought to use get_ocgs() but that can only read the properties, and not alter or write new ones. Is that correct?

What attribute can I utilize to alter the /OCProperties?

Thanks!

[JorjMcKie]

If the target PDF has no /OCProperties at all yet, generate one by adding an arbitrary OCG (add_ocg()).
Then read the array /OCGs:

1. xref = doc.pdf_catalog()
2. ocgs = doc.xref_get_key(xref, "OCProperties/OCGs")
3. will return something like this: ('array', '[4 0 R 5 0 R]'). Add the xref of every of the copied OCGs like this: if 4711 is such an xref, then the string must become '[4 0 R 5 0 R 4711 0 R]'. And so on - "0 R" are required indicators for xrefs.
4. Then update: doc.xref_set_key(xref, "OCProperties/OCGs", "[4 0 R 5 0 R 4711 0 R]").
5. That was the difficult part: we now have published all OCG xrefs.
6. Next distribute each of the copied OCGs to the ON or OFF arrays. This is done via doc.set_layer():

doc.set_layer(-1,  # the standard layer
    on=[list of OCG xrefs],  # normal Python list of integers
    off=[list of OCG xrefs],
    )

[erand96]

Hello again,

I've been working on this code for a few days and have made progress, but still hung-up on a few parts.

Your first comment stated that I need to:

scan the target PDF's complete xref table for OCG objects
What attribute would I use for that? .xref_object()?

As far as I know, besides that part, I have everything else written and ready to go.

Thanks again!

[JorjMcKie]

Roughly should work like this:

# doc = target PDF
for xref in range(1, doc.xref_length()):  # loop through all object of target PDF
    if not "/Type/OCG" in doc.xref_object(xref, compressed=True):
        continue
    _, name = doc.xref_get_key(xref, "Name")
    name = name[1:-1]  # remove the brackets
    pos = name.find(".fitz")  # check if OCG fro some source
    if pos < 0:
        continue ## not for us
    old_name = name[:pos]
    ocg_key = name[pos+1:]
    ocg_state = ocg_states[ocg_key]
    doc.xref_set_key(xref, "Name", old_name)  # change back to old name
    # now put the xref into /OCProperties arrays as dicussed in pre post