Error when rendering shapes to existing pdf

[carlaTV]

Please provide all mandatory information!

Describe the bug (mandatory)

When I commit a set of shapes to a blank pdf, the result is as expected. However, when I try to commit them to an existing pdf, they are turned 90 degrees (upside down), rescaled and sometimes the text does not appear anymore.

To Reproduce (mandatory)

Consider an input as "original_document.pdf" (attached), where shapes and text coexist.
original_document.pdf

Fitz can successfully extract the text and the shapes, and when I render them separately it works as expected (see only_shapes.pdf and only_text.pdf).
only_shapes.pdf
only_text.pdf
This is how I generate each file:
TEXT EXTRACTION:

import fitz
import pdfkit

filename = 'original_document.pdf'

doc = fitz.open(filename)
page = doc[0]                    # it has only one page
text = page.get_text("html")
with open('text.html', 'w') as f:
    f.write(text)
pdfkit.from_file('text.html', 'only_text.pdf')

SHAPES EXTRACTION

import fitz

# open original document
doc = fitz.open(path_input+filename)
page = doc[0]                   # it has only one page

outpdf = fitz.open()         # open new document
outpage = outpdf.new_page(width=page.rect.width, height=page.rect.height)         # create new page
shape = outpage.new_shape()  # make a drawing canvas for the output page

paths = page.get_drawings()  # extract existing drawings

for path in paths:
    # ------------------------------------
    # draw each entry of the 'items' list
    # ------------------------------------
    for item in path["items"]:  # these are the draw commands
        if item[0] == "l":  # line
            shape.draw_line(item[1], item[2])
        elif item[0] == "re":  # rectangle
            shape.draw_rect(item[1])
        elif item[0] == "c":  # curve
            shape.draw_bezier(item[1], item[2], item[3], item[4])
        else:
            raise ValueError("unhandled drawing", item)
    # ------------------------------------------------------
    # all items are drawn, now apply the common properties
    # to finish the path
    # ------------------------------------------------------
    shape.finish(
        fill=path["fill"],  # fill color
        color=path["color"],  # line color
        dashes=path["dashes"],  # line dashing
        even_odd=path["even_odd"],  # control color of overlaps
        closePath=path["closePath"],  # whether to connect last and first point
        lineJoin=path["lineJoin"],  # how line joins should look like
        lineCap=max(path["lineCap"]),  # how line ends should look like
        width=path["width"],  # line width
        stroke_opacity=path["opacity"],  # same value for both
        fill_opacity=path["opacity"],  # opacity parameters
        )
# all paths processed - commit the shape to its page
shape.commit()

outpdf.save('only_shapes.pdf')

THE ISSUE: COMMITING SHAPES TO AN EXISTING DOCUMENT
In order to add the extracted shapes to the doument that only contains the shapes (in other words: committing the shapes to only_text.pdf), instead of creating a new document and a new page within it with these lines of code:

outpdf = fitz.open()         # open new document
outpage = outpdf.new_page(width=page.rect.width, height=page.rect.height)         # create new page
shape = outpage.new_shape()  # make a drawing canvas for the output page

I open the document "only_text.pdf" and define its first page as outpage :

outpdf = fitz.open('only_text.pdf')                     # open existing document
outpage = outpdf[0]                                           # define outpage as the first (and only) page
shape = outpage.new_shape()                          # make a drawing canvas for the output page

text_and_shapes__same_file.pdf

It has to be mentioned that if I create a new page to an existing document, it correctly commits the shapes to this new page (however, this is not what I want).
See text_and_shapes__sep_file.pdf

Expected behavior (optional)

What I want is to be able to extract text and shapes from a pdf and generate them in a new document.

Screenshots (optional)

What I want:

I am able to generate the shapes in a new page:

However, when I try to commit them to an existing document, they appear upside down and rescaled (and in some cases, like this one, the text disappears):

I assume that I am doing something wrong when I define outpage as an existing pdf page.

Your configuration (mandatory)

• Operating system: MacOS Catalina
• Python 3.6.4
• PyMuPDF Version: 1.18.14, installed with pip.

For example, the output of print(sys.version, "\n", sys.platform, "\n", fitz.__doc__) would be sufficient (for the first two bullets).
3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
darwin

PyMuPDF 1.18.14: Python bindings for the MuPDF 1.18.0 library.
Version date: 2021-06-01 08:11:38.
Built for Python 3.6 on darwin (64-bit).

Additional context (optional)

Thanks a lot!!

[JorjMcKie]

My compliments for this detailed issue description!

The error occurs b/o a global geometry change in the target PDF that has not been wrapped propertly in protecting / localizing stacking commands "q" / "Q". See here for an explanation and how to solve: https://pymupdf.readthedocs.io/en/latest/faq.html#misplaced-item-insertions-on-pdf-pages.
Either do page.wrap_contents() or page.clean_contents() before you insert anything (shapes, text, images, ...).
Then everything should work fine.

To detect / prevent this from happening,, check page.is_wrapped and conditionally execute the wrapping.

[carlaTV]

It worked regarding the position of the shapes, now they are correctly rendered.
However, sometimes it replaces the text, and only the shapes appear. Do you have any idea about why this happens?

Thank you very much!

[JorjMcKie]

sometimes it replaces the text, and only the shapes appear.

Hm, the only thing I can think of now is that (some of) the shapes have a fill color, which then of course hides text underneath.
How about choosing option shape.commit(overlay=False)?

[JorjMcKie]

That would put the shape creating instructions before those painting the text ...

[JorjMcKie]

This approach would also have resolved your ominous problem with the "global geometry change".

[JorjMcKie]

Another option is of course simply not setting the shape fill color.
Or using some transparency (opacity) value < 1 ...

[carlaTV]

I set fill to None and shape.commit(overlay=False) and it completely worked:

Thank you very much!!

[JorjMcKie]

Great!
But each of fill=None and shape.commit(False) alone should already work.