Convert all text's color in PDF to black while ensuring text is selectable

[qwertynik]

Looking for ways to change the color of all text in a PDF to black.

Any ideas for the same?

Code in font replacement sample scripts are useful, but that adds their own complexity. For instance, sometimes the space between words is much larger than in the original PDF.

[JorjMcKie]

That's not as easy as it sounds, unfortunately. You could of course walk through the page's /Contents object(s) and remove all color specifications. This will cause the default color black be used.
The problem here is, that text may also be contained in other stream objects, outside the page's contents (XObjects).
And of course, there may be things (drawings) which would be taken away their color, too.
Text objects are wrapped between pairs of bytestrings b"BT"/b"ET". Text color instructions need not be contained therein ... but you could well start with this assumption and only remove any colors inside these text objects and see how far you get.

[qwertynik]

Not particularly concerned about the color of other objects in the PDF being taken away. Given that, would it take this color conversion process easier @JorjMcKie?

[JorjMcKie]

No, it wouldn't be easier. Just less side effects if you knew that a color is meant for text only.
You know how to do the simple solution?

1. page.clean_contents()
2. read cleaned contents and split it into lines.
3. remove all lines (i.e. replace with b"") ending with any of b"rg", b"RG", b"k", b"K".
4. join resulting line: b"\n".join(lines)
5. replace the page's contents with this bytes object.

[qwertynik]

@JorjMcKie The color is meant for other objects too. But it is just that, other objects are not of concern. So, will experiment with the above step.
Does step 4 involve combining all the lines to one? Unclear about the steps.
Will experiment and see if it becomes clearer, or, seek help here.

[JorjMcKie]

Re 4.:

cont = b"\n".join(lines)  # makes a bytes object of the single lines
doc.update_stream(xref, cont)

Where xref has been obtained before like this:

page.clean_contents()  # cleans and combines all /Contents objects into one
xref = page.get_contents()[0]  # the resulting content's xref
lines = doc.xref_stream(xref).splitlines()
# now walk through the lines:
for i in range(len(lines)):
    if lines[i].endswith((b"k", b"K", b"rg", b"RG")):
        lines[i] = b""
cont = b"\n".join(lines)
doc.update_stream(xref, cont)

[qwertynik]

Thanks for sharing the snippet @JorjMcKie.

Added a few more lines of code to make it fully functional.

import fitz

inputFilePath = "pdfs/Demo.pdf"
doc = fitz.open(inputFilePath)
page = doc[0]

page.clean_contents()  # cleans and combines all /Contents objects into one
xref = page.get_contents()[0]  # the resulting content's xref
lines = doc.xref_stream(xref).splitlines()
# now walk through the lines:
for i in range(len(lines)):
    if lines[i].endswith((b"k", b"K", b"rg", b"RG")):
        lines[i] = b""
cont = b"\n".join(lines)
doc.update_stream(xref, cont)

doc.ez_save("output/TestBlack.pdf")

However, the text color is not converted to Black for all PDFs.
It is converted for this PDF ColorTest.pdf
But not for this one: Demo.pdf

Any ideas on what can be done here?

[JorjMcKie]

Sure, my bad. This creator was more creative with hiding color specs, but that script makes tabula rasa:

import fitz

skips = (b"k", b"K", b"rg", b"RG", b"sc", b"SC", b"scn", b"SCN", b"gs", b"cs")
doc = fitz.open("Demo.pdf")
for page in doc:
    page.clean_contents()
    xref = page.get_contents()[0]
    lines = page.read_contents().splitlines()
    for i in range(len(lines)):
        if lines[i].endswith(skips):
            lines[i] = b""
    doc.update_stream(xref, b"\n".join(lines))
doc.ez_save("x.pdf", pretty=True)

The last page is now completely black BTW 😂

There will still be cases that won't work even with this version. I told you before, it won't be a completely trivial thing:
A page may invoke so-called "XObjects". This works much like images. If you use page.show_pdf_page() this object type will be created under the hood.
The point is that XObjects may have their own content stream and also contain text. This text's color will not be touched by above logic. But one can also walk through the xobject streams and do the same type of thing with those.

[JorjMcKie]

Here is a better version, which only makes text completely black, other stuff will be a light gray. So the last page is readable again.

import fitz

skips = (b"k", b"K", b"rg", b"RG", b"sc", b"SC", b"scn", b"SCN", b"gs", b"cs")
doc = fitz.open("Demo.pdf")
for page in doc:
    page.clean_contents()
    xref = page.get_contents()[0]
    lines = page.read_contents().splitlines()
    for i in range(len(lines)):
        if lines[i].endswith(skips):
            lines[i] = b""
            continue
        if lines[i] == b"q":
            lines[i] = b"q 0.9 g 0.9 G"
        elif lines[i] == b"BT":
            lines[i] = b"BT 0 g 0 G"
        elif lines[i] == b"ET":
            lines[i] = b"ET 0.9 g 0.9 G"
    doc.update_stream(xref, b"\n".join(lines))
doc.ez_save("x.pdf", pretty=True)

[JorjMcKie]

replied my result to your yahoo account

[qwertynik]

@JorjMcKie For some reason, this condition was missing in the script being used. Checked the script you shared and added the same.

        elif lines[i] == b"ET":
            lines[i] = b"ET 0.9 g 0.9 G"

Post that the rectangles are no longer turning black.

[JorjMcKie]

I had updated the posted script, and you probably used the version before that.

[qwertynik]

@JorjMcKie Yes, that's most likely the case.
While exploring methods here, came across a utility called Ghostscript with a command-line utility that allows passing an option to convert text to Black in a PDF. Looks equally impressive. Given your involvement with PDFs, posting here in case it helps trigger some ideas.

[qwertynik]

The point is that XObjects may have their own content stream and also contain text. This text's color will not be touched by above logic. But one can also walk through the xobject streams and do the same type of thing with those.

@JorjMcKie Curious to know if this snippet replaces the relevant portions in the content stream of these XObjects.

[JorjMcKie]

The meaning of all those cryptic operators can be looked up in the PDF spec in chapter "Appendix A: Operator Summary", page 985 (old version) or 643 (new version).
The mini-language for them treats spaces and linebreaks equivalently as token delimiters, so b"ET 0.9 g 0.9 G" is the same as b"ET\n0.9g\n0.9 G", and multiples of each are treated as one.
Another property of this language is that the operators (functions) are preceeded by their arguments: 1 2 3 4 5 6 cm is nothing else than fitz.Matrix(1, 2, 3, 4, 5, 6) in PyMuPDF.

[qwertynik]

That helps in understanding the PDF format better. Thanks for adding the info @JorjMcKie

[JorjMcKie]

MuPDF and Ghostscript are both products of Artifex. When they acquired the "Fitz" project, their original plan was to replace Ghostscript with it, but changed their mind and Fitz became the product MuPDF.
So they have two PDF renderers now. Undoubtedly MuPDF is easily embeddable in other applications (PyMuPDF being a living proof) and Ghostscript definitely is not.
Both have a large overlap, but each also offers functionality not contained in the other.

[qwertynik]

In the back of my mind, there seemed to be some connection between the two. And indeed there is. Good to know the close association of the two. Hopefully, mupdf team comes with a simpler approach to deal with this use case. Of course, it might be doing the same as the script you posted here does.

[JorjMcKie]

Once you have located an XObject that the page invokes, the same code snippet can be used for its stream. An XObject is a stream object. So this goes along the logic:

page.clean_contents()  # clean the page /Contents, but also each XObject!
xobjects = page.get_xobjects()  # again each item has an xref as first subitem
for xobj in xobjects:
    xref = xobj[0]
    lines = doc.xref_stream(xref).splitlines()
    # etc.: logic like above
    doc.update_stream(xref, b"\n".join(lines)
# that's about it!

[JorjMcKie]

So maybe create a Python function that accepts and returns a curated stream, or even receives an xref representing a stream object and curates its contents as described.

[qwertynik]

Once you have located an XObject that the page invokes

Can you confirm if this line of code xobjects = page.get_xobjects() would be taking care of locating XObjects?

Upgraded the previous script for xobject.

import fitz

doc = fitz.open("pdfs/RPHT1.pdf")
skips = (b"k", b"K", b"rg", b"RG", b"sc", b"SC", b"scn", b"SCN", b"gs", b"cs")


def change_text_color_to_black(xref, lines):
    for i in range(len(lines)):
        if lines[i].endswith(skips):
            lines[i] = b""
            continue
        if lines[i] == b"q":
            lines[i] = b"q 0.9 g 0.9 G"
        elif lines[i] == b"BT":
            lines[i] = b"BT 0 g 0 G"
        elif lines[i] == b"ET":
            lines[i] = b"ET 0.9 g 0.9 G"
    doc.update_stream(xref, b"\n".join(lines))


for page in doc:
    page.clean_contents()
    xref = page.get_contents()[0]
    lines = page.read_contents().splitlines()
    change_text_color_to_black(xref, lines)

    # Find xobject and change text color to black
    xobjects = page.get_xobjects()
    for xobj in xobjects:
        xref = xobj[0]
        lines = doc.xref_stream(xref).splitlines()
        change_text_color_to_black(xref, lines)

doc.ez_save("output/x.pdf", pretty=True)

Strangely this sometimes fails with an exit code of -1073740940 (0xC0000374)

[JorjMcKie]

shouldn't happen - can you once again send me the file?

[qwertynik]

Sure, sent it now @JorjMcKie

[JorjMcKie]

This failure is a storage violation inside PyMuPDF. This only happens intermittently and only if the document is deleted by garbage collection (usually automatically at end of script) while there still exists at least one page object.

As an intermediate solution for you, do one of the following:

• issue an explicit doc.close()/del doc/doc = None.
• access the document as a context manager: this will automatically do a doc.close().
• explicitely remove any existing Page objects (again like page = None/del page)

[qwertynik]

@JorjMcKie

Skipped this approach

access the document as a context manager: this will automatically do a doc.close().

Because this would add another level of indentation in the document, or some other construct needs to be used.

issue an explicit doc.close()/del doc/doc = None.

doc.close() after save has eliminated the intermittent error.

Thanks!

[JorjMcKie]

Thanks for the confirmation!
Tracing shows that automatic garbage collection executes the removal of the document object without first removing any living page objects. This has not happened before, so I need to trace that down ...

[qwertynik]

Strange and interesting. This new adventure should make the library more robust I assume.

Emailing something related to PDF processing now, not sharing here since it would shift attention.

[JorjMcKie]

As per the error I reported:
Was a hard one! The reason is that Python garbage collection at the end of the program / script calls the __del__ methods of still existing objects in arbitrary sequence ... and apparently asynchronously. This can lead to removing the document before living page objects are deleted. If in that case page.__del__() is invoked, dropping the page shadow object inside MuPDF may reference an already invalid document storage address. This is why the phenomenon is so intermittent.
Anyway, I am now confirming the document is still there in page.__del__().
Otherwise just do not try page removal and accept the consequential memory leakage - which will anyway exist for a small amount of time only.
If a script wellbehaves, and either actively closes a document or removes no longer needed pages, the problem will not occur.
There are Linux and OSX pre-version wheels in this repo.

[qwertynik]

Was a hard one! The reason is that Python garbage collection at the end of the program/script calls the del methods of still existing objects in arbitrary sequence ... and apparently asynchronously. This can lead to removing the document before living page objects are deleted. If in that case page.del() is invoked, dropping the page shadow object inside MuPDF may reference an already invalid document storage address. This is why the phenomenon is so intermittent.

That's insightful to know @JorjMcKie. Thanks for sharing the inner workings of Python based on your deep dive - definitely helps improve understanding of the language.

There are Linux and OSX pre-version wheels in this repo.
Thanks for the link. Will attempt using the latest version as early as possible.

Seeing 2 two version upgrades stemming from our discussions is great to look at and also shows the agility and speed with which the library is improving.