Hidden text is being extracted even using scrub()

[victor-ab]

Describe the bug

The scrub() method is not completely sanitizing the pdf. It is still possible to extract hidden text from it.

To Reproduce

I can't share this PDF, but I guess it was generated in Excel.

doc = fitz.open(pdf_dir)
doc.scrub()
blocks = doc[0].getText("rawdict")["blocks"]

Check the X_0 coord for each char

	x_0	y_0	x_1	y_1	c
0	256.35	327.575	262.353	339.941	S
1	262.353	327.575	268.851	339.941	R
2	268.851	327.575	273.855	339.941	#
3	273.855	327.575	276.357	339.941
4	276.357	327.575	281.361	339.941	2
5	281.361	327.575	286.365	339.941	2
6	286.365	327.575	291.369	339.941	3
7	291.369	327.575	296.373	339.941	4
8	296.373	327.575	301.377	339.941	8
9	301.377	327.575	306.381	339.941	5
10	306.381	327.575	311.385	339.941	8
11	311.385	327.575	316.389	339.941	3
12	316.389	327.575	321.393	339.941	9
13	321.393	327.575	326.397	339.941	7
14	326.397	327.575	328.899	339.941
15	328.899	327.575	335.901	339.941	O
16	335.901	327.575	341.4	339.941	T
17	341.4	327.575	347.898	339.941	H
18	347.898	327.575	350.4	339.941
19	350.4	327.575	356.898	339.941	H
20	356.898	327.575	362.901	339.941	V
21	362.901	327.575	368.904	339.941	A
22	368.904	327.575	375.402	339.941	C
23	375.402	327.575	377.904	339.941
24	377.904	327.575	380.901	339.941	-
25	380.901	327.575	383.403	339.941
26	383.403	327.575	390.405	339.941	O
27	390.405	327.575	394.905	339.941	v
28	394.905	327.575	399.909	339.941	e
29	399.909	327.575	402.906	339.941	r
30	397.6	327.575	402.604	339.941	3
31	402.604	327.575	407.608	339.941	8
32	407.608	327.575	412.612	339.941	1
33	412.612	327.575	415.114	339.941	.
34	415.114	327.575	420.118	339.941	0
35	420.118	327.575	425.122	339.941	0

While .getText("dict") extracted it just as SR# 2234858397 OTH HVAC - Over381.00

Expected behavior

I expected the scrub() method to sanitize hidden text like this. Also, a new word block for "381.00" would be nice here.

Screenshots

Your configuration

3.7.7 (default, May  6 2020, 11:45:54) [MSC v.1916 64 bit (AMD64)] 
 win32 
 
PyMuPDF 1.17.6: Python bindings for the MuPDF 1.17.0 library.
Version date: 2020-08-26 14:54:32.
Built for Python 3.7 on win32 (64-bit).

[JorjMcKie]

Text may be invisible for a number of reasons.
Not all alternatives are detectable by this method. E.g. if text just happens to be hidden because something is positioned over it, then this cannot be detected. I only detect hidden text which is declared as such (i.e. PDF text rendering mode 3).
So in order to see what is actually happening in your case, I do need a reproducing file.

Also, a new word block for "381.00" would be nice here.

Don't understand this comment.

[JorjMcKie]

I understand better now, what you were pointing at:
Because the x0 of "381,00" is smaller than that of the "r" of "Over", you think that I should have deleted the "r"?
Or even everything after "HVA" because a white box is visible there?
If that's true, then it falls under the category I mentioned before: not doable by the method.

[victor-ab]

I see. In my view, everything after "HVA" should be excluded.
I guess the text wasn't declared as hidden. Is there a workaround for this?

I just reproduced the behaviour printing from excel.
test.pdf

There's a hidden text being extracted

[{'size': 11.019074440002441,
      'flags': 0,
      'font': 'F1',
      'color': 0,
      'text': 'long long long test',
      'origin': (52.80000305175781, 79.91999816894531),
      'bbox': (52.80000305175781,
       71.65569305419922,
       135.17018127441406,
       82.67476654052734)}]

Also, a new word block for "381.00" would be nice here.

I meant that in this extracted string "SR# 2234858397 OTH HVAC - Over381.00", the "381.00" was supposed to be separated in another span.
This wasn't reproduced in the pdf I sent you.

[JorjMcKie]

The fact that some text is overlapping some other text cannot in general be a criterion for regarding any of the participant as being hidden - it could be intentional after all.
Not even mentioning more complex situations - e.g. when the overlap is partial only: what threshold should determine deletion ...?

But even if you are able to formulate a set of rules covering your special situation, it is still impossible (or rather: not implemented anywhere in the base library MuPDF) to delete single selected characters while letting survive others occupying (parts of) the same space.

... the "381.00" was supposed to be separated in another span.

This is part of heuristics that MuPDF is using: A new span is created only, if between two encountered characters some aspect of their fonts has changed (either different font or just a different property for the same font: color, size, ...). If the y1 of their bboxes is different, a new line is started (automatically also starting a new span).
If neither of this is the case, the old span continues.

[JorjMcKie]

your example page contains 2 blocks with each having 2 lines with 1 span each:

>>> blocks=page.getText("dict")["blocks"]
>>> for block in blocks:
	for line in block["lines"]:
		for span in line["spans"]:
			print(span["text"])

			
test
test
long long long test
test

This shows that my previous criteria list of "what starts a new span" was incomplete:

• If the "source code" defining the page appearance (in some /Contents object) contains a forced new line, then this will be honored.
• If the distance between two consecutive characters is too large (in comparison to their fontsize), a new span is started, even if their fonts and y1 are the same

[victor-ab]

Got this. Everything makes more sense to me so far.

But what about this case? There's no overlap between texts.

test.pdf

doc = fitz.open('test.pdf')
blocks = doc[0].getText("dict")["blocks"]
for block in blocks:
    for line in block["lines"]:
        for span in line["spans"]:
            print(span["text"])
doc.close()

test
not overlapping long long long test
test

[JorjMcKie]

There's no overlap between texts.

Correct. But as I said: hiddenness detection is limited to the command 3 Tr (text rendering). There is no effort to re-invent the wheel by building yet another PDF interpreter. In this context, marked content controlled visibility (which is the case here) is ignored and respective text be extracted.
If MuPDF some day in the future may provide this kind of "visibility layer" information, I will certainly consider to reflect that in the appropriate extraction methods.
As per today, this is not the case. Even detecting 3 Tr is my own logic and as such represents some maintainability risk.
Depending on your inclination to adventures, you might want to try analyzing the MuPDF trace output. This contains some more details and conveniently is in XML format.
mutool trace test.pdf > test.xml delivers this:

<document filename="test.pdf">
<page number="1" mediabox="0 0 612 792">
<set_default_colorspaces gray="DeviceGray" rgb="DeviceRGB" cmyk="DeviceCMYK" oi="None"/>
<group bbox="0 0 612 792" isolated="1" knockout="0" blendmode="Normal" alpha="1">
    <layer name="P"/>
    <clip_path winding="eofill" transform="1 0 0 -1 0 792">
        <moveto x="51.36" y="723.48"/>
        <lineto x="293.93" y="723.48"/>
        <lineto x="293.93" y="737.04"/>
        <lineto x="51.36" y="737.04"/>
        <closepath/>
    </clip_path>
        <fill_text colorspace="DeviceGray" color="0" transform="1 0 0 -1 0 792">
            <span font="BCDEEE+Calibri" wmode="0" bidi="0" trm="11.04 0 0 11.04">
                <g unicode="t" glyph="410" x="52.8" y="726.6" adv=".33496095"/>
                <g unicode="e" glyph="286" x="56.498399" y="726.6" adv=".4975586"/>
                <g unicode="s" glyph="400" x="62.029439" y="726.6" adv=".39111329"/>
                <g unicode="t" glyph="410" x="66.34608" y="726.6" adv=".33496095"/>
            </span>
        </fill_text>
    <pop_clip/>
    <end_layer/>
    <layer name="P"/>
    <clip_path winding="eofill" transform="1 0 0 -1 0 792">
        <moveto x="51.36" y="708.94"/>
        <lineto x="105.264" y="708.94"/>
        <lineto x="105.264" y="722.524"/>
        <lineto x="51.36" y="722.524"/>
        <closepath/>
    </clip_path>
        <fill_text colorspace="DeviceGray" color="0" transform="1 0 0 -1 0 792">
            <span font="BCDEEE+Calibri" wmode="0" bidi="0" trm="11.04 0 0 11.04">
                <g unicode="n" glyph="374" x="52.8" y="712.06" adv=".5253906"/>
                <g unicode="o" glyph="381" x="58.56288" y="712.06" adv=".52734377"/>
                <g unicode="t" glyph="410" x="64.43616" y="712.06" adv=".33496095"/>
                <g unicode=" " glyph="3" x="68.13456" y="712.06" adv=".22607422"/>
                <g unicode="o" glyph="381" x="70.66272" y="712.06" adv=".52734377"/>
                <g unicode="v" glyph="448" x="76.535999" y="712.06" adv=".45166017"/>
                <g unicode="e" glyph="286" x="81.57024" y="712.06" adv=".4975586"/>
                <g unicode="r" glyph="396" x="87.06815" y="712.06" adv=".3486328"/>
                <g unicode="l" glyph="367" x="90.92111" y="712.06" adv=".22949219"/>
                <g unicode="a" glyph="258" x="93.46031" y="712.06" adv=".4790039"/>
                <g unicode="p" glyph="393" x="98.748477" y="712.06" adv=".5253906"/>
                <g unicode="p" glyph="393" x="104.48927" y="712.06" adv=".5253906"/>
                <g unicode="i" glyph="349" x="110.25215" y="712.06" adv=".22949219"/>
                <g unicode="n" glyph="374" x="112.79135" y="712.06" adv=".5253906"/>
                <g unicode="g" glyph="336" x="118.53215" y="712.06" adv=".47070313"/>
                <g unicode=" " glyph="3" x="123.68783" y="712.06" adv=".22607422"/>
                <g unicode="l" glyph="367" x="126.18287" y="712.06" adv=".22949219"/>
                <g unicode="o" glyph="381" x="128.72208" y="712.06" adv=".52734377"/>
                <g unicode="n" glyph="374" x="134.59537" y="712.06" adv=".5253906"/>
                <g unicode="g" glyph="336" x="140.35825" y="712.06" adv=".47070313"/>
                <g unicode=" " glyph="3" x="145.51393" y="712.06" adv=".22607422"/>
                <g unicode="l" glyph="367" x="148.00897" y="712.06" adv=".22949219"/>
                <g unicode="o" glyph="381" x="150.54817" y="712.06" adv=".52734377"/>
                <g unicode="n" glyph="374" x="156.42147" y="712.06" adv=".5253906"/>
                <g unicode="g" glyph="336" x="162.18434" y="712.06" adv=".47070313"/>
                <g unicode=" " glyph="3" x="167.34003" y="712.06" adv=".22607422"/>
                <g unicode="l" glyph="367" x="169.83507" y="712.06" adv=".22949219"/>
                <g unicode="o" glyph="381" x="172.37427" y="712.06" adv=".52734377"/>
                <g unicode="n" glyph="374" x="178.24756" y="712.06" adv=".5253906"/>
                <g unicode="g" glyph="336" x="184.01044" y="712.06" adv=".47070313"/>
                <g unicode=" " glyph="3" x="189.16613" y="712.06" adv=".22607422"/>
                <g unicode="t" glyph="410" x="191.66117" y="712.06" adv=".33496095"/>
                <g unicode="e" glyph="286" x="195.39269" y="712.06" adv=".4975586"/>
                <g unicode="s" glyph="400" x="200.89061" y="712.06" adv=".39111329"/>
                <g unicode="t" glyph="410" x="205.20725" y="712.06" adv=".33496095"/>
            </span>
        </fill_text>
    <pop_clip/>
    <end_layer/>
    <layer name="P"/>
    <clip_path winding="eofill" transform="1 0 0 -1 0 792">
        <moveto x="51.36" y="708.94"/>
        <lineto x="293.93" y="708.94"/>
        <lineto x="293.93" y="722.524"/>
        <lineto x="51.36" y="722.524"/>
        <closepath/>
    </clip_path>
        <fill_text colorspace="DeviceGray" color="0" transform="1 0 0 -1 0 792">
            <span font="BCDEEE+Calibri" wmode="0" bidi="0" trm="11.04 0 0 11.04">
                <g unicode="t" glyph="410" x="274.25" y="712.06" adv=".33496095"/>
                <g unicode="e" glyph="286" x="277.9484" y="712.06" adv=".4975586"/>
                <g unicode="s" glyph="400" x="283.47944" y="712.06" adv=".39111329"/>
                <g unicode="t" glyph="410" x="287.79609" y="712.06" adv=".33496095"/>
            </span>
        </fill_text>
    <pop_clip/>
    <end_layer/>
</group>
</page>
</document>

But again:
There is no way to selectively delete parts of this ... except you are willing to analyze and modify the page appearance creation syntax (which lead to the above XML):

>>> page=doc[0]
>>> page.cleanContents()
>>> print(page.readContents().decode())
q
/P <</MCID 0>> BDC  # <=== start layer MCID 0
51.36 723.48 242.57 13.56 re
W*
n
BT
/F1 11.04 Tf
1 0 0 1 52.8 726.6 Tm
/GS8 gs
0 g
/GS9 gs
0 G
0 G
[(te)-2.9999998(st)] TJ
ET
Q
EMC  # <=== end of layer MCID 0
q
/P <</MCID 1>> BDC  # <=== start layer MCID 1
51.36 708.94 53.904 13.584 re
W*
n
0 g
0 G
BT
/F1 11.04 Tf
1 0 0 1 52.8 712.06 Tm
0 G
[(n)2.9999998(o)-5.0000007(t )-2.9999998(o)-5.0000007(v)-4(erlap)5.0000007(p)2.9999998(in)5.0000007(g)4( lo)-5.0000007(n)2.9999998(g)4( lo)-5.0000007(n)2.9999998(g)4( lo)-5.0000007(n)2.9999998(g)4( t)-2.9999998(est)] TJ
ET
Q
EMC  # <=== end of layer MCID 1
q
/P <</MCID 2>> BDC  # <=== start layer MCID 2
51.36 708.94 242.57 13.584 re
W*
n
0 g
0 G
BT
/F1 11.04 Tf
1 0 0 1 274.25 712.06 Tm
0 G
[(te)-2.9999998(st)] TJ
ET
Q
EMC  # <=== end of layer MCID 2
q
Q
>>> 

[victor-ab]

That's quite a good start to dig deep into solving this. Not sure if it is worthy for my application, but I'll share if I find any workaround.

Anyway, thanks for helping me understanding this!
Have a good saturday!

[victor-ab]

@JorjMcKie, I just found a PDF with hidden text using white color. 'color': 16777215,

The funny part is when it overlaps with a black-colored text:

It seems like the white-colored text is behind, or maybe just transparent.

Got any ideas on how to detect this?
raw-Copy.pdf