pypdf.errors.PdfReadError: font not set: is PDF missing a Tf operator

[blushingpenguin]

Environment

Which environment were you using when you encountered the problem?

$ python -m platform
Linux-6.8.0-51-generic-x86_64-with-glibc2.39

$ python -c "import pypdf;print(pypdf._debug_versions)"
pypdf==5.1.0, crypt_provider=('cryptography', '43.0.3'), PIL=none

Code + PDF

This is a minimal, complete example that shows the issue:

from pypdf import PdfReader
temp_file = "/home/mark/test.pdf"
reader = PdfReader(temp_file)
for page in reader.pages:
    print(page.extract_text(extraction_mode="layout") + "\n")

I'm sorry -- I can't share this pdf file as it's a) sensitive and b) customer data. I can provide debug traces if those help. Also if there's a good way of anonymizing the data in the pdf without rewriting it then please let me know and I'll attempt that.

The PDF appears to have been generated by this library: https://bfo.com/products/report/?version=work-20200610T1518-r36819M according to the metadata

My reading of this problem is that extract_text in layout mode ends up calling down into text_show_operations, then in this particular pdf the Tf operator is ignored -- it seems the code is expecting it to be between a q/Q or BT/ET pair, which in this particular pdf it isn't. (It may be that this is a naughty pdf, but Okular and the Mozilla built in pdf reader both handle it ok).

from logging I added:
text_show_operations b'K', [0, 0, 0, 1]
text_show_operations b'rg', [0.341176, 0.392157, 0.462745]
text_show_operations b'Tf', ['/R1', 22] <--- ignored
text_show_operations b'BT', [] <-- some text ops with no font = boom

adding a couple of lines to handle Tf to text_show_operations fixes this:

•        elif op == b"Tf":
  

•            state_mgr.set_font(fonts[operands[0]], operands[1])
         else:  # set Tc, Tw, Tz, TL, and Ts if required. ignores all other ops
             state_mgr.set_state_param(op, operands)
  

and the output text looks to be about right.

However I'm not entirely sure if this is an appropriate fix (if it seems right, I'm happy to open a PR for it if that helps).

Traceback

Traceback (most recent call last):
  File "/home/mark/dev/vendeq-pdf-api/test.py", line 5, in <module>
    print(page.extract_text(extraction_mode="layout") + "\n")
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mark/dev/vendeq-pdf-api/.venv/lib/pypy3.10/site-packages/pypdf/_page.py", line 2368, in extract_text
    return self._layout_mode_text(
  File "/home/mark/dev/vendeq-pdf-api/.venv/lib/pypy3.10/site-packages/pypdf/_page.py", line 2261, in _layout_mode_text
    bt_groups = _layout_mode.text_show_operations(
  File "/home/mark/dev/vendeq-pdf-api/.venv/lib/pypy3.10/site-packages/pypdf/_text_extraction/_layout_mode/_fixed_width_page.py", line 279, in text_show_operations
    bts, tjs = recurs_to_target_op(
  File "/home/mark/dev/vendeq-pdf-api/.venv/lib/pypy3.10/site-packages/pypdf/_text_extraction/_layout_mode/_fixed_width_page.py", line 171, in recurs_to_target_op
    _tj = text_state_mgr.text_state_params()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mark/dev/vendeq-pdf-api/.venv/lib/pypy3.10/site-packages/pypdf/_text_extraction/_layout_mode/_text_state_manager.py", line 92, in text_state_params
    raise PdfReadError(
pypdf.errors.PdfReadError: font not set: is PDF missing a Tf operator?

[stefan6419846]

Thanks for the report. Such issues are hard to follow without access to an actual PDF file showing this behavior, although redacting confidential information most likely requires modifying the file on the internal operator level.

Feel free to submit a corresponding PR. Ideally, you are able to provide a test there as well which shows the issue an in isolated manner - I suspect that this should be doable with the original PDF file you have without giving away confidential information.

[blushingpenguin]

test-anon.pdf

I've attached an anonymized version of the pdf that shows the error (it's the source PDF, but I wrote something to go through the file and replace the text in the TJ operators with "*", the images with a 1x1 pixel, and the metadata with some sample data out of the spec). It still seems to render ok in Okular + triggers the bug.

If my fix looks ok I'm happy to open a PR and add this as a test case.

[stefan6419846]

Thanks for further looking into this and generating a suitable PDF file.

I just did some local checks with your proposed change and evaluated it against the detailed output of mupdf (mutool convert -o out.stext test-anon.pdf). As far as I can see, setting the font size to 22 here is correct:

<?xml version="1.0"?>
<document>
<page id="page1" width="612" height="792">
<block bbox="40.8 35.46 99.16599 65.754">
<line bbox="40.8 35.46 99.16599 65.754" wmode="0" dir="1 0">
<font name="Helvetica-Bold" size="22">
<char quad="40.8 35.46 49.357999 35.46 40.8 65.754 49.357999 65.754" x="40.8" y="59" color="#566475" c="*"/>
<char quad="49.358 35.46 57.916 35.46 49.358 65.754 57.916 65.754" x="49.358" y="59" color="#566475" c="*"/>
<char quad="57.036 35.46 65.594 35.46 57.036 65.754 65.594 65.754" x="57.036" y="59" color="#566475" c="*"/>
<char quad="64.934 35.46 73.492 35.46 64.934 65.754 73.492 65.754" x="64.934" y="59" color="#566475" c="*"/>
<char quad="73.492 35.46 82.049999 35.46 73.492 65.754 82.049999 65.754" x="73.492" y="59" color="#566475" c="*"/>
<char quad="82.049999 35.46 90.607997 35.46 82.049999 65.754 90.607997 65.754" x="82.049999" y="59" color="#566475" c="*"/>
<char quad="90.607997 35.46 99.16599 35.46 90.607997 65.754 99.16599 65.754" x="90.607997" y="59" color="#566475" c="*"/>
</font>
</line>
</block>
<block bbox="40.8 70.47502 127.91099 136.84096">
<line bbox="40.8 70.47502 127.91099 82.84103" wmode="0" dir="1 0">
<font name="Helvetica" size="9">
...

b'K' [0, 0, 0, 1]
b'rg' [0.341176, 0.392157, 0.462745]
b'Tf' ['/R1', 22]
b'BT' []
b'q' []
b'Tf' ['/R3', 9]
b'BT' []
b'q' []
b'rg' [0.2, 0.2, 0.2]
b'Tf' ['/R1', 9]
b'BT' []
b'q' []
b'q' []
b'k' [0, 0, 0, 1]
b'Tf' ['/R1', 8]
b'BT' []
b'q' []
b'BT' []
b'q' []
b'BT' []
b'q' []
b'BT' []
b'q' []
b'BT' []

I appreciate a corresponding PR.