Wrong BOM detection for strings starting with \x00\xfe\x00\xff

[gyrrhe]

I get an AssertionError when merging a PDF that contains both latin and Chinese characters. This happens with a real PDF I work with, but for this issue I've created a small PDF that triggers the error. Perhaps the fact that the PDF contains both latin and Chinese characters has nothing to do with the issue, but I suspect it does.

Environment

$ python3 -c "import pypdf;print(pypdf._debug_versions)"
pypdf==6.5.0, crypt_provider=('cryptography', '43.0.0'), PIL=11.1.0

Code + PDF

import pypdf

reader = pypdf.PdfReader("minimal.pdf")
writer = pypdf.PdfWriter()
for page in reader.pages:
    writerPage = writer.add_blank_page(reader.pages[0].mediabox.width, reader.pages[0].mediabox.height)
    writerPage.merge_page(page)

minimal.pdf

I do not know why this particular set of characters triggers the error, but it's the minimal set I could find that triggers it. Feel free to add the PDF to your tests.

Traceback

Traceback (most recent call last):
  File "/home/user/tmp/minimal.py", line 7, in <module>
    writerPage.merge_page(page)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/user/tmp/pypdf/_page.py", line 1068, in merge_page
    self._merge_page(page2, over=over, expand=expand)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/tmp/pypdf/_page.py", line 1147, in _merge_page
    page2content.operations.insert(
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/tmp/pypdf/generic/_data_structures.py", line 1410, in operations
    self._parse_content_stream(BytesIO(self._data))
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/tmp/pypdf/generic/_data_structures.py", line 1303, in _parse_content_stream
    operands.append(read_object(stream, None, self.forced_encoding))
                    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/tmp/pypdf/generic/_data_structures.py", line 1452, in read_object
    return ArrayObject.read_from_stream(stream, pdf, forced_encoding)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/tmp/pypdf/generic/_data_structures.py", line 261, in read_from_stream
    arr.append(read_object(stream, pdf, forced_encoding))
               ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/tmp/pypdf/generic/_data_structures.py", line 1450, in read_object
    return read_hex_string_from_stream(stream, forced_encoding)
  File "/home/user/tmp/pypdf/generic/_utils.py", line 35, in read_hex_string_from_stream
    return create_string_object(bytes(arr), forced_encoding)
  File "/home/user/tmp/pypdf/generic/_utils.py", line 168, in create_string_object
    retval = TextStringObject(string.decode("utf-16be"))
  File "/home/user/tmp/pypdf/generic/_base.py", line 673, in __new__
    assert org is not None, "mypy"
           ^^^^^^^^^^^^^^^
AssertionError: mypy

Typst code

The PDF I share above was created with the Typst typesetting system, version 0.14.2, from the following code:

#set document(date: none)
#set text(font: ("Noto Sans CJK SC"))

接司智对UQfiffI:'!'bSMEftN;Tpghlxo?Ldcrm,quva sine.取器效论证反结件以量说明讨用公高认带的起最准都你单和愿计兰它由，亲台本及集游术行具马扩充要读好您才蜂无击步方并急标是多哪该度去朋被小整任译形始成中之应种么识社织调放弃己基理次倒人预况有灵从经个康能建、手现同作慧队

划时但AA轮更力战付者关互AA文型强导工密们各关已（意络共关采利一体张关开关通确信关测关提关关模到立关语关关备这养入关压关第味程关）团群如另言动情；来规培关果没枪关过治自网。就参主地系德着解构做旦政重目感出切大协受令关想翻友关内诸比为把在相先可看机领

I don't think this bug comes from Typst creating a malformed PDF, but if you find that it does I'll be happy to create an issue on their side.

[stefan6419846]

Thanks for the report. This usually indicates that the corresponding string haas a strange prefix. You are of course invited to investigate this further, as my knowledge of non-Latin character sets and BOMs is rather limited.

[gyrrhe]

Thanks for your answer! I tried to investigate, but I'm no expert, so please correct me if I'm wrong. In the PDF I shared, a content stream contains:

BT
0 Tr
/f0 11 Tf
1 0 0 -1 0 0 Tm
[<00FE00FF>] TJ
ET

Decoding the <00FE00FF> string is what leads to the AssertionError.

In create_string_object in generic/_utils.py, and then in TextStringObject.__new__ in generic/_base.py, pypdf interprets these bytes as a BOM. But I think Typst actually meant to "display" a U+FEFF (ZERO WIDTH NO-BREAK SPACE) character code and did not mean for it to be interpreted as a BOM?

Do you think my reasoning is correct? And who is at fault, should pypdf interpret this differently or should Typst not produce this output?

[stefan6419846]

I am not completely sure about this - it would be required to further do some analysis. Especially, having a zero width space inside content of a PDF file does not sound correct directly. In general, #3470 could be the same, although it has no example PDF file and there have been some discussions about the sequence possibly being a BOM indicator (which I could not find any evidence of), but this does not seem to apply to your case directly.

Are all of your faulty examples plain 00FE00FF, or are there cases where this is just a prefix? For plain \x00\xfe\xff\x00, we could add an exception, although I am not sure about the general reasoning for this being a zero width space. The actual characters displayed should depend on the used font, and some quick testing did not indicate that decoding this as a variant of UTF-16 would work directly.

The "faulty" change most likely is 0c81f3c, but without any direct clues on what the correct output should be in this case, there is no viable way to solve this.

[gyrrhe]

Yes, I'm probably wrong about the zero width space interpretation. To answer your question, there are examples where 00FE00FF is just a prefix. For example, adding 枪 at the end of the Typst code I shared above generates a PDF with:

BT
0 Tr
/f0 11 Tf
1 0 0 -1 0 0 Tm
[<00FE00FF00D6>] TJ
ET

And the same error is triggered. I feel like the next step should be to understand why Typst outputs this? I'll try to investigate more.

[gyrrhe]

Alright, I think I got it! The error triggers with the following Typst code (I can send the PDF if needed):

#set document(date: none)
#set text(font: ("Noto Sans CJK SC"))

无译形带r的参z慧队手行c要枪互工先调uC一在你k该方导最xT况M味政没出v大同团想急压游这体构主m基重张预另做内已织程术并U种规被中应s过小立就公测和F更为BS把强型w利qfJ现能您关文）己个言VW是Z亲社y。说准密令K络通自力i诸旦明量放及I成战康养d都蜂多开pE次提朋动比台有培愿A确l充计标去人如么b灵N它g弃语看X；j轮HG采共由地友入（器Y果感建切理情从集德翻a单第识任Q模eh目经相哪受起时着DR用好o备划付信、度解效作协读O讨高具击始者意群治扩到P才兰网认t马倒来本整L们n系可论，步各之但

12

This results, again, in the PDF content stream containing the following:

BT
0 Tr
/f0 11 Tf
1 0 0 -1 0 0 Tm
[<00FE00FF>] TJ
ET

As you can see, in the document, there are exactly 253 different characters on the first line (no duplicates), and 2 characters (different from the first 253) on the second line. These last two characters, 1 and 2, get assigned the values 00FE (= 254) and 00FF (= 255) because they are the 254th and 255th unique characters to appear in the document, respectively. I assume this must come from Typst's font subsetting system.

In short, to trigger the error, the 254th and 255th unique characters to appear in the document must appear one after the other at the beginning of a line. It doesn't seem to matter what these characters are, I used 1 and 2 here but I could also have used Chinese characters.

[stefan6419846]

Could you please provide the example files for the last examples as well?

Looking into this, it seems like we can decode these characters as UTF_16BE:

>>> b'\x00\xfe'.decode('utf-16be')
'þ'
>>> b'\x00\xfe'.decode('utf-16le')
'︀'
>>> ord(b'\x00\xfe'.decode('utf-16be'))
254

At first sight, the string does not look like the BOM due to using two characters and in the standalone case are non-interpretable:

>>> b'\x00\xfe\x00\xff'.decode('utf-16be').startswith('\xfe\xff')
True

A possible approach would be to catch exact matches, which would solve the case from #3587 (comment). However, #3587 (comment) is still an issue afterwards.

The problem is the input for the check being a string, combined with .startswith(). This accepts invalid BOMs where each BOM character is written as two bytes, which is just wrong. Instead, we would have to check whether the first two bytes/characters of the string are matching, for example using o[:2] in {"\xfe\xff", "\xff\xfe"} instead of the current check.

Could you please check whether replacing

pypdf/pypdf/generic/_base.py

Line 672 in 97d47a0

if o.startswith(("\xfe\xff", "\xff\xfe")):

with the new condition solves your issue and yields correct results?

If yes, I will try to prepare a corresponding PR next week.

[gyrrhe]

Here's the file from my last example: minimal.pdf

In TextStringObject.__new__ in generic/_base.py, replacing:

if o.startswith(("\xfe\xff", "\xff\xfe")):

with:

if o[:2] in {"\xfe\xff", "\xff\xfe"}:

does not solve the issue, the AssertionError still triggers.

I'm really not sure but couldn't the condition rather be:

if bytes(o[:2], "utf-16") in {"\xfe\xff", "\xff\xfe"}:

?

[stefan6419846]

Thanks for testing. I wanted to avoid having to convert the input to bytes again.

I have opened a corresponding PR which only triggers the conversion if there indeed are original bytes and they start with the proper BOM, as this is the only case where converting bytes to an unicode string indeed make sense for me. Feel free to have a look at it before I merge it.

[gyrrhe]

I tested the PR and it fixes the issue and seems good! Thanks a lot of fixing this so quickly and more generally for maintaining pypdf.