The list of cloned objects is too large while using PdfWriter with incremental=True

[m32]

Environment

Linux-6.1.0-31-amd64-x86_64-with-glibc2.36
pypdf==5.3.0, crypt_provider=('cryptography', '44.0.0'), PIL=11.1.0

Code

from pypdf import PdfReader, PdfWriter

reader = PdfReader('pdf-in.pdf')
writer = PdfWriter(reader, incremental=True)
with open("pdf-out.pdf", "wb") as fp:
    writer.write_stream(fp)

After running the above code, the generated pdf should add new objects starting from the element = /Size and starts adding from /Size+1.
What can be seen from the object number /Xref

[stefan6419846]

Thanks for the report. Does this have any side effects or is this just something which looks odd to you?

[m32]

The next execution of the same python file on the generated file generates a warning in the PdfWriter class constructor.
I haven't checked what the xref array looks like, it may have holes and I don't know what this could result in.

[stefan6419846]

Sorry, but I cannot reproduce this issue:

from pypdf import PdfReader, PdfWriter

reader = PdfReader('sample-files/002-trivial-libre-office-writer/002-trivial-libre-office-writer.pdf')
writer = PdfWriter(reader, incremental=True)
with open("pdf-out.pdf", "wb") as fp:
    writer.write_stream(fp)

reader = PdfReader('pdf-out.pdf')
writer = PdfWriter(reader, incremental=True)
with open("pdf-out2.pdf", "wb") as fp:
    writer.write_stream(fp)

does not yield any warnings or other output. Could you please check again whether this only occurs for specific files and provide a corresponding file?

[m32]

print('0')
import fpdf
doc = fpdf.FPDF()
#doc.pdf_version = options.pdf_version
doc.set_compression(0)
font = 'helvetica'

for i in range(2):
    doc.add_page()
    doc.set_font(font, '', 13.0)
    link = None
    doc.cell(w=75.0, h=22.0, align='C', text='Hello, world page=%d.' % i, border=0, new_x=fpdf.XPos.LEFT, new_y=fpdf.YPos.NEXT, link=link)
    if font != 'helvetica':
        doc.cell(w=75.0, h=22.0, align='C', text='ąćęłńóśżź', border=0, new_x=fpdf.XPos.LEFT, new_y=fpdf.YPos.NEXT)
doc.output('pdf.pdf')

from pypdf import PdfReader, PdfWriter

print('1')
#reader = PdfReader('sample-files/002-trivial-libre-office-writer/002-trivial-libre-office-writer.pdf')
reader = PdfReader('pdf.pdf')
writer = PdfWriter(reader, incremental=True)
with open("pdf-out.pdf", "wb") as fp:
    writer.write_stream(fp)

print('2')
writer = PdfWriter('pdf-out.pdf', incremental=True)
with open("pdf-out2.pdf", "wb") as fp:
    writer.write_stream(fp)

print('3')
writer = PdfWriter("pdf-out2.pdf", incremental=True)
with open("pdf-out3.pdf", "wb") as fp:
    writer.write_stream(fp)

print('4')

Output:

0
1
2
Object 11 0 not defined.
3
Object 11 0 not defined.
4

After using the file sample-files/002-trivial-libre-office-writer/002-trivial-libre-office-writer.pdf PdfWriter behaves perfectly - it creates a copy of the input file, hence the behavior is not revealed

[m32]

/Size of pdf.pdf is 11, so next object should start at 11 not 12:

pdf.pdf:

xref
0 11
0000000000 65535 f 
0000000009 00000 n 
0000000102 00000 n 
0000000205 00000 n 
0000000285 00000 n 
0000000413 00000 n 
0000000493 00000 n 
0000000621 00000 n 
0000000718 00000 n 
0000000805 00000 n 
0000000892 00000 n 
trailer
<<
/Size 11
/Root 2 0 R
/Info 10 0 R
...

pdf-out.pdf:

12 0 obj<<
/Type /XRef
/Size 13
/Root 2 0 R
/Filter /FlateDecode
/Index [ 3 1 5 1 ]
/W [ 1 4 1 ]
/Prev 948

[stefan6419846]

it creates a copy of the input file, hence the behavior is not revealed

This is why we ask for reproducing code and the code used by me has been what you included in your initial report.

Apart from this: I appreciate any further analysis or fix (through a PR) leading to this issue being resolved.