Repeating characters

[samkit-jain]

I'm facing a weird problem wherein characters are repeated when using extract_text() or extract_tables(). Example, SSttaatteemmeenntt ooff AAccccoouunnttss is printed instead of Statement of Accounts.

Sometimes, it happens in a portion of the PDF and sometimes in the whole PDF. When this happens in a portion of PDF, it is fixable (not completely) via extract_text(x_tolerance=0, y_tolerance=0) but not when the issue affects the whole PDF. Also, note that I do not face this issue in all PDFs but in some.

Lines are also repeated. Example,

Year-to-date totals do not reflect any fee or interest refunds
Year-to-date totals do not reflect any fee or interest refunds
you may have received.
you may have received.

[samkit-jain]

On doing first_page.extract_words(x_tolerance=0, y_tolerance=0), there are two instances of a single word

{'x0': Decimal('231.532'), 'x1': Decimal('252.251'), 'top': Decimal('916.343'), 'bottom': Decimal('925.422'), 'text': 'reflect'}
{'x0': Decimal('231.533'), 'x1': Decimal('252.252'), 'top': Decimal('916.383'), 'bottom': Decimal('925.462'), 'text': 'reflect'}

And repeating characters are still present for some words,

{'x0': Decimal('489.040'), 'x1': Decimal('506.160'), 'top': Decimal('269.320'), 'bottom': Decimal('277.480'), 'text': 'ttooddaayy'}

[jsvine]

That's strange, indeed. My hunch is that there really are two copies of each letter in the PDF. (One set of letters might be transparent, perhaps?) What happens if you try extracting the text with another tool, such as poppler-utils's pdftotext? (https://en.wikipedia.org/wiki/Pdftotext)

[samkit-jain]

No such problem with pdftotext. This is the output,

No repeating lines

Year-to-date totals do not reflect any fee or interest refunds
you may have received.

No repeating characters

today

Statement of Accounts

[jsfenfen]

I've encountered this problem as well. In my case it was cropping up in fillable pdfs, and I theorized that the folks filling out the pdf were somehow resaving it on top of the original text. I found it was easier to just remove duplicate characters via script than make sense of the pdf. I dunno for sure, I suspect that other pdf output tools are removing duplicate characters.

I'm not really sure what the right solution is, but possibly adding a 'remove duplicate characters' option would make this more manageable? My case involved exact matches--characters occurring at exactly the same spot--so a fix was easy... I suppose if they were slightly offset it would be more challenging.

[NaveenBandi]

Getting same issue, please pass some resolution

[BryanKoo]

AFAIK, duplicated characters are also for bold representation and there will be cases with small offset.
Deduplication is possible by checking overlap ratio of all characters using coordinates.

[tiagosamaha]

Any solution to it? I have the same issue.

[hannylicious]

I recently stumbled across this issue - just tossing it out there to let folks know it's a continuing thing.

[samkit-jain]

@hannylicious and other watchers of this issue, if you have a PDF with this issue that you can share publicly, please do so that this issue can be investigated in further detail.

I am pretty sure I have a PDF with this issue but it will take me some time to find it.

[hannylicious]

Unfortunately - I dabble with PDF's very infrequently. I just happened across it this time because another library (pyPDF2) didn't see any text at all - whereas pdfplumber saw the text, but it was duplicated. The PDF I'm working with at this time has some information that I can't publicly display so I won't be of much assistance I'm afraid.

I resolved my use case simply by grabbing the first of the results and using that.

Pdfplumber is a great tool - I will most likely be using this from now on! If I run across this issue on a PDF that I can link up - I definitely will!

[jsvine]

Thanks, @hannylicious! If you have the time, you could try using https://github.com/JoshData/pdf-redactor to remove the sensitive information without altering the PDF structure. If the result still produces the same character-duplication, then it could be very helpful for resolving this issue.

[hannylicious]

Thanks @jsvine - I will definitely have a look at that pdf-redactor library. If it works - I'll be sure and post that PDF here!

[tiagosamaha]

I would like to help, but my file has confidential content. Anyone have some issue file?

[pajaskowiak]

Same issue here.