Skip to content

ROB: improve inline image extraction #2622

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 48 commits into from
May 27, 2024
Merged

ROB: improve inline image extraction #2622

merged 48 commits into from
May 27, 2024

Conversation

pubpub-zz
Copy link
Collaborator

closes #2598

@codecov Codecov
Copy link

codecov bot commented May 4, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.07%. Comparing base (c8d722c) to head (7be1fd6).
Report is 64 commits behind head on main.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #2622      +/-   ##
==========================================
+ Coverage   95.02%   95.07%   +0.04%     
==========================================
  Files          50       51       +1     
  Lines        8366     8490     +124     
  Branches     1674     1694      +20     
==========================================
+ Hits         7950     8072     +122     
- Misses        258      263       +5     
+ Partials      158      155       -3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@pubpub-zz
Copy link
Collaborator Author

new test file:
Pages.62.73.from.0560-22_WSP.Plan_July.2022_Version.1.pdf

@pubpub-zz pubpub-zz marked this pull request as draft May 4, 2024 14:47
@pubpub-zz
Copy link
Collaborator Author

pubpub-zz commented May 5, 2024 •
edited
Loading

use of:
https://github.com/mozilla/pdf.js/blob/master/test/pdfs/bug1065245.pdf
as test input
8 images all looking like:
img

@pubpub-zz
Copy link
Collaborator Author

pubpub-zz commented May 7, 2024 •
edited
Loading

new test:
https://github.com/mozilla/pdf.js/blob/master/test/pdfs/bug1065245.pdf

image 0 :
img

@pubpub-zz pubpub-zz mentioned this pull request May 8, 2024
Closed
@pubpub-zz
Copy link
Collaborator Author

@stefan6419846, @MartinThoma, @MasterOdin
In order to improve test coverage, I'm looking for PDF files with inline images. Can you provide me with some ?

@stefan6419846
Copy link
Collaborator

It is rather unlikely that I am able to provide further (real) files for testing. While I have access to lots of PDF files in theory, most of them are confidential. Additionally, the usual routines I use completely omit any inline images as they provide no value for my use cases.

If possible, I would indeed prefer to avoid two different implementations of the same filter algorithms - we already have sufficient coverage for the filters outside the inline image extraction and thus re-using them would make more sense and avoid larger coverage issues.

@pubpub-zz
Copy link
Collaborator Author

@stefan6419846
thanks for trying.
There is not really changes in the filter/image processing. The change only applies to improve data extraction from contents.
Will try to find a way to generate some data.

@pubpub-zz
Copy link
Collaborator Author

add test file for RL encoding:
RL.pdf
image:
RL

stefan6419846
stefan6419846 reviewed May 20, 2024
View reviewed changes
@stefan6419846
Copy link
Collaborator

Besides the above remarks, there are two bigger issues I would like to talk about:

  • Importing pypdf._xobj_image_helpers without Pillow is marked as deprecated. Why? Does this mean you want to completely drop support for installations without Pillow?
  • Could we please use more verbose names for the filter methods while using all-lowercase function names as usually recommended by PEP8?

pubpub-zz and others added 7 commits May 20, 2024 10:36
@pubpub-zz @stefan6419846
Update pypdf/_page.py
2874e56 
Co-authored-by: Stefan <96178532+stefan6419846@users.noreply.github.com>
@pubpub-zz @stefan6419846
Update pypdf/_page.py
81e1f30 
Co-authored-by: Stefan <96178532+stefan6419846@users.noreply.github.com>
@pubpub-zz @stefan6419846
Update pypdf/_page.py
90fe459 
Co-authored-by: Stefan <96178532+stefan6419846@users.noreply.github.com>
@pubpub-zz @stefan6419846
Update pypdf/_page.py
54e4c1d 
Co-authored-by: Stefan <96178532+stefan6419846@users.noreply.github.com>
@pubpub-zz @stefan6419846
Update pypdf/generic/_data_structures.py
d9841dd 
Co-authored-by: Stefan <96178532+stefan6419846@users.noreply.github.com>
@pubpub-zz @stefan6419846
Update pypdf/generic/_data_structures.py
ecdba02 
Co-authored-by: Stefan <96178532+stefan6419846@users.noreply.github.com>
@pubpub-zz
update from comments
ae9fdfc
@pubpub-zz pubpub-zz requested a review from stefan6419846 May 20, 2024 10:22
@stefan6419846 stefan6419846 merged commit 23a81ba into py-pdf:main May 27, 2024
16 checks passed
@pubpub-zz
Copy link
Collaborator Author

@stefan6419846
Can you tell me if I'm wrong. The import should be covered by test_image_without_pillow within test_filters.py,Am I wrong ?

@stefan6419846
Copy link
Collaborator

Yes, this is correct.

stefan6419846 added a commit that referenced this pull request Jun 23, 2024
@stefan6419846
REL: 4.3.0
d3ef5e5 
## What's new

### New Features (ENH)
- Accept ETen-B5 and UniCNS-UTF16 encodings (#2721) by @pubpub-zz
- Add decode_as_image() to ContentStreams (#2615) by @pubpub-zz
- context manager for PdfReader (#2666) by @tibor-reiss
- Add capability to set font and size in fields (#2636) by @pubpub-zz
- Allow to pass input file without named argument (#2576) by @pubpub-zz

### Bug Fixes (BUG)
- Fix deprecation for Ressources when using old constants (#2705) by @stefan6419846
- Fix images issue 4 bits encoding and LUT starting with UTF16_BOM (#2675) by @pubpub-zz
- Reading large compressed images takes huge time to process (#2644) by @snanda85
- Highlighted Text Cannot Be Printed (#2604) by @Nifury
- Fix UnboundLocalError on malformed pdf (#2619) by @farjasju

### Documentation (DOC)
- Various improvements on docstrings and examples by @j-t-1

### Robustness (ROB)
- Cope with missing Standard 14 fonts in fields (#2677) by @pubpub-zz
- Improve inline image extraction (#2622) by @pubpub-zz
- Cope with loops in Fields tree (#2656) by @pubpub-zz
- Discard /I in choice fields for compatibility with Acrobat (#2614) by @pubpub-zz
- Cope with some issues in pillow (#2595) by @pubpub-zz
- Cope with some image extraction issues (#2591) by @pubpub-zz

### Maintenance (MAINT)
- Deprecate interiour_color with replacement interior_color (#2706) by @j-t-1
- Add deprecate_with_replacement to PdfWriter.find_bookmark (#2674) by @j-t-1

### Code Style (STY)
- Change Link to be a non-markup annotation (#2714) by @j-t-1

[Full Changelog](4.2.0...4.3.0)
@stefan6419846
Copy link
Collaborator

@pubpub-zz I have stumbled upon some PDF files which I see side effects of this PR with in the meantime - there is not explicit inline image parsing involved, but apparently some bad interaction. See #2927 for some more details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stefan6419846 stefan6419846 stefan6419846 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

ValueError: invalid literal for int() with base 16: b'F:'
2 participants
@pubpub-zz @stefan6419846