Skip to content

Conversation

@dendenxu
Copy link

Add part_indices for the Python File API

This can specify the specific parts of a multi-part EXR file to be loaded to avoid excessive IO when only some parts of the file are needed by the program. This functionality could be particularly useful when the file is very large, and only a very small portion of it is required in memory.

Test EXR file:
https://drive.google.com/file/d/1A0ciiL0BeF0RA6Do-8sd81slqdckdP0p/view?usp=sharing

Test script:

# Load only the first part
# 100 loops, best of 5: 7.56 msec per loop
python -m timeit -n 100 -s "import OpenEXR" "f = OpenEXR.File('000000.exr', part_indices=[0])"

# Load the full EXR
# 1 loop, best of 5: 1.7 sec per loop
python -m timeit -n 1 -s "import OpenEXR" "f = OpenEXR.File('000000.exr')"

This commit fixes the issue where an unfinished multi-part file could
not be loaded properly in the python warpper for OpenEXR. This happens
when headers for the multi-part file are written to disk but the actual
pixels never finish being written.

The current fix skips the corrupted or empty parts with no pixel data
while retaining the ability for loading the correct ones.

Signed-off-by: dendenxu <zhenx@zju.edu.cn>
This can specify the specific parts of a multi-part EXR file to be
loaded to avoid excessive IO when only some parts of the file is needed
by the program.

This functionality could be particular useful when the file is very
large and only a very small portion of it is required in memory.

Signed-off-by: dendenxu <zhenx@zju.edu.cn>
@cary-ilm
Copy link
Member

This looks fine, but I'd like to have a test to validate the behavior. Can you add a test to src/wrappers/python/tests/test_unittest.py?

Your test file is 268Mb, that's too big to add to the test suite. Either come up with a much smaller file, or just have the test generate a file on the fly and then test reading it.

@dendenxu
Copy link
Author

Make sense! The large test file is mainly there to show the difference in IO performance between naively reading all parts or doing so more selectively. I'll let the test suite randomly generate a test file to test this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants