"Too many open files" when trying to slice a very large image using crop

[Marchiuzzz]

What did you do?

Tried slicing satellite image of 10 000 x 10 000 px (150mb) image to 100x100px images using crop.

What did you expect to happen?

Images to get sliced without errors

What actually happened?

After exactly 8187 correctly sliced and saved images I get a "Too many open files" error

What are your OS, Python and Pillow versions?

• OS: Windows 10
• Python: 3.10.5
• Pillow: 9.1.1

from PIL import Image
from itertools import product
import os
def tile(filename, dir_in, dir_out, d):
    name, ext = os.path.splitext(filename)
    with Image.open(os.path.join(dir_in, filename)) as img:
        w, h = img.size

        grid = product(range(0, h-h % d, d), range(0, w-w % d, d))
        for i, j in grid:
            box = (j, i, j+d, i+d)
            out = os.path.join(dir_out, f'{name}_{i}_{j}{ext}')
            temp = img.crop(box)
            temp.save(out)
            temp.close()
        img.close()

tile('image.tif', './orig', './sliced', 100)

I cannot upload the image used because it is protected by a signed agreement that I would not share the image. Any guidance to the right direction would be much appreciated.

[radarhere]

I've created a simple 10000 by 10000 image - image.tif.zip

Testing on my macOS machine, I'm unable to replicate your error.
Using GitHub Actions, I tested this on Windows. I was still unable to replicate your error - https://github.com/radarhere/Pillow/actions/runs/3295763124/jobs/5434643216. There is an ValueError: Operation on closed image message, but that is because you are using both img.close() and a context manager. If you choose only one, that should be resolved.

Your error shouldn't have anything to do with your specific image. Could you download the zipped TIFF that I have attached, try that, and see if your error persists?

Could you provide the full traceback of the error?

[Marchiuzzz]

I've tried with other 10000 x 10000 px which had much smaller file size and it worked fine as well, the problem is somehow connected to the file being large (over 100mb). Here's the trace

I've also tried running this, though I'm not sure if the info it provides is relative

import psutil
proc = psutil.Process()
print(proc.open_files())

The output is this

[

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\renderer6.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\exthost6\\output_logging_20221020T213106\\4-IntelliCode API Examples.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_4_20221018T092520\\extensions.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_4_20221018T122436\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_3_20221019T132419\\extensions.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_5_20221018T191811\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\renderer5.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\renderer1.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_5_20221018T191805\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_4_20221014T135510\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\renderer4.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\exthost6\\output_logging_20221020T213106\\11-ESLint.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_3_20221014T101605\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_1_20221017T173919\\extensions.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_4_20221014T164956\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\main.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\exthost6\\output_logging_20221020T213106\\12-Markdown Language Server.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_1_20221019T132405\\extensions.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_4_20221014T135516\\extensions.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\extensionTelemetry.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_3_20221014T101017\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_4_20221018T092520\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_3_20221014T101605\\extensions.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\exthost6\\output_logging_20221020T213106\\9-API Usage Examples LSP.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\exthost6\\vscode.microsoft-authentication\\Microsoft Authentication.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_5_20221018T191805\\extensions.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_6_20221020T213038\\extensions.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_1_20221017T100420\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\exthost6\\exthost.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_4_20221017T173326\\extensions.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_1_20221019T132405\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_4_20221017T095541\\extensions.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\.ipython\\profile_default\\history.sqlite', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_3_20221014T135346\\extensions.log', fd=-1), 

popenfile(path='C:\\Windows\\System32\\en-US\\KernelBase.dll.mui', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_4_20221016T213713\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_4_20221019T145449\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_3_20221014T101006\\extensions.log', fd=-1), 

popenfile(path='C:\\Windows\\System32\\en-US\\kernel32.dll.mui', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_4_20221017T210557\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\editSessions.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_3_20221019T132419\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_4_20221017T173326\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_1_20221017T100420\\extensions.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_1_20221014T100547\\extensions.log', fd=-1), 

popenfile(path='C:\\Windows\\System32\\en-US\\tzres.dll.mui', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\renderer3.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\userDataSync.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\exthost6\\output_logging_20221020T213106\\10-Python Language Server.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_4_20221014T135516\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_3_20221014T101033\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_4_20221014T164956\\extensions.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\exthost6\\output_logging_20221020T213106\\8-Python Test Log.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_1_20221019T082146\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_3_20221014T101033\\extensions.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\exthost6\\output_logging_20221020T213106\\5-VS IntelliCode.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_3_20221014T101006\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_3_20221014T135431\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\exthost6\\vscode.github-authentication\\GitHub Authentication.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_4_20221019T145449\\extensions.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_3_20221014T135431\\extensions.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\exthost6\\output_logging_20221020T213106\\7-Python.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_1_20221014T100547\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_4_20221018T122436\\extensions.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\telemetry.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_4_20221017T210557\\extensions.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_4_20221014T135510\\extensions.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_3_20221014T135346\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_6_20221020T213038\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\exthost6\\output_logging_20221020T213106\\3-Prettier.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_5_20221018T191811\\extensions.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_1_20221019T082146\\extensions.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_1_20221017T173919\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_3_20221014T101017\\extensions.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_4_20221016T213713\\extensions.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\sharedprocess.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\exthost6\\output_logging_20221020T213106\\2-Git.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\exthost6\\output_logging_20221020T213106\\6-Jupyter.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\output_4_20221017T095541\\tasks.log', fd=-1), 

popenfile(path='C:\\Users\\Myuser\\AppData\\Roaming\\Code\\logs\\20221014T100544\\exthost6\\output_logging_20221020T213106\\1-Auto Rename Tag.log', fd=-1)]

So perhaps it's an issue with log files?

[radarhere]

The traceback tells me that your TIFF images are being saved with compression. By using a compressed TIFF as the original image, I was able to replicate the error on Windows.

So, an immediate solution would be to either

• change temp.save(out) to temp.save(out, compression=None)
  or
• save the images in a format other than TIFF.

[Marchiuzzz]

@radarhere That worked! Thank you so much, I've spent so long trying to figure it out and apparently it was a matter of one extra parameter...

[radarhere]

This has been re-opened as #6985

[radarhere]

This should now be fixed with compression, thanks to #6986