Too many open files -> Issue #123

andybuki · 2021-06-12T15:16:52Z

andybuki
Jun 12, 2021

Hi i have a question, how to solve it. I have a list of files (ca. 10000 files) and i'll try to create the same numbers of manifests from
this list. After ca. 1200 created manifest i get following issue:

Exception in thread "main" java.lang.IllegalStateException: failed to create a child event loop
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)
	at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:78)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:73)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:60)
	at io.vertx.core.net.impl.transport.Transport.eventLoopGroup(Transport.java:153)
	at io.vertx.core.impl.VertxImpl.<init>(VertxImpl.java:143)
	at io.vertx.core.impl.VertxImpl.vertx(VertxImpl.java:92)
	at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:40)
	at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:32)
	at io.vertx.core.impl.VertxFactoryImpl.vertx(VertxFactoryImpl.java:27)
	at io.vertx.core.Vertx.vertx(Vertx.java:75)
	at info.freelibrary.iiif.presentation.v3.io.Manifestor.<init>(Manifestor.java:26)
	at org.crossasia.manifest.IIIFPresentationDlllm.main(IIIFPresentationDlllm.java:46)
Caused by: io.netty.channel.ChannelException: failed to open a new selector
	at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:180)
	at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:146)
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:138)
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
	... 15 more
Caused by: java.io.IOException: Zu viele offene Dateien -> Too many open files
	at java.base/sun.nio.ch.EPoll.create(Native Method)
	at java.base/sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:79)
	at java.base/sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
	at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:178)
	... 19 more
Disconnected from the target VM, address: '127.0.0.1:46303', transport: 'socket'

Process finished with exit code 1

I use Linux and i settings it is possible to use up to 65000 files. Do you have any idea how
to convert all the files ones, without splitting the whole list into small parts.

Answered by ksclarke

Jun 13, 2021

Looking at the above example I'd say it's because you're opening a file stream at new FileInputStream(file) (inside the loop) but never closing it, so running out of file handles.

As a test, I used the following Bash script to create 5000 test JSON files:

#! /bin/bash

mkdir -p /tmp/test /tmp/test-out

for i in $(seq 1 5000);
do
  cp src/test/resources/cookbook/0001-mvm-image.json "/tmp/test/${i}.json";
done

Then I use the following code to read and write them:

final Manifestor manifestor = new Manifestor();

Files.list(Path.of("/tmp/test")).forEach(path -> {
    final File oldFile = path.toFile();
    final File newFile = new File("/tmp/test-out/" + FileUtils.stripExt(oldFile) + "-new.j…

View full answer

ksclarke · 2021-06-13T20:10:11Z

ksclarke
Jun 13, 2021
Maintainer

Hi, can you give me a little more info... are you using Manifestor's readManifest(File) or readManifest(File, Promise) or are you reading CSV files or something and creating the manifests from them? Can you provide some test code showing me how you're looping over all the files? Are you writing to JSON output files as you read them?

Would you also show me the output of ulimit -n on your Linux system (running it as the user that's running the program)?

1 reply

andybuki Jun 13, 2021
Author

Hi Kevin, thanks for the answer:
i use a list of json files (i added one example)
prepared_manifest.zip
I use Manifestor:

Manifestor manifestor = new Manifestor();
manifestor.write(manifest, newFile);

Here the main file:

package org.crossasia.manifest;

import info.freelibrary.iiif.presentation.v3.*;
import info.freelibrary.iiif.presentation.v3.io.Manifestor;
import info.freelibrary.iiif.presentation.v3.properties.*;
import info.freelibrary.iiif.presentation.v3.services.image.ImageService3;
import org.crossasia.manifest.attributes.DllmAttributes;
import org.crossasia.manifest.json.StaticJsonCaller;
import org.crossasia.manifest.metadata.*;
import org.crossasia.manifest.metadata.Date;
import org.json.JSONArray;
import org.json.JSONObject;
import org.json.JSONTokener;
import org.springframework.boot.autoconfigure.SpringBootApplication;

import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.*;

import static org.crossasia.manifest.ManifestDllmCollection.readJsonFromUrl;

@SpringBootApplication
public class IIIFPresentationDlllm  {

    private static final String SERVER = "https://iiif-content.crossasia.org/xasia/";
    private static final String THUMBNAIL_PATH = "/full/150,/0/default.jpg";
    private static final String MANIFEST_COLLECTION="dllm+dllm_";
    private static final String LOGO_LINK= "https://ngcs-beta.staatsbibliothek-berlin.de/dc/logo-small";

    public static void main(String[] args) throws IOException {

        String quote = "\u005c\u0022";
        File absolutePath = new File("/mnt/b-isiprod-udl.pk.de/itr/archive/dllm/presentation/splitter/");

        File dir = new File(String.valueOf(absolutePath));
        File[] filesInDir = dir.listFiles();
        int counter = 1;

        for (File file : filesInDir) {
            DllmAttributes dllmAttributes = new DllmAttributes();
            File created = new File("/mnt/b-isiprod-udl.pk.de/itr/archive/dllm/presentation/result2/");
            StringBuilder sb = new StringBuilder();
            JSONObject jsonObj = new JSONObject(new JSONTokener(new FileInputStream(file)));

            Manifestor manifestor = new Manifestor();

            StaticJsonCaller.staticJsonCaller(dllmAttributes, jsonObj);

            I18n i18n_title_Roman = getStringsLabel(dllmAttributes); // get Label
            Manifest manifest = new Manifest(String.valueOf(file), new Label(i18n_title_Roman));

            StaticFields.staticFields(counter, manifest); //all static fields
            counter++;

            //Metadata metadata_title = getMetadataTitlesRomanThai(dllmAttributes, manifest);
            Metadata metadata_title = Title.getMetadataTitle(dllmAttributes);
            Metadata metadata_language = Language.getMetadataLanguageRomanThai(dllmAttributes);

            Metadata metadata_documentsID = DocumentsID.getMetadataDocumentsID(dllmAttributes);
            Metadata metadata_documentsCodeNumber = CodeNumber.getMetadataDocumentsCodeNumber(dllmAttributes);
            Metadata metadata_documents_roll = DocumentsRoll.getMetadataDocumentsRoll(dllmAttributes);

            Metadata metadata_extent = ExtentMethod.getMetadataExtentMethod(dllmAttributes);
            Metadata metadata_description = Description.getMetadataDescription(dllmAttributes);

            Metadata metadataDllmOriginal = OriginalDllm.getMetadataDllmOriginal(dllmAttributes);
            Metadata metadata_place = Places.getMetadataPlaces(dllmAttributes);
            Metadata metadata_full_location_name = FullLocationName.getMetadataFullLocationName(dllmAttributes);

            Metadata metadata_has_colophon = HasColophon.getMetadataHasColophon(dllmAttributes);
            Metadata metadata_is_illustrated = IsIllustrated.getMetadataIsIllustrated(dllmAttributes);
            Metadata metadata_is_color= IsColor.getMetadataIsColor(dllmAttributes);
            Metadata metadata_bundle_id= BundleID.getMetadataBundleID(dllmAttributes);

            Metadata metadata_is_complete= StaticJsonCaller.getMetadataIsComplete(dllmAttributes);
            Metadata metadata_pages_count= PagesCount.getMetadataPagesCount(dllmAttributes);
            Metadata metadata_material= Material.getMetadataMaterial(dllmAttributes);
            Metadata metadata_location_types_name= LocationTypes.getMetadataLocationTypesName(dllmAttributes);

            Metadata metadata_latitude= LatLon.getMetadataLatitude(dllmAttributes);
            Metadata metadata_longitude= LatLon.getMetadataLongitude(dllmAttributes);
            Metadata metadata_keywords= Keywords.getMetadataKeywords(dllmAttributes);
            Metadata metadata_categories= Categories.getMetadataCategories(dllmAttributes);

            Metadata metadata_script= Script.getMetadataScript(dllmAttributes);
            Metadata metadata_index= Index.getMetadataIndex(dllmAttributes);
            Metadata metadata_legibilities= Legibilities.getMetadataLegibilities(dllmAttributes);
            Metadata metadata_conditions= Conditions.getMetadataConditions(dllmAttributes);

            Metadata metadata_date= Date.getMetadataDate(dllmAttributes);
            Metadata metadata_date_original= DateOriginal.getMetadataDateOriginal(dllmAttributes);


            ArrayList<Metadata> metadataArrayList = new ArrayList<>();

            metadataArrayList.add(metadata_title);
            metadataArrayList.add(metadata_language);
            metadataArrayList.add(metadata_documentsID);

            metadataArrayList.add(metadata_documentsCodeNumber);
            metadataArrayList.add(metadata_documents_roll);
            metadataArrayList.add(metadata_extent);
            metadataArrayList.add(metadata_place);

            metadataArrayList.add(metadata_full_location_name);
            metadataArrayList.add(metadataDllmOriginal);
            metadataArrayList.add(metadata_description);
            metadataArrayList.add(metadata_has_colophon);

            metadataArrayList.add(metadata_is_illustrated);
            metadataArrayList.add(metadata_is_color);
            metadataArrayList.add(metadata_bundle_id);
            metadataArrayList.add(metadata_is_complete);

            metadataArrayList.add(metadata_pages_count);
            metadataArrayList.add(metadata_material);
            metadataArrayList.add(metadata_location_types_name);

            metadataArrayList.add(metadata_latitude);
            metadataArrayList.add(metadata_longitude);
            metadataArrayList.add(metadata_keywords);
            metadataArrayList.add(metadata_categories);

            metadataArrayList.add(metadata_script);
            metadataArrayList.add(metadata_index);
            metadataArrayList.add(metadata_legibilities);
            metadataArrayList.add(metadata_conditions);

            metadataArrayList.add(metadata_date);
            metadataArrayList.add(metadata_date_original);

            Iterator<Metadata> iter = metadataArrayList.iterator();

            while (iter.hasNext()) {
                Metadata md = iter.next();
                if (md==null)
                    iter.remove();
            }
            manifest.setMetadata(metadataArrayList);

            String book_ID=dllmAttributes.getDocuments_id();
            String page_ID="484597";

            String MANIFEST_URI = SERVER + MANIFEST_COLLECTION + book_ID +page_ID  + "/manifest";
            String MANIFEST_THUMBNAIL_URI = SERVER + MANIFEST_COLLECTION + book_ID+"+"+ page_ID+   THUMBNAIL_PATH;

            ImageService3 manifestThumbService = new ImageService3(ImageService3.Profile.LEVEL_TWO, SERVER + MANIFEST_COLLECTION+ book_ID+"+"+ page_ID);

            //start adding images
            JSONArray pages = null;
            if (jsonObj.has("pages")) {
                pages = (jsonObj.getJSONArray("pages"));
            }

            int page_int = pages.length();
            String manifestID ="";
            String canvasID ="";
            String annoID ="";
            String imageID ="";
            String annoPageID ="";

            Canvas canvas = new Canvas("");
            ArrayList<Canvas> canvases = new ArrayList<Canvas>();
            AnnotationPage<PaintingAnnotation> annoPage = new AnnotationPage<>("");
            PaintingAnnotation anno = new PaintingAnnotation("",canvas);
            ImageContent imageContent = new ImageContent("");

            for (int j = 0; j < pages.length(); j++) {
                JSONObject pagesObj = (JSONObject) pages.get(j);

                String pages_position = "";
                String pages_image_file = "";
                String pages_id = "";
                String first_pages_id = "";
                String pages_document_id = "";
                int number_of_pages = pages.length();

                if (pagesObj.has("pages_position"))
                    pages_position = (String) pagesObj.get("pages_position").toString();

                if (pagesObj.has("pages_image_file"))
                    pages_image_file = (String) pagesObj.get("pages_image_file").toString();

                if (pagesObj.has("pages_id")) {
                    pages_id = (String) pagesObj.get("pages_id").toString();
                    first_pages_id = dllmAttributes.getDocuments_id();
                }

                if (pagesObj.has("pages_document_id"))
                    pages_document_id = (String) pagesObj.get("pages_document_id").toString();
                JSONObject json = null;
                try {
                    json = readJsonFromUrl("https://iiif-content.crossasia.org/xasia/dllm" + "+dllm_000" + pages_document_id + "+" + pages_id + "/info.json");

                    manifestID = SERVER + MANIFEST_COLLECTION + "000" + pages_document_id + "+" + pages_id;
                    canvasID = manifestID + "/canvas";
                    imageID = manifestID + "/full/full/0/default.jpg";
                    annoID = manifestID + "/annotation";
                    annoPageID = manifestID;

                    int weight = (int) json.get("width");
                    int height = (int) json.get("height");

                    canvas = new Canvas(canvasID).setWidthHeight(weight, height);
                    imageContent = new ImageContent(imageID).setWidthHeight((Integer) weight, (Integer) height);

                }catch (IOException e) {
                    System.out.println(e +" - "+ file.getName());
                }

                annoPage = new AnnotationPage<>(annoPageID);
                anno = new PaintingAnnotation(annoID, canvas);
                annoPage.addAnnotations(anno.setBody(imageContent).setTarget(canvasID));
                canvases.add(canvas.setPaintingPages(annoPage));
                manifest.setCanvases(canvases);

                MANIFEST_URI = SERVER + MANIFEST_COLLECTION + book_ID +pages_id  + "/manifest";
                MANIFEST_THUMBNAIL_URI = SERVER + MANIFEST_COLLECTION + book_ID+"+"+ pages_id+   THUMBNAIL_PATH;

                manifestThumbService = new ImageService3(ImageService3.Profile.LEVEL_TWO, SERVER + MANIFEST_COLLECTION+ book_ID+"+"+ pages_id);
                manifest.setThumbnails(new ImageContent(MANIFEST_THUMBNAIL_URI).setServices(manifestThumbService));
                manifest.setStart(new Start("https://iiif-content.crossasia.org/xasia/dllm/dllm_000"+book_ID+"+"+pages_id + "/canvas/"+pages_id));

            }
            //end adding images

            Provider provider = new Provider("https://example.org/about",
                    String.valueOf(new Label("en", String.valueOf(new Value(new I18n("en", "Test"))))));

            provider.setLogos(new ImageContent(LOGO_LINK).setServices(manifestThumbService));
            manifest.setProviders(provider);
            File newFile = null;
            newFile = new File(created + "/" + dllmAttributes.getDocuments_id() + ".json"/*file.getName()*/);
            manifestor.write(manifest, newFile);

        }
    }
    private static I18n getStringsLabel(DllmAttributes dllmAttributes) {
        ArrayList<String> titlesArrayList = new ArrayList<>();
        I18n i18n_title_Roman = new I18n("en", "");
        if (dllmAttributes.getDllm_title_roman()!=null) {
            for (int i = 0; i < dllmAttributes.getDllm_title_roman().length(); i++) {
                titlesArrayList.add(dllmAttributes.getDllm_title_roman().get(i).toString());
                i18n_title_Roman = new I18n("en", titlesArrayList);
            }
            return i18n_title_Roman;
        } else
            return i18n_title_Roman;
    }

}

If it is nessessary i can add a project on git.
Yes i am reading and writing in JSON files.

22:52:26_root@b-lx0147:/proc$ ulimit -n
65000

Let me know if you need extra information.

ksclarke · 2021-06-13T21:33:48Z

ksclarke
Jun 13, 2021
Maintainer

Looking at the above example I'd say it's because you're opening a file stream at new FileInputStream(file) (inside the loop) but never closing it, so running out of file handles.

As a test, I used the following Bash script to create 5000 test JSON files:

#! /bin/bash

mkdir -p /tmp/test /tmp/test-out

for i in $(seq 1 5000);
do
  cp src/test/resources/cookbook/0001-mvm-image.json "/tmp/test/${i}.json";
done

Then I use the following code to read and write them:

final Manifestor manifestor = new Manifestor();

Files.list(Path.of("/tmp/test")).forEach(path -> {
    final File oldFile = path.toFile();
    final File newFile = new File("/tmp/test-out/" + FileUtils.stripExt(oldFile) + "-new.json");
    final Manifest manifest = manifestor.readManifest(oldFile);

    manifestor.write(manifest, newFile);
});

It works. Yours is a little different because you're using a different JSON processing library but, unless that JSONObject is somehow closing that FileInputStream in code we can't see in the example, I'd bet that's it. You need to put that input stream into a variable and close it at the bottom of the for loop (before that exits -- or use a try-with-resources syntax). Also, you should move the Manifestor constructor outside the for loop. With it inside, it's creating a new instance each time the loop circles. It only needs to be created once, and then its read/write methods called against that single instance.

Hope that helps!
Kevin

5 replies

andybuki Jun 13, 2021
Author

Thanks a lot Kevin, i'll try tomorrow, and let you know.

ksclarke Jun 13, 2021
Maintainer

Fwiw, it looks like you're already invested in org.json's library but Jiiify Presentation comes with a JsonObject by virtue of using the Vert.x stack: https://vertx.io/docs/apidocs/io/vertx/core/json/JsonObject.html

I should probably put that in the documentation. Probably not useful for you since you're already using the other, but just thought I'd mention it.

andybuki Jun 14, 2021
Author

Hi Kevin, how you already said. I put manifestor outside for loop and it works.
Best regards
Andrey

ksclarke Jun 14, 2021
Maintainer

Hi Andrey, great! So, the problem was not the FileInputStream, but just the Manifestor?

andybuki Jun 14, 2021
Author

Yes first of all i want to try the easiest way just to put Manifestor outside the loop, and it was enough.
Best regards Kevin
and thanks for your help.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Too many open files -> Issue #123

{{title}}

Replies: 2 comments 6 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Too many open files -> Issue #123

andybuki Jun 12, 2021

Replies: 2 comments · 6 replies

ksclarke Jun 13, 2021 Maintainer

andybuki Jun 13, 2021 Author

ksclarke Jun 13, 2021 Maintainer

andybuki Jun 13, 2021 Author

ksclarke Jun 13, 2021 Maintainer

andybuki Jun 14, 2021 Author

ksclarke Jun 14, 2021 Maintainer

andybuki Jun 14, 2021 Author

andybuki
Jun 12, 2021

Replies: 2 comments 6 replies

ksclarke
Jun 13, 2021
Maintainer

andybuki Jun 13, 2021
Author

ksclarke
Jun 13, 2021
Maintainer

andybuki Jun 13, 2021
Author

ksclarke Jun 13, 2021
Maintainer

andybuki Jun 14, 2021
Author

ksclarke Jun 14, 2021
Maintainer

andybuki Jun 14, 2021
Author