Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rdf2hdt stops without error message #267

Open
KonradHoeffner opened this issue Dec 5, 2022 · 3 comments
Open

rdf2hdt stops without error message #267

KonradHoeffner opened this issue Dec 5, 2022 · 3 comments

Comments

@KonradHoeffner
Copy link

I am using rdf2hdt from within Docker on a 30 GB N-Triples file on a machine with 32 GB RAM and no swap.
RAM usage goes up until around 90% and hovers around that value for a few minutes and then rdf2hdt just exits without producing the output file and without warning.
The command I'm using is docker run -m 50G -it --entrypoint /bin/bash -v $PWD:/data hdt using the Dockerfile from this repository and I assume that the system being out of memory would result in the usual console message, so I think it is not getting hit by the Linux out of memory killer but I'm not sure.
Docker also does not give me any warning or error message.
How can I find out whether the RAM is actually the issue here or if there is something else?

@ate47
Copy link

ate47 commented Dec 5, 2022

It takes a lot of memory to generate an HDT file and 32GB of ram for a 30GB NT file is probably not enough and sometimes the process crash without any log. If you can't give more ram to you process, you have the Java version of this CLI with disk options to reduce your ram usage

@mielvds
Copy link
Member

mielvds commented Dec 5, 2022

It is probably the memory, especially with the sudden stop, but I don't know why the OS is not giving you feedback

@KonradHoeffner
Copy link
Author

Using memory compression via a 16 GB swap partition in zram solved the issue, but I still don't understand why there wasn't an error message so I leave it up to you whether you want to close this issue as solved or whether you want to investigate this further.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants