You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am using rdf2hdt from within Docker on a 30 GB N-Triples file on a machine with 32 GB RAM and no swap.
RAM usage goes up until around 90% and hovers around that value for a few minutes and then rdf2hdt just exits without producing the output file and without warning.
The command I'm using is docker run -m 50G -it --entrypoint /bin/bash -v $PWD:/data hdt using the Dockerfile from this repository and I assume that the system being out of memory would result in the usual console message, so I think it is not getting hit by the Linux out of memory killer but I'm not sure.
Docker also does not give me any warning or error message.
How can I find out whether the RAM is actually the issue here or if there is something else?
The text was updated successfully, but these errors were encountered:
It takes a lot of memory to generate an HDT file and 32GB of ram for a 30GB NT file is probably not enough and sometimes the process crash without any log. If you can't give more ram to you process, you have the Java version of this CLI with disk options to reduce your ram usage
Using memory compression via a 16 GB swap partition in zram solved the issue, but I still don't understand why there wasn't an error message so I leave it up to you whether you want to close this issue as solved or whether you want to investigate this further.
I am using rdf2hdt from within Docker on a 30 GB N-Triples file on a machine with 32 GB RAM and no swap.
RAM usage goes up until around 90% and hovers around that value for a few minutes and then rdf2hdt just exits without producing the output file and without warning.
The command I'm using is
docker run -m 50G -it --entrypoint /bin/bash -v $PWD:/data hdt
using the Dockerfile from this repository and I assume that the system being out of memory would result in the usual console message, so I think it is not getting hit by the Linux out of memory killer but I'm not sure.Docker also does not give me any warning or error message.
How can I find out whether the RAM is actually the issue here or if there is something else?
The text was updated successfully, but these errors were encountered: