out_of_core configuration and documentation #3845
Labels
documentation 📜
Updates and issues with the documentation
External
Pull requests and issues from people who do not regularly contribute to modin
Needs more information ❔
Issues that require more information from the reporter
P3
Very minor bugs, or features we can hopefully add some day.
System information
I'm still unsure as to what the documentation is suggesting here.
Does the line below, as it stands, disable out-of-core, or does it only disable out-of-core when _plasma_directory=None? If it does disable out-of-core as it stands, how does one specify a desired directory for spilling, instead of the default spilling directory (as I cannot use the default)?
ray.init(_plasma_directory="/tmp") # setting to disable out of core in Ray
Currently, the following is my setup:
I just want to know if this is the most memory efficient setup, which would prevent my program running out of RAM, by spilling onto disk, no matter how large the dataframe (within the bounds of my disk space)? And furthermore, does that extend to doing very expensive operations such as merge?
I would really appreciate it if you could settle this for me.
Thanks!
Pej
Originally posted by @Peji-moghimi in #3705 (comment)
The text was updated successfully, but these errors were encountered: